You should learn to love the evil eye.
Seriously, that is the answer. People are going to do what they are going to do. You have integrity, you did not disrupt their cult activity and you handled yourself, from what you said, like a gentleman. If that irritates them, they can all get bent double.
However. If their irritation irritates you, get used to it. You live in a nation of cults. Like it or not, for now, you are the odd man out. In my opinion you behaved properly. There isn't much else you can do - until you are the speaker and do not offer a prayer.
Yup.. they do that at my work. I don't live in the US, but a large part of South Africa is as deluded as those you might be familiar with. I stare at the wall, like you did. I am not legally obliged to pray in the workplace - what about the Hindu girl at the end of the table? Would you make her pray to your god?
From what I've gathered, if you look like a duck and walk like a duck, you must believe in Jehovah. Some people just can't comprehend that non-believers exist. I brought it up with a Christian colleague (without letting my particular cat out the bag), and she just spurted out some very confused lines about atheists believing in the power of nothing, and that they could pray to that...
1.) They can't comprehend that someone couldn't possibly believe in god
2.) If they did, they'd think those people are bad and will go to hell
It's just ignorance. Nothing more.
As to what to do, we had a great and fun discussion on this. http://www.thinkatheist.com/forum/topics/bow-your-head-close-your-eyes
We live in Utah Valley. Here, you have to go to BYU to attend graduations, sports games, and many other things for public gatherings if you want to stay with the crowd. At BYI, they always start every meeting with a prayer.
Complaining aside, I just close my eyes so I don't offend. Yeah... I need to grow a pair and piss some people off.