Romney, who led 2002's Winter Games in Salt Lake City, said he also worried about security preparations but predicted Russia has done everything it can to prevent violence and to protect athletes at the Sochi games.
"I don't think any major global event that brings people from all over the world together like that can possibly escape the threat of violence," said Romney, whom Republicans picked to challenge President Barack Obama's re-election bid in 2012.
Twin suicide bombings in the Russian city of Volgograd left 34 victims dead last week, just 400 miles from where the Sochi Games will be held.
No group has claimed responsibility for the bombings, but they highlight just how vulnerable the games could be to militant attacks. Separatists seeking to carve out an Islamic state in the North Caucasus have mounted attacks across Russia for nearly two decades, following wars between separatist Chechen rebels and Russian forces.
"Russia has a special problem given the threats that have been leveled," Romney said.
"There's no question about it. It's very, very frightening to have any kind of Olympic event on your national soil," he added.
Romney's criticism of Russia is hardly new. During his presidential campaign, Romney called Russia "our No. 1 geopolitical foe."
Picking up that criticism, Romney said Russian President Vladimir Putin has not done enough to pressure Iran to curb its nuclear program or push Syrian President Bashar Assad from power. He also criticized Russia for harboring Edward Snowden, a former National Security Agency contractor who leaked documents disclosing several secret spy programs, and for shifting away from democratic reforms that Olympic organizers sought to reward when they picked Sochi to host the games.
"Were it my choice, I would not vote for Russia to hold the Winter Olympics or the Summer Olympics. But it's not my choice," Romney said.
Russia also has faced criticism over its new anti-gay law, mounting costs and environmental concerns about the games.
Asked during a "Fox News Sunday" interview whether Russia's behavior undercut the Olympic games' sprit, Romney answered yes. He invoked the 1936 Berlin games that took place under Adolf Hitler and his fellow Nazis.
"Hitler's presence there, that certainly undercut the Olympic message," Romney said. "And surely that can happen in our time, theoretically or specifically."
Follow Philip Elliott on Twitter: http://www.twitter.com/philip—elliott