Top-ranked Rafael Nadal was also taken to three sets but emerged with a 6-3, 6-7 (3), 6-3 win over Tobias Kamke of Germany, while second-seeded David Ferrer joined Murray in making an early exit after a 6-4, 7-5 loss to Daniel Brands of Germany.
Murray was leading 6-3, 3-0 when the 40th-ranked Mayer started to play more aggressively, going for winners on nearly every shot. The German won 12 of the last 15 games, closing out the match with a net-cord winner to seal the win in 1 hour, 51 minutes.
Murray hit 25 winners, but the reigning Wimbledon champion also had 37 unforced errors.
"I didn't have extremely high expectations, because I haven't obviously played a match for a long time," Murray said. "You don't know exactly how your body is going to respond."
The last four times Murray played a tournament during the first week of the new season he captured the title: in 2008 and '09 at Doha, and 2012 and '13 at Brisbane.
Nadal trailed 4-1 in the second set but bounced back to level it at 4-4 before Kamke dominated the tiebreaker, taking a 5-0 lead.
Nadal earned three breaks in the match—one in each set—and each time Kamke double-faulted on break point.
"Important win," said Nadal, who next faces seventh-seeded Ernests Gulbis of Latvia. "I spent a lot of time on court, so that's important for my preparation, too."
Gael Monfils upset fellow Frenchman and defending champion Richard Gasquet 6-2, 7-5 in the final match of the night. The fifth-seeded Gasquet has been dealing with a back problem this week.
Two other seeded players, No. 6 Philipp Kohlschreiber and No. 8 Fernando Verdasco, were also eliminated.
Kohschreiber fell to qualifier and fellow German Peter Gojowczyk 7-6 (4), 7-6 (7), while Verdasco lost to Victor Hanescu of Romania 4-6, 7-6 (0), 6-2.