Federer had a match point in the tiebreaker but Chardy saved it with an improbable cross-court forehand on the run when it looked like Federer was about to close out the match.
After cruising through the first set easily, Federer began to commit unforced errors at an alarming rate and Chardy quickly took advantage amid swirling wind at the Foro Italico. The fourth-ranked Federer committed 43 unforced errors — 20 in the second set alone — to Chardy's 28. Federer also struggled to dictate play with his first serve.
Chardy hadn't beaten a top-10 player since defeating Juan Martin del Potro in the third round of last year's Australian Open.
Federer's boys, Leo and Lenny, were born last week, prompting the 17-time Grand Slam champion to withdraw from the Madrid Open. Federer and wife Mirka also have twin girls, Myla Rose and Charlene Riva, who turn 5 in July.
Federer said Tuesday that this week would be different for him and that it didn't matter whether he plays five matches or one. Still, Rome is a key clay-court warmup for the French Open, which starts in 11 days.
Also, Wimbledon champion Andy Murray beat Marcel Granollers of Spain 6-2, 7-5 and will next face Jurgen Melzer of Austria, who edged Marin Cilic of Croatia 6-2, 6-7 (5), 6-3.
Also advancing was 15th-seeded Tommy Haas, who eliminated Igor Sijsling of the Netherlands, 7-6 (6), 6-1.
In women's play, third-seeded Agnieszka Radwanska beat Paula Ormaechea of Argentina 6-3, 6-2 and will next face 2010 French Open winner Francesca Schiavone, who rallied past Garbine Muguruza 3-6, 6-1, 7-6 (5).
Tenth-seeded Sara Errani swept aside Ekaterina Makarova 6-2, 6-3; and 13th-seeded Carla Suarez Navarro beat 1999 champion Venus Williams 6-4, 6-2.
Also, Varvara Lepchenko upset 16th-seeded Sloane Stephens 6-2, 6-2 in an all-American match; American qualifier Christina McHale rallied to beat Italian wild card Camila Giorgi 1-6, 6-3, 6-1; and Australian veteran Sam Stosur defeated Elena Vesnina of Russia 6-2, 6-3.
Follow Andrew Dampf at http://twitter.com/asdampf