Top-seeded Serena was frustrated by a swirling wind, her inability to find her A-game, and a tenacious opponent as she went down 6-4, 6-7 (3), 6-4 to 72nd-ranked American compatriot Madison Brengle.
Older sister Venus withdrew from the tournament with an arm injury ahead of her scheduled second-round match.
Both players will now head to Melbourne to prepare for the first Grand Slam tournament of the season with less match play than they would have hoped.
Serena had shown signs of rustiness in her 6-3, 6-4 first round win over Pauline Parmentier of France, and her power-hitting game was again affected by gusting wind on the open-air center court.
She struggled with her ball toss in the windy conditions and made 88 unforced errors, including the double fault that ended the match and caused Brengle to clap her hand over her mouth in surprise.
Brengle, a 26-year-old Delaware native, had played Serena only once before, in 2015, losing 6-0, 6-1.
Williams, a 22-time Grand Slam tournament winner, played only eight events in 2016 the most recent at the U.S. Open in August but still got off to a good start, taking a 4-1 lead before faltering.
Brengle rattled off four straight games to lead 5-4, and then served out the set. Williams was down a break in the second set before breaking Brengle and then powering through her service game to lead 5-4.
She had four set points in Brengle's next service game but again her accuracy deserted her and she squandered every one. It finally took a fifth set point in the tiebreak for Williams to close out the set in 60 minutes.
Games went on serve in the third set, and Brengle's plan of taking the pace off the ball with her returns paid off as she broke Williams in the final game.
"I just tried to concentrate on every point and run down every ball," Brengle said.
"It was difficult conditions with the wind and I just tried to right for every point. I tried to use the slice and get as many balls back as I could and it ended up working out.