Neuer, who had not played a single official game since breaking a bone in his foot back in September, was handed the captaincy and looked fresh and confident but was powerless to stop Austria’s two second-half goals.
It was Germany’s fifth match without a win and their first defeat to Austria for 32 years.
Forward Nils Petersen was given his Germany debut at the age of 29 but was largely anonymous.
“It was a good comeback for him (Neuer),” said Germany coach Joachim Loew. “He made two or three good saves and I’m absolutely satisfied with his performance. He had no problem with his injury.”
Loew was less happy, however, with his team’s performance in a match which began one and three-quarter hours later after torrential rain twice forced the kickoff to be put back.
“Of course, the defeat angers me,” he said. “We lost too many balls which normally doesn’t happen with us, we missed chances after the goal, a lot of things were bad today.
“I learned some important things from the game. We have a lot to improve over the next 14 days.”
Austria gifted their bigger neighbours an eleventh minute lead when goalkeeper Joerg Siebenhandl sent a dreadful clearance straight to Mesut Ozil who scored from just inside the penalty area.
The hosts came back well and Neuer made a superb close-range save from Florian Grillitsch just after the half hour.
The 32-year-old was powerless, however, when David Alaba’s corner found Martin Hinteregger at the far post and he hammered a low volley past Neuer in the 53rd minute.
Neuer made another important stop, this time from Marko Arnautovic, but was beaten again from close range Alessandro Schoepf in the 69th minute after Stefan Lainer was allowed too much space to lay the ball off for him at the far post.
Germany team manager Oliver Bierhoff said on Friday that Neuer will be their first-choice goalkeeper at the World Cup if he makes the final squad, which will be announced on Monday when four players will be cut from the current 27.
Neuer missed almost the entire season after breaking a bone in his foot and undergoing surgery and his comeback for Bayern Munich was repeatedly delayed.