Scholz, 63, who over the past four years served as vice chancellor and finance minister in coalition with Merkel, won a majority of 395 votes from lawmakers in the lower house of parliament, Bundestag President Baerbel Bas said.
Wearing a black face mask, Scholz waved as he received a standing ovation from lawmakers and accepted bouquets of flowers and a basket of apples from leading members of parliament.
After being formally nominated by President Frank-Walter Steinmeier in the nearby Bellevue Palace, Scholz returned to the historic Reichstag building in the heart of the German capital to take the oath of office in front of lawmakers and become Germany's ninth chancellor since the end of World War Two.
Scholz swore to devote his energies to the welfare of the German people - but he did not ask for God's help, a phrase that is usually included in the oath of office and was spoken by Merkel.
Back in Bellevue Palace, the ministers of the new cabinet received their appointment certificates from the president.
In the afternoon, Merkel officially handed over the chancellery to Scholz and wished him a "steady hand" as the country faces a brutal fourth wave of coronavirus infections and challenges to its democratic order from authoritarian leaders.
In a short speech, Scholz hailed Merkel as a great chancellor and thanked her for a close and trustful relationship over the past years.
"I would like to build on the north-east German mentality, if I may say so, that has reigned here so far. Not much will change it this regard," Scholz said in reference to their shared northern roots in the port city of Hamburg.
With his down-to-earth and no-nonsense manner, Scholz has positioned himself as Merkel's natural successor and a safe pair of hands to steer Germany through challenges ranging from tackling the climate crisis to dealing with a more confrontational Russia and increasingly assertive China.
Scholz will lead an unprecedented three-way ruling coalition on the federal level with the pro-spending, environmentalist Greens and the fiscally more conservative, libertarian Free Democrats (FDP) - unlikely political bedfellows in the past.
Scholz is an experienced negotiator and veteran politician who as SPD party secretary general from 2002 until 2004 defended controversial labour market reforms and social welfare cuts under former Chancellor Gerhard Schroeder.
As labour minister from 2007-2009 in Merkel's first ruling coalition, Scholz pushed for a generous short-time work scheme that helped protect millions of workers from the fallout of the global financial crisis.
After being mayor of Hamburg from 2011-2018, Scholz returned to Berlin as Merkel's finance minister - a role in which he got rid of the goal of balanced budgets and enabled record new borrowing to shield companies and employees from the impact of the coronavirus pandemic.
On the European level, he worked closely with France and persuaded Merkel to support a debt-financed European Recovery Fund worth 800 billion euros ($900 billion) to help EU member states hit hardest by COVID-19.
Merkel, 67, a lover of opera and hiking in the little spare time she has had in the last 16 years, has given few clues about what she plans to do in retirement.