The withdrawal from the Incirlik base, approved by the German parliament last month, marks a further step in one of many bilateral disputes, ranging from a post-coup clampdown by Ankara to Turkish political campaigning in Germany.
German tornado jets were due to keep operating out of Incirlik at least until the end of July as part of a mission providing reconnaissance aircraft to support US-led coalition operations against Islamic State in Iraq and Syria.
In the meantime, the necessary material was to be moved to a new air base in Jordan, where the planes are scheduled to be deployed by October.
A German air tanker refueller left Incirlik for the Jordan base on Sunday, the ministry spokesman told Reuters.
A Turkish official confirmed that the withdrawal had started, saying Germany's defense minister had informed her Turkish counterpart of the withdrawal date when they met during the NATO summit in Brussels.
Turkey had refused to allow German lawmakers to make what they saw as a routine visit to the base, saying that Berlin needed to improve its attitude towards Turkey first.
Germany had banned some Turkish politicians from campaigning on its soil for a referendum on giving Erdogan sweeping new powers.
Ankara responded by accusing Berlin of "Nazi-like" tactics and reigniting the dispute over Incirlik.
German Chancellor Angela Merkel said on Saturday that a bilateral meeting with Turkish President Tayyip Erdogan had revealed "deep differences" between the two NATO allies.