Erdogan issued a decree to exit the 2011 treaty, which calls for signatories to protect women against violence, reports dpa news agency.
The move sparked Saturday's protests by women's rights groups, lawyers and opposition lawmakers in a number of cities, including the capital, Ankara, local media reported.
"Cancel the decision, implement the convention," protesters shouted behind a police barricade in Istanbul's Kadikoy district.
The protesters carried banners that read "Women will not bow" and "Equality for Women," along with rainbow flags.
Video on social media depicted demonstrators calling out the names of women murdered by men.
Turkey had pre-existing laws to defend women's rights, but the convention brought in practical measures to protect women and children, in particular from domestic violence.
The treaty allowed most women in Turkey to escape violence by using the courts to change their identity, move to a safe address or requiring the placement of ankle monitors on men, said a lawyer among the protestors, who asked to remain anonymous.
But with the convention now being scrapped, millions of women, children and lesbian, gay, bisexual and transgender (LGBT) people will now be deprived of these life-saving measures, a lawyer told dpa over the phone.
"You cannot deprive 42 million women of their rights in an overnight decree," Kemal Kilicdaroglu, leader of the main opposition Republican People's Party (CHP) said in a video shared on Twitter.
The party said it would appeal the decision to the country's constitutional court.
Violence against women is a widespread problem in Turkey.
Erdogan's ruling party officials raised the notion of abandoning the treaty last year as conservative groups argued the convention damaged families and religious values, while encouraging the LGBT community.
The debate had sparked street protests by rights groups.
Erdogan's Cabinet members, including ministers of family, justice and interior, moved to soothe concerns on Saturday, saying women's rights are "guaranteed" under Turkish law, according to state news agency Anadolu.
Conservative groups have argued that the treaty offers a pretext for the LGBT community to expand its base in the predominantly Muslim country because the convention includes language forbidding discrimination based on sexual preference.
Homosexuality is not illegal in Turkey, but attacks on LGBT communities and their supporters are common.
The Istanbul Convention, a treaty developed by the non-EU organization Council of Europe, aims to create a legal framework to prevent and fight against violence against women and domestic violence.
It was later ratified in Turkey, but, according to the country's We Will Stop Femicide Platform, it was never applied.
According to the platform, at least 300 women were murdered by men in Turkey last year alone.