"As a player you get to the point where you've had enough," Philander told Arikaans weekly Rapport.
"CSA's previous administration only looked after themselves; the players were the last people they worried about.
"Too many things went wrong and I had to decide what was the best way forward for me. I am 34 and have a good career behind me, but I would have considered playing longer if it wasn't for the chaos in our cricket administration."
Relationship between players and Cricket South Africa has taken several hits over the past decade. The worst for Philander came when he replaced Kyle Abbott, South Africa's best bowler in the 2015 World Cup, ahead of the semi-final of the tournament against New Zealand despite nursing a hamstring injury after then CEO Haroon Lorgat reminded the national team selectors of the transformation commitments. Philander said that he should not have played that match.
"It's difficult for me to look back on that," he said. "I blatantly and openly told the coach that the best player must play. He told me, 'you're the best man for the day, you play'. But they were clearly not open and honest with me and Kyle. There were things happening behind closed doors," he said.
"When I go to Durban, I have a beer with Kyle. There are no hard feelings between us two. But the point is: Cricket SA must sort out their stuff. What happened was a knock to both of us."
With former captain Grame Smith appointed as acting director of cricket and former wicketkeeper Mark Boucher appointed as coach, Philander said that there is hope for a turnaround but it is too late for him to reconsider his decision.
"Hopefully we will see a turnaround in the administration and on the playing field," he said. "We must put out heads together and decide which direction we are going in. Hopefully we can make the path for younger players better."
Philander said that more former players need to be brought back to helping South African cricket but that can happen only if they are paid well.
"In successful teams like Australia, England and India, former players are involved but in South Africa, we lost our former players to other countries where they do coaching because the money is much better," Philander said. "We have to decide what we are willing to pay to keep former players in the country and ensure that our cricket goes forward again."