He said those who are trolling him over the remark should first prove him wrong by producing counter research papers.
Ghosh, a MP from Midnapore Lok Sabha seat, earlier this week had courted controversy, by saying that Indian cow milk contains traces of gold and that is the reason the colour is yellow.
"I stand by my comment and there is no question of withdrawing it. My comment is backed by proper research work that is going on in the foreign countries. Those who are trolling me or criticising me should first prove me wrong by coming out with counter research papers," Ghosh told PTI over phone.
Hitting out at those who are criticising him as a "fool", Ghosh said "fools can never understand the significance of cow and its milk in Indian society".
Those who are trolling me on twitter and other social media platforms have no other work except attacking those who talk about India's culture and heritage, he said.
His remark last week drew sharp reactions from both ruling TMC and opposition Congress and CPI(M), who said "people of Bengal would be the best judge of such insane statements".
"What can we say about such statements? The people of Bengal will be the best judge of such insane statements," West Bengal Subrata Mukherjee had said.
Bengal Congress president Somen Mitra had said such comment has been made with a definite agenda of trying to divert attention from the failures of the BJP government at the centre.