The alleged ISIS operative, before his arrest, worked as a laboratory technician at Ankleshwar town's Sardar Patel Hospital, with which Patel had been associated as a trustee.
Gujarat Chief Minister Vijay Rupani had asked for Patel's resignation from Parliament over the issue.
Reacting to the charge, the political adviser to Congress president Sonia Gandhi said his election to the Rajya Sabha had become an "eyesore" for the BJP.
"When they face difficulties, they talk of nationalism and communalism. Recently, the chief minister said something about me, he should be ashamed (of it)," Patel said addressing a public gathering here in the presence of Congress vice president Rahul Gandhi.
"The way they weaved the story it seems that they have the sole contract over nationalism. Only they are nationalist, nobody else is," he said.
He said the Congress and its leaders had made many sacrifices for the country.
"It started with Mahatma Gandhiji, and after independence, Indiraji, Rajivji. Many party workers and leaders became target of terrorism...You cannot teach us nationalism, patriotism," he said.
"You cannot fight terrorism. You went and handed over terrorists in Kandahar (Afghanistan). How are you going to fight terrorism," Patel said, referring to the then NDA government's handing over of terrorist accused in exchange of passengers following the hijacking of Indian Airlines flight IC 814 in 1999.
Patel stressed he would not resign, adding that he had been elected by MLAs.
"They are asking for my resignation. My win in the Rajya Sabha polls has become an eyesore for the BJP. I will not resign on your demand. My 44 MLAs have elected me," Patel said, referring to the poll to the Upper House in which he had defeated his BJP rival despite cross-voting by some Congress MLAs.
The BJP recently accused Patel of having "deep relations" with the hospital in Bharuch district.
"I want to tell them why are you making such allegations when the country has so many security agencies. Let them investigate and come out with the truth," the senior Congress leader said.
He added that he had resigned from the hospital in 2015.
"I had given (my) resignation from the hospital way back in 2015. I had joined it as the people of our area wanted us to build a good hospital. The management is different," said Patel, who belongs to Bharuch district.
"Regarding the ISIS terrorist, I have done some research. He had earlier worked in two hospitals and one of them was inaugurated by BJP leaders. Why are there no allegations against them," he asked.