Four men were found "loitering" outside Verma's official residence and were being questioned by police, sources claimed Thursday.
Deputy Commissioner of Police Madhur Verma, however, denied detaining four men and questioning them.
Suffering from "Rafale-o-phobia", Prime Minister Narendra Modi is indulging in "snooping, surveillance and spying" of the Central Bureau of Investigation (CBI), Leader of Congress in the Lok Sabha Mallikarjuna Kharge and senior leader Abhishek Singhvi said at a press conference.
There was no immediate reaction from the Prime Minister's Office on the allegations.
Kharge has also written to the prime minister raising objections over the removal of Verma.
Kharge and Singhvi alleged that the Intelligence Bureau was "snooping" on an "officer who was about to unravel the murky dealings in the Rafale scam".
"'Big Brother Syndrome' is the prime modus operandi by a government who has utterly lost the plot," the two leaders said in a statement.
Congress' chief spokesperson Randeep Surjewala wrote on Twitter: "CBI director is illegally removed at 2 AM. Today, 4 IB operatives caught snooping outside his house. This is straight out of a page turning thriller where crime meets political intrigue."
Party president Rahul Gandhi Wednesday accused the prime minister of removing Verma in a bid to stop him from probing the Rafale "scam".
Finance Minister Arun Jaitley has, however, dismissed as "rubbish" the allegations that Verma was removed because he wanted to look into the Rafale fighter jet deal.
He had said the government's decision to remove Verma and CBI Special Director Asthana was based on the Central Vigilance Commission's recommendations.