The Congress leaders are among those invited by the prime minister to the ceremony.
They have accepted the invite, the sources said.
A number of opposition leaders have been invited to the oath-taking ceremony and many of them are attending. It will be the first time that the leaders will be together after an acrimonious Lok Sabha election campaign.
But West Bengal Chief Minister and Trinamool Congress leader Mamata Banerjee has cancelled her plan to attend the ceremony after the BJP invited to the event the kin of 54 party workers it claimed were murdered in political violence in West Bengal.
A day after she told reporters that she has accepted the invite as a matter of "constitutional courtesy", Banerjee termed as untrue BJP's allegation of political murders in her state and said she is "compelled" not to attend the ceremony.
As many as 8,000 guests, including VVIPs, will attend the swearing-in ceremony of Prime Minister Modi and his council of ministers on Thursday. This is the highest number of guests attending such a function, the sources said.
The ceremony will be of about 90 minutes and after that dinner will be served to around 40 dignitaries, including Vice President M Venkaiah Naidu and the prime minister, besides leaders of all BIMSTEC countries, Rashtrapati Bhavan spokesperson Ashok Malik said Wednesday.