According to the police, the arrested persons -- Mamunur Rashid, 22, two teenagers aged 15 and 16 and Kafil Uddin, a 50-year-old village doctor -- admitted to their crimes on Saturday.
The violent incident took place on Friday at the Kali Mandir in Kadim Maijhati area.
The miscreants entered the temple, vandalised five idols and fled.
The attack took place just as the worshipers were preparing to immerse the idol.
According to the police, Rashid, who is an Imam, led the attack.
Police Officer-in-Charge (OC) Shamsul Alam Siddiqui told IANS that Birendra Chandra Bormon, the vice president of the temple, lodged a case naming eight persons and some 35 unidentified others on Friday night.
At least four persons have been killed and 70 others injured in the incidents of violence including attacks on Hindu temples across the country in the last three days.
The radical attackers vandalised, torched, and looted more than 70 puja venues, 30 homes, and 50 shops of the minorities, according to Rana Dasgupta, General Secretary of Bangladesh Hindu Buddhist Christian Unity Council (BHBCUC).
Around hundreds of people were arrested over the attacks and the spread of communal hatred on social media.
BGB personnel were deployed in 24 districts and security was beefed up at places of worship, police claimed.