Armed men, who have not yet been identified, entered the towns of Jalulah and Saaiydiyah and used loudspeakers to tell local police that if they laid down their weapons and left their posts they would not be hurt, locals told Al Jazeera.
The security forces complied and left the towns, they said.
Witnesses and an Al Jazeera journalist in Diyala said the men then promised locals they would be unhurt.
Security sources told Reuters that several other villages in Diyala, around the Himreen mountains, which have long been a hideout for armed groups, had also fallen to rebels.
The Iraqi army fired artillery at Saaiydiyah and Jalulah from the nearby town of Muqdadiya, sending dozens of families fleeing towards Khaniqin, near the Iranian border, security sources said.
The armed men said Muqdadiya was their next target, according to locals.
Fighters from the Islamic State in Iraq and the Levant (ISIL) overran the northern city of Mosul earlier this week and
have since pressed south towards Baghdad in an onslaught against the government.
US President Barack Obama threatened US military strikes against the militant group who want to establish their own state in Iraq and Syria.