The suspect was taken into custody on suspicion of murder after barricading himself in his father’s home near the city of Nakano, northwest of the capital Tokyo, on Friday.
The attack began on Thursday afternoon when a local man working on a farm saw a woman “running from the road shouting, ‘help me’,” he told national broadcaster NHK.
“Behind her was a man wearing camouflage and carrying a large knife, who stabbed her in the back,” the 72-year-old witness added.
The suspect is accused of shooting dead two police officers who arrived at the scene after the witness called emergency services, Al Jazeera reported.
The two officers, who were allegedly shot in their car, and the woman were taken to hospital where they were pronounced dead within hours.
Another elderly woman also died after an apparent attack. She had been lying on the ground outside the house since Thursday afternoon and police had been unable to get to her, media said.
The rampage in a rural area of the western region was a rare instance of violent crime in Japan, which has a low murder rate and some of the world’s toughest gun laws.
No motive has yet emerged in the killings, nor has the suspect been formally identified, although several local media reported that he is the son of the speaker of Nakano’s city assembly.
Few other details were known about the attack, including the suspect’s motive.