A number of explosions were heard when officers in tactical gear entered the pharmacy in the city centre after first arriving on the scene at around 4:30 p.m.
"A special unit entered the pharmacy at 21:10; a male suspect was arrested," Karlsruhe police said on Twitter, Reuters reported.
Officers restrained a man they had brought out of the pharmacy, covered his head in a blanket and bundled him into a car.
The building was being searched and there appeared to be no injuries, they said.
The incident follows Thursday's deadly rampage at a Jehovah's Witnesses hall in Hamburg, putting the country on edge. Gun violence is rare in Germany.
Earlier police said in a joint statement with prosecutors that there were "multiple" hostages involved.
Police had cordoned off Karlsruhe city centre where multiple blue and grey police vehicles with flashing lights lined the streets.
The Stuttgarter Zeitung reported that two people had been taken hostage and that there was a demand for a ransom of a single-digit million euro sum.
The police spokesperson declined to comment on the report.
Earlier Germany's Bild newspaper had reported that the police were in contact with the alleged hostage-taker.
Karlsruhe, not far from the French border, is a city of some 300,000 people and home to the Federal Court of Justice, Germany's highest court.