Merz, 69, who led his CDU/CSU conservatives to win a federal election in February and since secured a coalition deal with the centre-left Social Democrats (SPD), won just 310 votes in the lower house of parliament, Bundestag President Julia Kloeckner said. He needed 316 to secure a majority, Reuters reported.
Kloeckner said she was interrupting the parliamentary session so that the parliamentary groups could consult on how to proceed.
The lower house of parliament, or Bundestag, now has 14 days to elect Merz or another candidate chancellor with an outright majority - and could attempt another vote already on Tuesday.
Merz's conservatives won national elections in February with 28.5% of the vote but need at least one partner to form a majority government.
On Monday they signed a coalition deal with the centre-left Social Democrats, who won just 16.4%, their worst result in German post-war history.