Libya's ambassador to the United Nations said that his government is refusing to give consent for UN action aimed at endorsing the plan because European governments had discussed it with Libyan militias that control coastal territory.
"The meeting... took place in the context of the preparations of the maritime operation EUNAVFOR MED, which will focus on dismantling the business model of those involved in human smuggling," a statement said, using an acronym for the planned force.
Federica Mogherini met with Libya's Mohamed al-Dayri along with Italian foreign minister Paolo Gentiloni and Britain's Philip Hammond in Brussels. The force's command centre would be Italy-based, while Britain is drafting the UN resolution giving the force its legal framework.
The scheme, backed by foreign and defence ministers in Brussels, will involve European warships and surveillance aircraft gathering intelligence and then raiding boats to crack down on people smugglers, AFP reported.
On May 18, EU nations approved plans for the unprecedented naval mission but the UN resolution must pass before it can begin to destroy boats that belong to smuggling gangs in Libyan waters, where political turmoil has created a safe harbour for traffickers.
The worst Mediterranean disaster in decades took place in April, when a trawler packed with migrants capsized off the coast of Libya, killing around 800 people.
Since the beginning of the year, more than 45,000 illegal migrants have arrived in Italy.