Sunday's announcement comes a day after Doctors without Borders (MSF) said it was halting the use of its largest boat in the area because of an "increasingly hostile environment for lifesaving rescue operations" of asylum seekers.
In a statement, the aid group Sea Eye said it was with "a heavy heart" that it had decided to follow suit after the Libyan government's "explicit threat against the private NGOs".
Tensions have risen since the Libyan navy on Thursday ordered foreign vessels to stay out of a coastal search-and-rescue zone; a measure it said was specifically aimed at non-governmental groups, AFP reported.
Libyan authorities have accused charities of aiding human smugglers with their rescues at sea, hampering efforts to crack down on the migration route.
"Under these circumstances, a continuation of our rescue work is not currently possible. It would be irresponsible towards our crews," Sea Eye founder Michael Buschheuer said.
But he cautioned that the retreat of the aid groups was putting lives at risk.
"We leave behind a deadly gap in the Mediterranean," he warned.
Italian Foreign Minister Angelino Alfano, however, welcomed Libya's stepped-up efforts to curb the flow of desperate people.
The Libyan government "is ready to put in place a search-and-rescue zone in its waters, work with Europe, and invest in its coast guards," Alfano told La Stampa daily on Sunday.
"This sends a signal that the balance is being restored in the Mediterranean."
Italy, which has borne the brunt of Europe's migration crisis this year, has itself moved to rein in NGOs helping the multinational rescue operations by making them sign up to a new code of conduct.
Six years since a revolution that toppled longtime dictator Muammar Gaddafi, Libya has become a key departure point for people risking their lives to cross the Mediterranean to Europe.
Tens of thousands have resorted to paying people traffickers for the journey, often on overcrowded and unseaworthy boats.
Sea Eye says it has helped save some 12,000 lives since April 2016.