The deportations violate international and Cameroonian law and constitute one of the world's largest recent cases of illegal forced repatriation, the rights group said in a report.
They could also have intensified one of the world's largest humanitarian crises by sending people in dire need of aid back to northeast Nigeria, it said, Reuters reported.
Cameroon's minister of communications declined to comment on HRW's report.
Nigeria is struggling to cope with millions of people dependent on emergency relief for food, shelter and medicine, victims of an eight-year conflict in which at least 20,000 have been killed.
At least 10.7 million people are in need of assistance in the region, mainly in Nigeria but also in Chad, Niger and Cameroon, according to the United Nations.
The forced returns are a "flagrant breach" of international and Cameroonian law and fly in the face of pleas by the United Nations and other aid groups, the report said, drawn in part from interviews with former asylum seekers and refugees.
"Cameroon's army has been aggressively screening newly arriving Nigerians at the border, subjecting some to torture and other forms of abuse, and containing them in far-flung and under-serviced border villages," the report said.
"This policy of blocking asylum seekers from accessing protection has made it easier for Cameroon to deport them," it said, adding that the UN refugee agency (UNHCR) has been denied access.
A UNHCR spokesman in Nigeria's capital of Abuja declined to comment, and the agency's representative in Cameroon did not respond to an email seeking comment.
Cameroon signed a treaty in March with Nigeria and the UNHCR to ensure that all refugee returns are voluntary but the rights group said Cameroon's violations had continued.
Cameroonian soldiers tortured, beat to death and sexually exploited some of them, HRW said.
"They humiliated us like animals and beat us like we were slaves," one interviewee told the rights group. His brother died from the resulting internal bleeding, he said.
Some deportees, including children, were so malnourished or sick they did not survive their return to Nigeria, HRW said.