The victims had set off in a parade of fishing boats from the tourist resort of Beruwala but halfway into their journey their craft overturned in choppy seas, police said.
"Other fishing boats and volunteers have been able to rescue 32 people, but we still do not have a count on the number of people on board," a police officer in the area told AFP by telephone.
He said searches by navy boats as well as aircraft were hampered by heavy winds and rain.
Sunday's boat parade came amid a warning to fishing communities from the meteorological department of winds up to 80 kilometers (50 miles) an hour.
Sunday's tragedy was the worst boat disaster since June 2013 when 54 fishermen were killed after several boats capsized during a freak storm in the same area.