Twenty-four construction workers who were at the site of the unfinished building when it collapsed have been removed from the rubble so far, while 51 remained unaccounted for, the municipality of George, a coastal city east of Cape Town, said in a statement.
Four among the rescued have been declared dead as of 5:15 a.m. (0315 GMT), it said.
A drone view of the site showed rescue workers removing debris in a bid to reach those still under the rubble.
"We are focusing on the areas... where we have actually been hearing people," Colin Diener, chief of disaster management for the Western Cape province, told reporters at the scene.
"It is going to get more and more difficult as the days go on," he said.
CCTV footage obtained by Reuters showed a cloud of dust as the building crumbled on Monday afternoon.