Other areas remain under evacuation orders, however, according to state fire officials, and those in San Andreas have been told to remain prepared, RT reported.
Images posted online show the fire creeping to the town, with smoke filling the air.
The "Butte Fire," as it's been called, was described as "extremely dangerous" by California State Fire Spokeswoman Nancy Longmore.
"It's expanding like a balloon," she said. "It's moving very fast. There's many homes threatened."
Though it began Wednesday, the Butte Fire had only burned through a few hundred acres through Thursday. By the end of Friday, however, it had engulfed 65,000 acres after being prodded by temperatures over 100 degrees Fahrenheit.
It's now only five miles away from San Andreas and only 5 percent contained, according to state officials.
California Governor Jerry Brown has declared a state of emergency in Amador County and Calaveras County, where San Andreas is located, in order to help battle the flames.
About 6,000 structures are in danger, according to Longmore, though she did not say how many are homes.
Eight structures have already been destroyed.
More than 1,500 firefighters and 16 helicopters have been dispatched to combat the blaze, the Los Angeles Times reported.