Locals were forced to leave in the early hours as health officials in masks and white overalls scrambled to work out whether the virus had spread through the 35-storey complex that houses some 3,000 people.
Hong Kong is on high alert for any potential "super spreader" events, especially in the towering housing blocks that make the city one of the world's most densely populated places.
During the 2003 outbreak of Severe Acute Respiratory Syndrome (SARS), which killed 299 people in Hong Kong, 42 deaths came from just one housing block where about 300 people were infected.
In that outbreak, the virus was found to have spread through faulty drainage pipes.
Officials said Tuesday's relocation of residents in Tsing Yi district was a precautionary measure after three members of the same family contracted the virus.
The family lived 10 floors directly below another man who had already been diagnosed as a carrier.
"We are not sure what was the exact route of transmission," Wong Ka-hing, from the Centre for Health Protection, told reporters.
"It could still be through the usual method of droplets or contact."
Nonetheless the occupants of 35 flats connected to the same drainage system were moved out.
Health secretary Sophia Chan said four residents who showed flu-like symptoms were taken to a hospital isolation ward but later tested negative for the virus. The others were taken to quarantine camps.
Residents on Tuesday morning found their neighbourhood filled with police and health officials.
"Of course I'm scared," a 59-year-old resident, who gave her surname as Chan, told AFP.
"I live with my son, daughter-in-law, grandchildren and my husband. We seldom go out already because we don't have enough masks. I don't allow my grandchildren to play in the hallway. Now we can't even stay at home."
There are now 49 confirmed cases of the virus in Hong Kong, including a cluster of 10 family members who had all shared a hotpot meal with an infected person.
The SARS epidemic left profound psychological scars on Hong Kong and saddled locals with a deep distrust of authorities in Beijing who initially covered up the outbreak.
The financial hub has been hit by panic buying even though the government has said imports remain steady.
There is an acute shortage of face masks -- including in hospitals where stocks are being rapidly depleted -- fuelling anger towards the city's pro-Beijing leaders.
On Saturday, the city began enforcing a 14-day mandatory quarantine for anyone arriving from mainland China after resisting calls to close the border.
So far, about 2,200 people have crossed the border. The vast majority have been told to self-quarantine at home while a few dozen without addresses have been taken to government facilities.
Yuen Kwok-yuen, an expert from a University of Hong Kong team that is studying the virus, said the city's bid to halt outbreaks should be easier now that arrivals from the mainland have been dramatically curbed.
"We are now still seeing the rise of case number because the infection chain is not broken yet," he told reporters.
"If we all have done what we can, the infection chain will soon be broken." - AFP