The head of the government's Muslim advisory committee, Ameer Ali, said on Friday that Howard's comments could fuel race riots like those at Sydney's Cronulla beach last December.
"We have already witnessed one incident in Sydney recently in Cronulla, I don't want these scenes to be repeated because when you antagonise the younger generation ... they are bound to react," Ali told local Sydney radio.
In December 2005, Howard said some Muslims were "raving on about jihad". Days later violence rocked Cronulla, when white Australian youths clashed with Lebanese-Australians.
In February, Howard again angered Australia's small Muslim community by saying he was concerned about extremist Muslim immigrants bent on jihad.
On Thursday, Howard told Sydney radio that migrants, particularly hardline Muslims, needed to conform to Australian values by learning English and treating women with respect.
His latest comments angered Muslims, but the prime minister said he would not apologise.
"I think they are missing the point and the point is that I don't care and the Australian people don't care where people come from," Howard told reporters on Friday.
"There's a small section of the Islamic population which is unwilling to integrate and I have said generally all migrants ... have to integrate," he said.
Almost a quarter of Australians were born overseas. Muslims have been in Australia for more than 200 years and make up 1.5 percent of the 20 million population.
Muslim leaders say their community feels under siege due to the U.S.-led war on terrorism, the Cronulla riot in 2005 and Howard's comments on Muslims.
Islamic leader Iktimal Hage-Ali said the prime minister was unfairly targeting the Muslim community.
"There's a whole lot of other ethnic communities whose parents, whose grandparents don't speak the English language and it's never a problem in the mainstream Australian community," she told reporters.
"Every tiny step that we take forward we have some politician come out and say something stupid and it takes us 10 steps back."