Nazri said while those who attended Dr Zakir’s speeches described them as being mild, it should be no surprise that those of other faiths would not feel the same.
He said it was all about perception and this perception was based on what Dr Zakir had done in the past, especially when he had talked about other religions.
"I think if you want to talk about religion, then just talk about your own.
"When making comparisons between religions, you are bound to conclude that your religion is better," Nazri was quoted in an interview with a news portal.
Nazri said Muslims in Malaysia "did not need to be told that they are on the right track" as Malaysians were not new to Islam and had their own Islamic scholars to refer to.
It was not a question of freedom of speech, he said, but of ensuring that Malaysians were not being unnecessarily divided by someone who has no ties to Malaysia.
Nazri said that Dr Zakir would give his speeches and leave the country while Malaysians would be left behind with all the problems that he may have caused.
Dr Zakir created controversy after he was invited to talk at Universiti Teknikal Malaysia in Malacca on the topic of Similarities between Hinduism and Islam.
The police had banned him from the talk but Deputy Prime Minister Datuk Seri Dr Ahmad Zahid Hamidi later allowed Dr Zakir's talk after changes were made to the topic of the talk.