Is that Tom Hanks speaking in Japanese? No, it’s just AI

A file photo of Hanks and his wife Rita Wilson on the red carpet in London. The AI tool studies how actors move their mouths and swaps the movements out according to the dubbed words in different languages, making it seem like Hanks can speak Japanese or Nicholson is fluent in French. — AFP

LOS ANGELES: Bad lip-syncing in dubbing and subtitles can put off audiences and hurt box office takings of foreign films.

AI may be about to change all that.

Startup Flawless AI, co-founded by film director Scott Mann, has a tool that it says can accurately recreate lip sync in dubbing without altering the performance of the actors.

The tool studies how actors move their mouths and swaps the movements out according to the dubbed words in different languages, making it seem like Tom Hanks can speak Japanese or Jack Nicholson is fluent in French.

Mann was inspired to come up with the tool when he saw how dubbing affected the narrative cohesion of his 2015 film Heist, starring Robert De Niro.

“I hate dubbing as it stands,” he told Reuters. “You have to change so many things to try and catch sync. You’re changing words filmmakers and performers have thought about so deeply. They’re thrown out to find a different word that fits, but it never really does.”

Mann decided to do something about this conspicuous mismatch. After some research, he discovered a white paper by Christian Theobalt of the Max Planck Institute for Informatics, laying out a new approach that uses artificial intelligence to recreate photo-real human visual effects.

The tool worked to translate a De Niro scene in Heist.

“It’s able to essentially take an ‘ooh’ sound from De Niro 20 minutes earlier and place that in a different moment in the film,” explained Mann. “It measures at the same time and blends it so that the performance is the same but it’s a different mouth movement.”

Flawless AI is now working with producers and studios to integrate the technology into post-production. The first films using this tool are around a year away.

Mann believes this technology will be a game-changer, helping to eradicate foreign-language categories at award shows and creating a more inclusive film industry with a real global stage.

“From a filmmaking point of view, you're going to see the rise of a much more diverse range of stars,” he said.

There might be a dissonance and a focus on the AI technology in the beginning when it is introduced, but eventually Mann hopes its use will be seamless.

“When it’s a common thing, we’re not going to think about it anymore and we’ll just enjoy the movies,” he said. – Reuters

Article type: free
User access status:
Join our Telegram channel to get our Evening Alerts and breaking news highlights

Dubbing , lip syncing


Next In Tech News

EV startup Rivian announces $2.5 billion funding round led by Amazon, Ford
Vodafone plans to launch additional share buy-back programmes this month
Electric air taxi startup Joby to add LinkedIn co-founder, Google exec to board
Exclusive-Facebook's Kustomer deal set to face EU antitrust investigation
Alphabet to launch robotics firm Intrinsic under its other bets unit
S.Africa's Transnet says has identified and isolated the source of IT disruption
As Spain swelters and COVID cases grow, pool renting app thrives
Explainer-Understanding structural EV batteries
White House sees YouTube, Facebook as 'Judge, Jury & Executioner' on vaccine misinformation
Trying to fight Covid misinformation, White House risks falling into same feedback loop

Stories You'll Enjoy