THE KOREA ECONOMIC DAILY, GLOBAL EDITION

South Korean startup Hudson AI is striving to break language barriers with its artificial intelligence-powered voice dubbing technology, enabling global viewers to fully enjoy a wide range of content produced by legacy media and individual creators from around the world.

“AI dubbing or TTS technology is nothing new. But almost no one has actively attempted to dub media content with this technology because the technology was not considered mature enough to be applied in the real world,” Shin Hyunjin, chief executive officer of Hudson AI, said in an interview with The Korea Economic Daily last week.

“We wowed the judges at a startup pitch hosted by SK Telecom in 2022 with the demonstration of our AI dubbing technology applied to a TV series, which sounded so natural.”

TTS is short for text-to-speech, a technology that reads written text in a synthetic voice.

Hudson AI has developed its proprietary “acting” TTS, which creates a replica of an actor's voice. Powered by machine learning, its technology can clone human voices to create lifelike speech with differing intonation patterns, emotion, rhythm and other key vocal characteristics, Shin said.

Just two months after its foundation, Hudson AI joined Korean telecom giant SK Telecom’s startup accelerator batch with its AI dubbing technology in August 2022.

TIMBR FOR INDIVIDUAL CONTENT CREATORS  

Hudson AI is currently carrying out closed global beta service testing for its content dubbing service called Timbr.

Using generative AI like ChatGPT, it first translates dialogue in a movie or a YouTube video into a desired language and creates a transcript that matches the original, native spoken language in length.

And then it uses voice conversion technology, like voice cloning, to create a replica of the original actors’ voices based on the transcript. The acting TTS technology enables AI to reflect actors’ emotions in the cloned voices, Shin said.

To perfect voice cloning, the company currently uses guiding voice actors to direct AI to generate more human-like voices.

The final step is to create immersive and expressive AI characters with its realistic lip-syncing technology that matches actors’ mouth movements to the spoken dialogue to ensure synchronization between the visual and auditory aspects of speech.

Timbr has been fine-tuned to be a more creator-friendly product from the company’s initial product called Seenchro, said Shin.

“We have made it easier to use to enable more people around the world to enjoy a variety of media content with our dubbing solution that can eliminate language barriers,” said Shin.

Creators can download this version, which is offered as a service-as-a-software (SaaS), to dub their content in multiple languages.

Hudson AI’s creator-friendly dubbing program comes after YouTube started offering creators a multi-language audio feature last year, Shin said, adding that the feature has helped creators increase their video viewership.

“Netflix hit the jackpot with dubbing services. There was no reason why YouTube wouldn't do the same,” said Shin. “Legacy media has proved dubbing contributed to increased audiences around the world. I expected YouTube to join the trend.”

Globally known YouTube creator MrBeast has seen a jump in his content viewership after dubbing his popular videos in different languages, encouraging other creators to follow suit, Hudson AI CEO said.

To access YouTube’s multi-language audio tracks, creators should partner with third-party dubbing providers like Hudson AI to offer their videos in different languages.  

Hudson AI has been working with dozens of Korean creators, such as GH’S with 23.2 million subscribers and FITVELY with 1.46 million subscribers, since April to help them meet global audiences in multiple languages, and many of those creators have seen a rise in viewers, Shin said.

“I expect the (AI dubbing) market for creators will grow faster than that for legacy media content,” said Shin.

PIONEER IN AI CONTENT DUBBING

The Hudson AI CEO said the startup is the only Korean company that offers a third-party AI dubbing service for media content.  

Global rivals, however, exist. And they are bigger.

US-based ElevenLabs, which was also founded in 2022, is best known for its browser-based speech generation app that can also create lifelike voices. It became a unicorn with a corporate value of over $1 billion early this year.

Other smaller global synthetic voice startups include Papercup, Deepdub, Acapela, Respeecher and Voice.ai, while Big Tech incumbents like Amazon, Microsoft and Google are also actively seeking to join the market.

The Korean startup is confident that its product can stand out among international rivals thanks to the technological outperformance of its AI audio dubbing tool, which has been used in legacy media content.

“Our service is better at acting, and we have moved faster and earlier than our rivals to meet the needs of creators,” said Shin.

Its dubbing service enables a legacy media company to save dubbing costs and time by more than 50% compared to a traditional movie dubbing project with human voice actors, Shin said.

YouTube creators can save up to 90% in terms of time and costs with the creator-friendly version, which does not require guiding voice actors, she added.   

ETHICAL AND SOCIAL ISSUES

Like its rivals, Hudson AI, however, faces pending ethical and social issues regarding its AI dubbing service.

Voice cloning services have been used to mimic famous people’s voices to deliver hateful messages or simulate family members’ voices for voice phishing.

Voice actors also raise concerns about the possible use of their voices without their knowledge. They are also worried that AI-generated voices will eventually replace their jobs, especially entry-level work, posing a threat to the overall voice-acting industry.

Shin said this is an area in which the government and stakeholders of the AI dubbing and voice-acting industries must come together to address.

“We use legitimately available voice data to train our AI,” said Shin. “We feel strongly responsible for preventing any harmful use of our voice dubbing technology like voice phishing.”

OCTOBER LAUNCH

Hudson AI is readying to launch the official service of Timbr in October, Shin said.

“We want to garner about 1 million global users for our service,” said Shin. “We will focus more on measures to improve our key performance indicators this year and then review the right time to seek more investment.”

It has raised a total of 4 billion won ($3 million) in investment since its inception. It has no imminent plans to raise new funds for now but may initiate a new funding round next year to cover costs to train AI and hire developers, Shin said.

Hudson AI’s dubbing program currently supports five languages -- Korean, English, Japanese, Latine American Spanish and Brazilian Portuguese. It plans to offer 20 to 25 languages in total by October.

“Hudson AI wants to be an indispensable partner of not only creators but also media content targeting the bigger global market,” said Shin.

Write to Sookyung Seo at skseo@hankyung.com

Jennifer Nicholson-Breen edited this article.