DreamTalk

Paid Self-hosted

What DreamTalk is

DreamTalk is a free, open-source framework for generating expressive talking-head videos from audio. Its system combines a denoising network, a style-aware lip expert, and a style predictor that can infer speaking style from audio without an expression reference video or text. Style can be adjusted using classifier-free guidance scale and style-code interpolation. Inference accepts WAV, MP3, M4A, or MP4 files with sound, a portrait image, and 3DMM parameter sequences for speaking style and head pose. Portraits should be larger than 256×256 pixels; they are cropped to 256×256, and frontal images are described as usually producing satisfactory results. Motion references are extracted from 25 FPS video. The inference script saves MP4 output, and audio longer than the configured duration limit is truncated. Examples cover seven languages, songs, and noisy audio. Setup instructions specify Python 3.7, PyTorch 1.8.0, and CUDA 11.1 packages; CPU execution is also possible. Public checkpoint downloads have ceased, and checkpoints may be requested for academic research use. The repository limits intended use to research and non-commercial purposes.

Who it suits

DreamTalk is aimed at researchers and others exploring audio-driven talking-head generation for non-commercial use. It may suit users able to work with its stated Python and machine-learning setup.

What it does well

Free and open source.
Predicts speaking style from audio without expression video or text.
Accepts WAV, MP3, M4A, and MP4 audio.
CPU execution is supported with the stated argument.

What to know first

Public checkpoint downloads have ceased.
Checkpoints may be requested for academic research use.
Repository states research and non-commercial use only.
Portraits are cropped to 256×256 pixels.

Limits

The maximum generation duration is configurable, and audio beyond that setting is truncated. Public checkpoint downloads have ceased; checkpoints may be requested by email for academic research use.

Getting started

Open the DreamTalk project website
Prepare a Python 3.7 environment with the listed PyTorch 1.8.0 and CUDA 11.1 packages
Provide a WAV, MP3, M4A, or MP4 audio file with sound
Provide a portrait larger than 256×256 pixels
Add 3DMM sequences for speaking style and head pose
Run inference and retrieve the MP4 from the output_video folder

Verdict

DreamTalk offers audio-driven expression and talking-head generation with several supported input formats and language examples. Checkpoint access and its stated non-commercial research scope are key limitations.

Questions

Is DreamTalk free?

Yes. It is listed as free and open source.

Which platforms does DreamTalk support?

It is self-hosted. The setup instructions target Python 3.7 with PyTorch 1.8.0 and CUDA 11.1, and CPU use is available with --device=cpu.

What audio formats can it use?

Inference accepts WAV, MP3, M4A, and MP4 files containing sound.

What portrait and motion inputs are required?

The portrait should be larger than 256×256 pixels and is cropped to 256×256. Speaking-style and head-pose references use 3DMM parameter sequences extracted from 25 FPS video.

Can DreamTalk be used commercially?

The repository says the method is intended for research and non-commercial use only.

Facts from DreamTalk's own pages

Checkpoint access: Public checkpoint downloads have been discontinued; the README asks researchers to request checkpoints by email and states that doing so implies consent to use them solely for academic research. · github.com, 9 Oct 2026
CPU support: The repository says the program can run on CPU by adding the --device=cpu argument. · github.com, 4 Oct 2026
Duration limit: The max_gen_len setting controls the maximum generated video duration in seconds, and longer input audio is truncated. · github.com, 9 Oct 2026
Expressions: Its style predictor predicts a target expression from audio without requiring an expression reference video or text. · dreamtalk-project.github.io, 9 Oct 2026
Founded: 2023 · dreamtalk-project.github.io, 28 Sept 2026
Generation approach: The framework combines a denoising network, a style-aware lip expert, and a style predictor. · dreamtalk-project.github.io, 4 Oct 2026
How it works: The framework combines a denoising network, a style-aware lip expert, and a style predictor. · dreamtalk-project.github.io, 9 Oct 2026
Inference inputs: Inference accepts WAV, MP3, M4A, or MP4 audio, a portrait image, and reference 3DMM sequences for speaking style and head pose. · github.com, 9 Oct 2026
Input audio: The inference instructions list WAV, MP3, M4A, and MP4 files with sound as compatible audio inputs. · github.com, 4 Oct 2026
Input coverage: The project reports generalization across songs, noisy audio, out-of-domain portraits, and speech in Chinese, French, German, Italian, Japanese, Korean, and Spanish. · dreamtalk-project.github.io, 9 Oct 2026
Languages and audio: The project page lists speech examples in Chinese, French, German, Italian, Japanese, Korean, and Spanish, as well as songs and noisy audio. · dreamtalk-project.github.io, 4 Oct 2026
License: The repository includes an MIT license, while its README says the method is intended for research and non-commercial use only. · github.com, 9 Oct 2026
Maintainers: The project page lists authors affiliated with Tsinghua University, Alibaba Group, and Huazhong University of Science and Technology. · dreamtalk-project.github.io, 9 Oct 2026
Motion references: Speaking-style and head-pose references use 3DMM parameter sequences extracted from videos, and the stated video frame rate is 25 FPS. · github.com, 4 Oct 2026
Output: The inference script saves the generated video as an MP4 file in the output_video folder. · github.com, 4 Oct 2026
Portrait requirements: The input portrait should be larger than 256×256 pixels; the instructions say frontal portraits usually produce satisfactory results and are cropped to 256×256. · github.com, 4 Oct 2026
Purpose: DreamTalk is a diffusion-based framework for generating expressive talking-head videos from audio. · dreamtalk-project.github.io, 9 Oct 2026
Resolution: The README says resolution was not considered in the initial design and describes optional super-resolution methods with tradeoffs in speed, temporal consistency, or facial emotion intensity. · github.com, 9 Oct 2026
Setup: The repository provides inference code and installation instructions for a Python 3.7 environment with PyTorch 1.8.0 and CUDA 11.1 packages. · github.com, 9 Oct 2026
Speaking styles: The project demonstrates speaking-style manipulation by adjusting classifier-free guidance and interpolating style codes. · dreamtalk-project.github.io, 9 Oct 2026
Style prediction: Its style predictor can infer a target speaking style from audio without an expression reference video or text. · dreamtalk-project.github.io, 4 Oct 2026
Support: The project page invites researchers interested in joining the team to email a resume to [email protected]. · dreamtalk-project.github.io, 9 Oct 2026
Use restriction: The repository states that the method is intended for research and non-commercial use only. · github.com, 4 Oct 2026
What it does: DreamTalk is a diffusion-based framework for generating expressive talking-head videos from audio. · dreamtalk-project.github.io, 4 Oct 2026

Where DreamTalk stands on Freedom251

AI Video Lip Sync Tools#24 of 28