Lip Forcing

Free plan LinuxSelf-hosted

What Lip Forcing is

Lip Forcing is a free, self-hosted method for generating lip-synchronized video from a reference video and speech audio. Its inference pipeline detects and aligns faces, processes them as 512×512 crops, and composites the generated result into the video. The documented command accepts a reference video and speech audio, then writes an output video.

Its causal student models generate each chunk in two denoising steps, without classifier-free guidance during inference. Streaming inference processes chunks as they arrive; the project reports sub-millisecond time to first frame and GPU memory use that stays constant as clip length grows under its stated setup. Reported performance includes 31 FPS for the 1.3B student on a single H100 GPU, and the 14B student running 39.8 times faster than its teacher at comparable reference fidelity.

The 14B student checkpoint is available as a merged, self-contained file. The 1.3B weights are listed as coming soon. Inference needs external components for video and audio processing, text encoding or precomputed embeddings, decoding, and mouth masking. The project reports single-GPU inference, with about 37 GB peak VRAM for the 14B model using precomputed text embeddings, or about 50 GB with runtime text encoding. The code repository states that Lip Forcing is released under the Apache License 2.0.

Who it suits

Lip Forcing suits developers and researchers who can run self-hosted video generation on Linux and provide the required external model components. It may fit projects that need streaming lip synchronization from a reference video and speech audio, provided the available GPU can meet the 14B model’s reported memory requirements.

What it does well

Free to use as a self-hosted method.
Accepts reference video and speech audio as inputs.
Automatically aligns faces and composites the generated result.
Generates each chunk in two denoising steps.
Reports sub-millisecond time to first frame for streaming inference.
Code is released under Apache License 2.0.

What to know first

Inference requires several external model components.
The 1.3B student weights are listed as coming soon.
The 14B model uses about 37–50 GB peak VRAM.
No Hugging Face Inference Provider deploys the model.

Getting started

Open the project repository at https://github.com/cvlab-kaist/LipForcing.
Prepare a Linux single-GPU environment; the documented setup was tested on Ubuntu 24.04.
Obtain the required external components, including the Wan VAE, wav2vec audio encoder, text encoder or precomputed embeddings, TAEW decoder, and mouth mask.
Use the available 14B student checkpoint; 1.3B weights are listed as coming soon.
Run the documented inference command with a reference video and speech audio to write an output video.

Verdict

Choose Lip Forcing if you need self-hosted video-to-video lip synchronization and can assemble its external components and GPU setup. Look elsewhere if you need a hosted inference service or the 1.3B checkpoint, whose weights are listed as coming soon.

Questions

How much does Lip Forcing cost?

It is free, with a free plan.

Where can I run it?

The listed platforms are Linux and self-hosted. The documented setup was tested on Ubuntu 24.04, and inference runs on a single GPU.

What model weights are available?

The release includes a 14B student checkpoint as a merged, self-contained file. The 1.3B student weights are listed as coming soon.

What does inference require?

It takes a reference video and speech audio and requires external components including the Wan VAE, wav2vec audio encoder, a text encoder or precomputed embeddings, TAEW decoder, and mouth mask.

Is Lip Forcing open source?

The code repository states that it is released under the Apache License 2.0.

Is a hosted inference service available?

The checkpoint page says the model is not deployed by any Hugging Face Inference Provider.

Facts from Lip Forcing's own pages

Automatic processing: The inference pipeline automatically detects and aligns faces to 512×512 crops and pastes the generated result back into the video. · github.com, 4 Oct 2026
Availability: The Hugging Face model card says the model is not deployed by an Inference Provider. · huggingface.co, 8 Oct 2026
Dependencies: Inference requires external components including the Wan VAE, wav2vec audio encoder, text encoder or precomputed embeddings, decoder, and mouth mask. · github.com, 8 Oct 2026
External components: Inference also requires external components including the Wan VAE, wav2vec audio encoder, UMT5-XXL text encoder or precomputed embeddings, TAEW decoder, and mouth mask. · github.com, 4 Oct 2026
Generation: Its causal student models generate each chunk in two denoising steps without classifier-free guidance at inference. · github.com, 4 Oct 2026
Hardware: The repository reports testing on Ubuntu 24.04 with an NVIDIA H200 and says inference runs on a single GPU. · github.com, 4 Oct 2026
Hosting: The checkpoint page states that the model is not deployed by any Hugging Face Inference Provider. · huggingface.co, 4 Oct 2026
Inference setup: The documented inference command takes a reference video and speech audio and writes an output video. · github.com, 4 Oct 2026
Input handling: Inference accepts a reference video and speech audio, and automatically detects, aligns, and composites the face. · github.com, 8 Oct 2026
Integrations: The checkpoint is hosted on Hugging Face, and the README also identifies external model components from Wan, wav2vec2, TAEW, LatentSync, and SyncNet sources. · github.com, 8 Oct 2026
License: The repository states that Lip Forcing is released under the Apache License 2.0. · github.com, 8 Oct 2026
Memory requirement: The 14B model uses about 37 GB peak GPU memory with precomputed text embeddings, or about 50 GB when encoding text at runtime. · github.com, 8 Oct 2026
Model sizes: The release includes a 14B student checkpoint; the 1.3B student weights are listed as coming soon. · github.com, 8 Oct 2026
Performance: The project reports that the 1.3B student reaches 31 FPS and the 14B student runs 39.8 times faster than its teacher at comparable reference fidelity. · cvlab-kaist.github.io, 4 Oct 2026
Platform: The documented installation was tested on Ubuntu 24.04 with Python 3.12, PyTorch 2.10.0, CUDA 12.8, and an NVIDIA H200; inference runs on a single GPU. · github.com, 8 Oct 2026
Project affiliation: The project lists KAIST AI and AIPARK affiliations for its authors. · github.com, 4 Oct 2026
Purpose: Lip Forcing is an autoregressive diffusion method for video-to-video lip synchronization that animates a reference video from audio. · github.com, 8 Oct 2026
Real-time performance: The project reports 31 FPS for the 1.3B student on a single H100 GPU. · cvlab-kaist.github.io, 8 Oct 2026
Resource limit: For the 14B model, the repository reports about 37 GB peak VRAM with precomputed text embeddings and about 50 GB with runtime text encoding. · github.com, 4 Oct 2026
Roadmap: The repository roadmap lists a Gradio or Hugging Face Space demo as planned. · github.com, 8 Oct 2026
Streaming: Streaming inference produces frames before the input clip finishes and reports sub-millisecond time-to-first-frame. · github.com, 8 Oct 2026
Training: Training is documented as two stages: Diffusion-Forcing initialization followed by Self-Forcing DMD with a SyncNet reward. · github.com, 8 Oct 2026
Two-step generation: Its causal student models generate each chunk in two denoising steps without inference-time classifier-free guidance. · github.com, 8 Oct 2026
Weights: The released 14B student checkpoint is a merged, self-contained file, while the 1.3B student weights are listed as coming soon. · github.com, 4 Oct 2026

Where Lip Forcing stands on Freedom251

AI Video Lip Sync Tools#17 of 28