FoleyCrafter

Paid Web

What FoleyCrafter is

FoleyCrafter is an open-source framework that generates audio from video using text prompts. It aims to produce sound effects that correspond to what appears in footage and align with its timing. A semantic adapter uses video features to guide the sound content, while a temporal controller combines onset detection with timestamps for synchronization. Prompts can steer generation, including positive and negative instructions. The framework is built on a pretrained text-to-audio generator that remains fixed during training. Code and models are provided through GitHub, with both a Gradio interface and command-line inference. Setup instructions use a Conda environment and Git LFS for checkpoints; inference can download checkpoints automatically, with manual downloads also documented. Command-line options include prompts, negative prompts, random seed, temporal alignment, input and checkpoint paths, and output path. FoleyCrafter is available on the web, produces WAV downloads, and its linked hosted demo is paused. The project uses Apache-2.0 licensing, while commercial use requires checking the Auffusion license.

Who it suits

FoleyCrafter suits developers and creators producing synchronized sound effects for video, including cinema and gaming projects. It is intended for users comfortable with Conda, Git LFS, Gradio, or command-line inference.

What it does well

Text prompts control video-to-audio generation
Semantic and temporal video alignment
Gradio and command-line interfaces
Code and models available through GitHub
WAV output and Apache-2.0 project license

What to know first

Hosted Hugging Face demo is paused
Setup requires Conda and Git LFS
Commercial-use terms require checking Auffusion’s license
Web platform is the only listed platform

Verdict

FoleyCrafter provides controllable, synchronized video-to-audio generation with both interface and command-line options. Its setup is developer-oriented, and commercial users need to review the related Auffusion licensing.

Facts from FoleyCrafter's own pages

Base model: FoleyCrafter is built on a pretrained text-to-audio generator, and the project page says the T2A model remains fixed during the framework's training. · foleycrafter.github.io, 9 Oct 2026
Checkpoints: The repository says checkpoints download automatically when running `inference.py` and also documents manual downloads. · github.com, 9 Oct 2026
Code and models: The project provides its models and code through its GitHub repository. · github.com, 9 Oct 2026
Dependencies: The repository setup instructions use a Conda environment and Git LFS to download checkpoints. · github.com, 9 Oct 2026
Hosted demo: The linked Hugging Face Space is paused, and its page says users can ask its authors to restart it. · huggingface.co, 10 Oct 2026
How to run: The repository documents a Gradio interface launched with `python app.py --share` and command-line inference through `inference.py`. · github.com, 9 Oct 2026
Implementation: The framework has a Gradio interface and command-line inference usage. · github.com, 10 Oct 2026
Integrations: The project is built upon Auffusion, CondFoleyGen, and SpecVQGAN. · github.com, 10 Oct 2026
Intended use: The project describes sound effects as useful for cinema and gaming to enhance realism, impact, and emotional depth. · github.com, 10 Oct 2026
License: The repository lists an Apache-2.0 license and advises commercial users to check the Auffusion license as well. · github.com, 10 Oct 2026
Makers: The project page lists authors affiliated with Shanghai AI Laboratory and the Chinese University of Hong Kong, Shenzhen. · foleycrafter.github.io, 9 Oct 2026
Prompt control: Text prompts can be used to control video-to-audio generation according to user intent. · foleycrafter.github.io, 9 Oct 2026
Prompt controls: Text prompts support controllable video-to-audio generation, and the project demonstrates both positive and negative prompts. · github.com, 10 Oct 2026
Prompt options: The command-line interface includes prompt and negative-prompt options, a random seed, temporal alignment controls, input and checkpoint paths, and an output path. · github.com, 9 Oct 2026
Purpose: FoleyCrafter is a text-based video-to-audio generation framework that produces high-quality audio semantically relevant and temporally synchronized with input video. · foleycrafter.github.io, 10 Oct 2026
Semantic alignment: Its semantic adapter conditions audio generation on video features to produce sounds relevant to the visual content. · foleycrafter.github.io, 10 Oct 2026
Setup: The README provides a Conda environment setup and says model checkpoints can be downloaded automatically by running inference.py. · github.com, 10 Oct 2026
Sound generation: The framework uses a pretrained text-to-audio model to generate sound effects for video. · foleycrafter.github.io, 10 Oct 2026
Support: The repository README lists contact email addresses for Yiming Zhang, YiCheng Gu, and Yanhong Zeng. · github.com, 10 Oct 2026
Temporal alignment: Its temporal controller uses an onset detector and timestamp-based adapter to align audio with video. · foleycrafter.github.io, 10 Oct 2026
Temporal synchronization: Its temporal controller uses an onset detector and timestamp-based adapter for audio-video alignment. · foleycrafter.github.io, 9 Oct 2026
What it does: FoleyCrafter is a text-based video-to-audio generation framework that generates high-quality audio semantically relevant to and temporally synchronized with input videos. · foleycrafter.github.io, 9 Oct 2026

Where FoleyCrafter stands on Freedom251

AI Sound Effect Generators#29 of 29