TV-AudioRemoverJoint Text‑Visual Guided Sound Removal with Multi‑Task Hard‑Mixture Curriculum

Xinyue Guo*,  Jianxuan Yang*†,  Daiguo Zhou*
Jiagao Hu,  Yuxuan Chen,  Fei Wang,  Jian Luan
MiLM Plus, Xiaomi Inc.   
*Equal contribution    †Corresponding author

Abstract

Visual object removal can eliminate a target from video frames, yet the target’s acoustic trace persists in the soundtrack, causing noticeable audio-visual inconsistency. Existing video inpainting models operate solely on pixels, while audio editing models, especially for the sound removal task, are typically text-driven and suffer from limited unimodal control, which is less effective than multimodal guidance that provides richer semantic grounding and temporal synchronization cues. We present Text-Visual Guided Sound Removal (\textbf{TV-AudioRemover}), a target sound removal framework that leverages natural-language instructions and optional visually edited videos to suppress the sound of the removed visual object from the original audio mixture. (1) To obtain high-quality training data, we design a pipeline to build a million-scale dataset of single-object audio-visual aligned samples, from which we synthesize mixture-target pairs tailored for model training. (2) To effectively exploit visual context and faithfully follow instruction intent, we augment the model architecture with task tokens, generalizable instruction modeling, and modality-specific global guidance. (3) We further adopt multi-task training to strengthen task-role understanding, together with a hard-mixture curriculum that employs semantically similar acoustic mixtures during fine-tuning to enhance fine-grained source discrimination. Beyond visually edited videos, the same selective sound-removal capability can support off-screen voice-over or narration cleanup, text-guided audio-only editing, and soundtrack-only removal of an in-frame sound from an otherwise unedited video.

Main Contributions

We propose TV-AudioRemover, a text‑visual guided sound removal model. The key contributions of TV-AudioRemover are as follows.

• A high-quality data construction pipeline.

It generates million-scale single-object audio-visual-aligned samples and synthesizes task-specific mixture-target pairs for training.

• A modality-decoupled audio editing architecture.

With task tokens and global modality guidance, the model effectively distinguishes preservable and suppressible audio under joint visual-textual conditions.

• A multi-task curriculum training strategy.

Combined multi-task supervision and two-stage curriculum learning enhance task understanding and fine-grained acoustic discrimination.

• AV-Remove-Bench.

An audio-visual target removal benchmark with rich scene diversity and broad acoustic coverage, equipped with two removal-specific metrics introduced in this work—Target Source Suppression Ratio (TSSR) and Preserved Source Fidelity (PSF)—and an MLLM-based evaluation protocol.

Experiments demonstrate that our method achieves state-of-the-art performance on both subjective and objective metrics for text-only-guided and text-visual-guided audio removal tasks.

Remove Task Samples

Demo1

Chopping Sound

Remove the chopping sound and all sound-producing objects that emit chopping sounds.

Expand

Demo2

Brick Knocking Sound

Remove the brick knocking sound and all sound-producing objects that emit brick knocking sounds.

Expand

Demo3

Woodpecker Pecking Sound

Remove the woodpecker pecking sounds and all sound-producing objects that emit woodpecker pecking sounds.

Expand

Demo4

Wolf Howling Sound

Remove the wolf howling sounds and all sound-producing objects that emit wolf howling sounds.

Expand

Demo5

Donkey Braying Sound

Remove the donkey braying sounds and all sound-producing objects that emit donkey braying sounds.

Expand

Demo6

Dog Barking Sound

Remove the dog barking sounds and all sound-producing objects that emit dog barking sounds.

Expand

Demo7

Guitar Sound

Remove the guitar sounds and all sound-producing objects that emit guitar sounds.

Expand

Demo8

Woman Speaking Sound

Remove the woman speaking sounds and all sound-producing objects that emit woman speaking sounds.

Expand

Demo9

Man Speaking Sound

Remove the man speaking sounds and all sound-producing objects that emit man speaking sounds.

Expand

Demo10

Bike Bell sound

Remove the bike bell sounds and all sound-producing objects that emit bike bell sounds.

Expand

Performance

The evaluation datasets consist of two parts: a removal subset of 50 samples from MMAE, an open-source text-guided audio editing benchmark, and our proposed text-visual-guided sound removal benchmark AV-Remove-Bench. We compare TV-AudioRemover with three categories of models on the AV-Remove-Bench: joint audio-visual editing models (AVI-Edit, InstructAV2AV); text-guided audio editing models (ZETA, Audio-Omni, UNISON); and video- and text-guided audio extraction model (SAM Audio, MMAudioSep). For the MMAE removal subset, we only evaluate the sound removal performance of audio-only editing models under pure text guidance.

objective1
objective2

TV-AudioRemover achieves state-of-the-art performance for text-only-guided and text-visual-guided audio removal tasks.

AV-Remove-Bench

We propose AV-Remove-Bench, a comprehensive audio-visual object removal benchmark with 77 samples. All samples are audio-visual clips of about six seconds, covering speech, music, and sound effects. Each sample falls into either a multi-sounding-object setting or a foreground-target-plus-stable-background setting, and the goal is to remove a specified sounding object from both the visual and audio streams. The data distribution is visualized below.

av-remove-bench

Citation

@misc{guo2026tvaudioremoverjointtextvisualguided,
  title        = {TV-AudioRemover: Joint Text-Visual Guided Sound Removal with Multi-Task Hard-Mixture Curriculum},
  author       = {Xinyue Guo and Jianxuan Yang and Daiguo Zhou and Jiagao Hu and Yuxuan Chen and Fei Wang and Jian Luan},
  year         = {2026},
  eprint       = {2609.25864},
  archivePrefix = {arXiv},
  primaryClass = {cs.MM},
  url          = {https://arxiv.org/abs/2609.25864}
}

Data & Acknowledgments

Data sources

We gratefully acknowledge the third-party datasets used for research, training-data construction, and evaluation.

VGGSound [1] CC BY 4.0 AudioSet CC BY 4.0 AVSpeech [2] CC BY 4.0 Condensed Movies CC BY 4.0 WavCaps Research only ACAVCaps CC BY-NC 4.0 MUSIC-AVQA CC BY-NC 4.0 MUSIC-Duet [3] AVSBench CC BY-NC 4.0

Usage note

All third-party resources are used for academic and non-commercial demonstration purposes only. Please consult the original dataset pages for their complete licenses and terms.

Acknowledgments

This project is inspired by stable-audio-tools, MMAudio, MMAudioSep, Make-An-Audio 2, Synchformer, and BigVGAN. We thank their authors for making these resources available.

Dataset citations

Note: The references below credit datasets used by this project. They are not citations to TV-AudioRemover; the project citation will be added above after the arXiv release.

[1] VGGSound
@inproceedings{chen2020vggsound,
  title     = {VGGSound: A Large-Scale Audio-Visual Dataset},
  author    = {Chen, Honglie and Xie, Weidi and Vedaldi, Andrea and Zisserman, Andrew},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  year      = {2020}
}
[2] AVSpeech
@article{ephrat2018looking,
  title   = {Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separation},
  author  = {Ephrat, Ariel and Mosseri, Inbar and Lang, Oran and Dekel, Tali and Wilson, Kevin and Hassidim, Avinatan and Freeman, William T. and Rubinstein, Michael},
  journal = {ACM Transactions on Graphics},
  year    = {2018}
}
[3] MUSIC-Duet / Sound of Pixels
@inproceedings{zhao2018sound,
  title     = {The Sound of Pixels},
  author    = {Zhao, Hang and Gan, Chuang and Rouditchenko, Andrew and Vondrick, Carl and McDermott, Josh and Torralba, Antonio},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2018}
}