TV-AudioRemoverJoint Text‑Visual Guided Sound Removal with Multi‑Task Hard‑Mixture Curriculum

Xinyue Guo*,  Jianxuan Yang*†,  Daiguo Zhou*
Jiagao Hu,  Yuxuan Chen,  Fei Wang,  Jian Luan
MiLM Plus, Xiaomi Inc.   
*Equal contribution    †Corresponding author
arXiv Code Model (Coming Soon)

Abstract

Visual object removal can eliminate a target from video frames, yet its acoustic trace persists in the soundtrack, causing obvious audio-visual inconsistency. Existing video inpainting models operate solely on pixels, while audio editing models, especially for the sound removal task, are typically driven by text and therefore rely on limited single-modal control, which is less effective than multimodal guidance that provides stronger semantic grounding and temporal synchronization cues. We present Text‑Visual Guided Sound Removal (TV-AudioRemover), a target sound removal framework that leverages the visually edited video together with a natural-language instruction to suppress the sound associated with the removed visual object from the original audio mixture. (1) To acquire high-quality training data, we devise a pipeline to construct a million-scale dataset of single-object audio-visual aligned samples, from which we synthesize mixture-target pairs customized for model training. (2) To effectively leverage visual context and follow instruction intent, we augment the model architecture with task tokens, generalizable instruction modeling, and modality-specific global guidance. (3) We further adopt multi-task training to strengthen task-role comprehension, and employ a hard-mixture curriculum that leverages semantically similar acoustic mixtures during fine-tuning to enhance fine-grained source discrimination. Beyond visually edited videos, the same selective sound-removal capability can support off-screen voice-over or narration cleanup, text-guided audio-only editing, and soundtrack-only removal of an in-frame sound from an otherwise unedited video.

fig1
Overview of the Text-Visual Guided Target-Sound Removal framework TV-AudioRemover.

Main Contributions

We propose TV-AudioRemover, a text‑visual guided sound removal model. The key contributions of TV-AudioRemover are as follows.

• A high-quality data construction pipeline.

It generates million-scale single-object audio-video aligned samples and synthesizes task-specific mixture-target pairs for target removal from these samples.

• A modality-decoupled audio editing architecture.

With task tokens and global modal guidance, the model effectively distinguishes preservable and suppressible audio under joint visual-textual conditions.

• A multi-task curriculum training strategy.

Combined multi-task supervision and two-stage curriculum learning enhances task understanding and fine-grained acoustic discrimination capacity.

• AV-Remove-Bench.

The first audio-visual target removal benchmark with comprehensive scene diversity and broad acoustic coverage, equipped with two removal-specific metrics introduced in this work—Target Source Suppression Ratio (TSSR) and Preserved Source Fidelity (PSF)—and an MLLM-based evaluation protocol.

Extensive experiments demonstrate that TV-AudioRemover achieves state-of-the-art performance on both subjective and objective metrics.

fig2
Overview of the TV-AudioRemover model architecture.

Remove Task Samples

Demo1

Chopping Sound

Remove the chopping sound and all sound-producing objects that emit chopping sounds.

Expand

Demo2

Brick Knocking Sound

Remove the brick knocking sound and all sound-producing objects that emit brick knocking sounds.

Expand

Demo3

Woodpecker Pecking Sound

Remove the woodpecker pecking sounds and all sound-producing objects that emit woodpecker pecking sounds.

Expand

Demo4

Wolf Howling Sound

Remove the wolf howling sounds and all sound-producing objects that emit wolf howling sounds.

Expand

Demo5

Donkey Braying Sound

Remove the donkey braying sounds and all sound-producing objects that emit donkey braying sounds.

Expand

Demo6

Dog Barking Sound

Remove the dog barking sounds and all sound-producing objects that emit dog barking sounds.

Expand

Demo7

Guitar Sound

Remove the guitar sounds and all sound-producing objects that emit guitar sounds.

Expand

Demo8

Woman Speaking Sound

Remove the woman speaking sounds and all sound-producing objects that emit woman speaking sounds.

Expand

Demo9

Man Speaking Sound

Remove the man speaking sounds and all sound-producing objects that emit man speaking sounds.

Expand

Demo10

Bike Bell sound

Remove the bike bell sounds and all sound-producing objects that emit bike bell sounds.

Expand

Performance

We compare TV-AudioRemover with three categories of models on the AV-Remove-Bench: joint audio-visual editing models (AVI-Edit, InstructAV2AV); text-guided audio editing models (ZETA, Audio-Omni, UNISON); and video- and text-guided audio extraction model (SAM Audio).

objective

TV-AudioRemover achieves state-of-the-art performance across almost all objective metrics.

AV-Remove-Bench

We propose AV-Remove-Bench, a comprehensive audio-visual object removal benchmark with 77 samples. All samples are audio-visual clips of about six seconds, covering speech, music, and sound effects. Each sample falls into either a multi-sounding-object setting or a foreground-target-plus-stable-background setting, and the goal is to remove a specified sounding object from both the visual and audio streams.

Citation 

@misc{guo2026tvaudioremoverjointtextvisualguided,
  title        = {TV-AudioRemover: Joint Text-Visual Guided Sound Removal with Multi-Task Hard-Mixture Curriculum},
  author       = {Xinyue Guo and Jianxuan Yang and Daiguo Zhou and Jiagao Hu and Yuxuan Chen and Fei Wang and Jian Luan},
  year         = {2026},
  eprint       = {2609.25864},
  archivePrefix = {arXiv},
  primaryClass = {cs.MM},
  url          = {https://arxiv.org/abs/2609.25864}
}

Data & Acknowledgments

Data sources

We gratefully acknowledge the third-party datasets used for research, training-data construction, and evaluation.

VGGSound [1] CC BY 4.0 AudioSet CC BY 4.0 AVSpeech [2] CC BY 4.0 Condensed Movies CC BY 4.0 WavCaps Research only ACAVCaps CC BY-NC 4.0 MUSIC-AVQA CC BY-NC 4.0 MUSIC-Duet [3] AVSBench CC BY-NC 4.0

Usage note

All third-party resources are used for academic and non-commercial demonstration purposes only. Please consult the original dataset pages for their complete licenses and terms.

Acknowledgments

This project is inspired by stable-audio-tools, MMAudio, MMAudioSep, Make-An-Audio 2, Synchformer, and BigVGAN. We thank their authors for making these resources available.

Dataset citations

Note: The references below credit datasets used by this project. They are not citations to TV-AudioRemover; the project citation will be added above after the arXiv release.

[1] VGGSound
@inproceedings{chen2020vggsound,
  title     = {VGGSound: A Large-Scale Audio-Visual Dataset},
  author    = {Chen, Honglie and Xie, Weidi and Vedaldi, Andrea and Zisserman, Andrew},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  year      = {2020}
}
[2] AVSpeech
@article{ephrat2018looking,
  title   = {Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separation},
  author  = {Ephrat, Ariel and Mosseri, Inbar and Lang, Oran and Dekel, Tali and Wilson, Kevin and Hassidim, Avinatan and Freeman, William T. and Rubinstein, Michael},
  journal = {ACM Transactions on Graphics},
  year    = {2018}
}
[3] MUSIC-Duet / Sound of Pixels
@inproceedings{zhao2018sound,
  title     = {The Sound of Pixels},
  author    = {Zhao, Hang and Gan, Chuang and Rouditchenko, Andrew and Vondrick, Carl and McDermott, Josh and Torralba, Antonio},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2018}
}