The first AI that uses both what it sees and hears to hold a real conversation in a noisy room.

AV-Dialog: Spoken Dialogue Models with
Audio-Visual Input

Tuochao Chen^1,2 Bandhav Veluri¹ Hongyu Gong² Shyamnath Gollakota¹

¹University of Washington ²Meta AI Research

(ACL 2026 Main conference) The 64th Annual Meeting of the Association for Computational Linguistics

Paper arXiv Code （Coming Soon） Demo Videos

Demo Videos

	Interference Speech1	Interference Speech2	Background Noise	Interference Speech3
Prior (SE+Moshi)	Demo1: Audio-only baseline with interfering speakers	Demo2: Audio-only baseline with interfering speakers	Demo3: Audio-only baseline with background noise	Demo4: Audio-only baseline with interfering speakers
Ours (AV-Dialog)	Demo1: AV-Dialog with interfering speakers	Demo2: AV-Dialog with interfering speakers	Demo3: AV-Dialog with background noise	Demo4: AV-Dialog with interfering speakers

AV-Dialog teaser figure — AV-Dialog uses both audio and visual cues to track the target speaker, predict turn-taking, and generate coherent responses in noisy, multi-speaker environments.

67% → 31.7%
WER under interference
+1.66 N-MOS
over SE+Moshi baseline
+1.90 H-MOS
over SE+Moshi baseline

Abstract

Dialogue models falter in noisy, multi-speaker environments, often producing irrelevant responses and awkward turn-taking. We present AV-Dialog, the first multimodal dialog framework that uses both audio and visual cues to track the target speaker, predict turn-taking, and generate coherent responses. By combining acoustic tokenization with multi-task, multi-stage training on monadic, synthetic, and real audio-visual dialogue datasets, AV-Dialog achieves robust streaming transcription, semantically grounded turn-boundary detection and accurate responses, resulting in a natural conversational flow. Experiments show that AV-Dialog outperforms audio-only models under interference, reducing transcription errors, improving turn-taking prediction, and enhancing human-rated dialogue quality. These results highlight the power of seeing as well as hearing for speaker-aware interaction, paving the way for spoken dialogue agents that perform robustly in real-world, noisy environments.

Key Contributions

🎯

First Audio-Visual Dialogue Framework Built on LLaMA3-8B, AV-Dialog processes audio and video in a streaming manner (40ms chunks) to extract target speaker text tokens under interference and predict turn-change tokens. We explore both dual and unified architectures.
🔊

Acoustic Tokens for Noisy Multi-Speaker Settings We use general-purpose Descript Audio Codec (DAC) tokens instead of semantic tokenizers trained on single-speaker speech. Acoustic tokens preserve both semantic and raw acoustic information, enabling inherent speaker differentiation — reducing WER under strong interference from 67% to 31.7%.
📋

Multi-Task, Multi-Stage Training Recipe Stage 1 trains on text prediction, ASR, AVSR, and audio captioning to strengthen audio-visual understanding. Stage 2 fine-tunes on real conversational data with synthetic mixture augmentation to simulate noisy environments (SNR between −8 dB and 8 dB).

Method

AV-Dialog processes multi-stream inputs at each timestep: a continuous visual stream from AV-HuBERT and 16 audio streams from the DAC tokenizer. The model outputs a streaming transcription of user speech and turn-taking event tokens. When a turn-taking token is detected, the system switches from LISTENING to SPEAKING mode and generates a response.

AV-Dialog model pipeline — **Model architecture.** AV-Dialog integrates a visual stream (AV-HuBERT) and 16 acoustic token streams (DAC) into LLaMA3-8B. The dual architecture separates AV understanding from response generation; the unified architecture handles all tasks in a single model.

Results

Table 1. Streaming AVSR — Word Error Rate (%) ↓
Model	Dataset	Clean	BG Noise	Interference
Auto-AVSR	VoxCeleb2	26.8	48.2	71.8
AV-Dialog (Ours)	VoxCeleb2	17.4	35.6	38.8
Auto-AVSR	LRS2	15.8	34.0	60.0
AV-Dialog (Ours)	LRS2	9.53	24.0	28.4

Table 2. Turn-Taking & Dialogue Quality
Model	Response Ratio ↑	FTO Error ↓	N-MOS ↑	H-MOS ↑
SE + Moshi	54.0%	3.48	2.46	2.12
AV-Dialog Dual+ICL (Ours)	74.5%	1.86	4.12	4.02
AV-Dialog Unified (Ours)	68.1%	1.68	3.67	3.09
Ground Truth	—	—	3.98	3.62

Human evaluation on N=24 subjects. N-MOS: naturalness mean opinion score; H-MOS: helpfulness mean opinion score. Visual input improves turn-taking accuracy from 54% to 79% under interfering speakers (+13% gain). AV-Dialog achieves +1.66 N-MOS and +1.90 H-MOS over SE+Moshi.