Projected Audio Tokens Gain Retrieval and Lose Grounded Generation in Multimodal LLMs

Abstract

We freeze Large Language and Vision Assistant 1.6 (LLaVA-1.6), a multimodal large language model (LLM) with a Mistral-7B language backbone, together with five frozen audio encoders. Each encoder supplies either a raw token, formed by padding or truncating to 1,024 dimensions, or a projected token learned in contrastive language-image pretraining (CLIP) space, and the token budget k is the number of retained visual patch tokens. On a 1,006-clip evaluation pool, projection raises audio-to-video recall at one (R@1) for every encoder, including ImageBind from 0.10% to 15.7%, although contrastive language-audio pretraining (CLAP) and Whisper R@1 values are lower bounds for their original configurations. At k=150, the CLAP-based reference-free grounding score (B1) decreases from 0.113–0.132 to 0.062–0.088, a 23–53% relative reduction, and the CLAP result is read with source recall (B3) because the encoder and scorer share a checkpoint family. B3 decreases from 0.161–0.185 to 0.090–0.112, a 31–48% relative reduction, while caption-to-frame CLIP similarity (B2) decreases for four encoders and is unchanged at printed precision for Wav2CLIP. Across 3,568 disjoint paired clips, projection increases centered kernel alignment with visual embeddings in all five encoders, while the norm outside the visual principal subspace and nearest-neighbor overlap with the raw audio graph decrease. The noncausal comparisons show that raw-token bilingual evaluation understudy (BLEU) exceeds projected-token BLEU in nine of ten audio-aware-reference cells and that raw-over-projected generation scores predominate across five frozen backbones from 7B to 34B wherever raw-token generation is stable.

Controlled Audio-Token Substitution

off-screen sound projected token generic caption video match raw token source caption

SoundCLIP compares raw and projected tokens from AudioCLIP, Wav2CLIP, ImageBind, CLAP, and Whisper while keeping the audio encoders and LLaVA-1.6-Mistral-7B frozen. The raw token is padded or truncated to 1,024 dimensions. The projected token uses a three-layer MLP with layer normalization and GELU activations, trained to align the audio token with the paired CLIP visual embedding.

audio encoder projected token raw token visual patches frozen LLaVA video retrieval

AVE-2 Dataset

The evaluation pool is drawn from Audio-Visual Event 2 (AVE-2). The corrected local rebuild contains 570,138 three-second audio-video segments sourced from AudioSet. The paper uses AVE-2 for evaluation and geometry analysis. Its active-source fields support source recall (B3); machine-generated alignment scores are not used as ground truth.

from datasets import load_dataset
dataset = load_dataset("ali-vosoughi/ave-2")

Interactive Demo

These archived examples illustrate raw and projected captions from the earlier project page. They are not results from the current evaluation pool. Click an encoder mode to inspect an earlier generated caption; bracketed ellipses mark omitted passages.

Key Findings

Within the tested configurations, projection raises audio-to-video retrieval and lowers audio grounding and source recall. The main comparison uses frozen LLaVA-1.6-Mistral-7B.

Retrieval and Grounded Generation

At k=150, B1 decreases by 23–53% and B3 decreases by 31–48% relative to raw tokens across all five encoders. B2 decreases for four encoders and is unchanged at printed precision for Wav2CLIP.

B1 is caption-to-audio CLAP similarity, B2 is caption-to-middle-frame CLIP similarity, and B3 is semantic recall of active sound sources. CLAP and Whisper R@1 values are lower bounds for their original encoder configurations. Interpret CLAP B1 with B3 because the encoder and scorer share a checkpoint family.

Representation Geometry

Across 3,568 disjoint paired clips, projection raises centered kernel alignment with visual embeddings and effective rank for every encoder. Orthogonal residual and nearest-neighbor overlap with the raw audio graph decrease. These comparisons do not identify a causal mechanism.

Scope and Backbone Stability

Raw-over-projected generation scores predominate across five frozen backbones from 7B to 34B wherever raw-token generation is stable. Raw-token BLEU exceeds projected-token BLEU in nine of ten audio-aware-reference cells. The study is limited to the tested encoders, backbones, and evaluation pool.

Code and Resources

Read the current paper and explore the project repository and AVE-2 documentation.

Citation

If you use SoundCLIP or the AVE-2 dataset in your research, please cite our paper:

@unpublished{vosoughi2026soundclip,
  title={{Projected Audio Tokens Gain Retrieval and Lose Grounded Generation in Multimodal LLMs}},
  author={Vosoughi, Ali and Bi, Jing and Liu, Pinxin and Tang, Yolo Y. and Xu, Chenliang},
  note={Preprint; submitted to ICASSP 2027},
  year={2026},
  url={https://ali-vosoughi.github.io/SoundCLIP/paper/main.pdf}
}