GLARE: Generating Listening Heads with Appropriate Reactions

📌 Highlights

GLARE reframes listening-head generation as reaction-aware social behavior generation rather than generic head-motion synthesis.

The paper's strongest empirical claim is that explicit reaction supervision plus reaction-centric evaluation improves both conventional visual metrics and listener-reaction metrics on RealTalk and Seamless.

The main caveat is that reaction appropriateness is multi-valid, but the automatic metrics still compare against a single ground-truth reaction timeline.

• Dataset: approximately 147 hours, 107,149 speaker-audio/listener-video pairs, and 64,557 annotated reaction instances.
• Model: flow-matching transformer with listener reference image, speaker audio, emotion latent, Qwen2-Audio prosody conditioning, and temporal reaction loss.
• Metrics: R-F1, R-tIoU, R-ATD, and R-FID evaluate reaction occurrence, timing, asymmetric temporal deviation, and reaction-region visual quality.
• Best RealTalk row: Ours reports PSNR 17.972, FID 35.697, FVD 142.454, R-F1 0.594, R-tIoU 0.704, R-ATD 57.388, and R-FID 15.192.
• Best-supported limitation: prosody conditioning helps occurrence but can hurt timing metrics, so semantic context remains under-modeled.

🎯 Introduction

The paper targets dyadic listener synthesis: given speaker-side cues and a listener reference, generate a listener head video that reacts naturally. The authors argue that prior talking-head progress has focused on speakers, while listeners communicate attention, agreement, hesitation, and affect through non-verbal feedback such as nodding and smiling.

The main prior limitation is twofold. Existing dyadic datasets such as RealTalk, Seamless Interaction, and SpeakerVid-5M provide conversational videos but not event-level reaction annotations of type, timing, and duration. Existing evaluation protocols emphasize reconstruction fidelity, perceptual realism, and motion quality, but do not directly ask whether the generated listener reacted with the right type at the right moment.

GLARE's paper-level objective is to address data, modeling, and evaluation together: curate reaction-labeled speaker-listener pairs, build a reaction-aware audio-driven baseline, and evaluate generated listeners with reaction-specific metrics.

• Task: listening-head generation for dyadic conversation.
• Prior bottleneck: lack of fine-grained reaction labels and reaction-oriented metrics.
• Objective: generate listener reactions with appropriate type, timing, duration, and visual quality.

fig:curation_pipeline

Caption: Overview of our data curation pipeline: Step 1, we leverage conversational data from the RealTalk and Seamless datasets; Step 2, videos are cropped into facial regions; Step 3, we apply audio separation and diarization to disentangle listening and speaking segments in each conversation with timestamps, and pair speaker audio clips with corresponding listener clips; and Step 4, we define the reaction classes and detect reactions in each paired clip, resulting in multi-class frame-wise reaction annotations.

Why It Matters: It shows the source of the reaction-aware supervision that prior datasets lacked.

🔬 Methodology

The dataset pipeline starts from RealTalk and Seamless Interaction, filters low-quality or unstable face crops, separates and diarizes audio when needed, pairs active speaker audio with synchronized listener portrait video, and annotates six listener reaction classes. The detector produces dense reaction scores r in [0,1]^{T x 6}, then converts high-confidence regions into event annotations with type, score, start time, and end time.

The GLARE model follows a latent generation design. A frozen LIA image autoencoder extracts listener reference motion and identity latents, Wav2Vec2 encodes speaker audio, an emotion encoder supplies speaker-emotion latents, and Qwen2-Audio-7B-Instruct produces frame-aligned prosody intensity signals. These are concatenated and projected into driving conditions for a DiT-style flow-matching transformer that predicts listener motion latents.

The key technical addition is temporal reaction supervision. Current-frame hidden latents are mapped to a 192-dimensional representation, split into six 32-channel class-specific subspaces, reduced through class-specific linear layers, and sigmoid-activated into frame-wise reaction predictions. Smooth-L1 reaction loss is combined with flow-matching and velocity losses.

• Prosody signal: frame-aligned scalar p in [0,1] for speaker prosodic or temporal affective variation.
• Driving condition: listener motion latent, audio latent, emotion latent, and prosody latent.
• Inference: FMT predicts vector fields, an ODE solver integrates current listener motion latents, and the decoder produces video frames.

fig:overview

Caption: Overview of our Prosody-conditioned Reacting Listener GLARE. Given speaker audio, a listener reference image, and temporal context, our model predicts listener motion latents using a conditional flow matching transformer (FMT). We further introduce prosody conditioning to capture speaker-side temporal prosodic variations, and a temporal reaction loss to explicitly supervise frame-wise listener reactions. The resulting motion latents are decoded into listener video frames.

Why It Matters: It ties together prosody conditioning, flow matching, and temporal reaction loss in one architecture view.

📊 Experiments

The main experiments train and test on the curated RealTalk and Seamless subsets with a 9:1 train/test split. Baselines are L2L, DIM, ViCo, ListenFormer, and DyStream. Conventional metrics include FID, FVD, PSNR, SSIM, Var, LPIPS, rPCC, and DI-Sync; proposed reaction metrics include R-F1, R-tIoU, R-ATD, and R-FID. Higher is better for PSNR, SSIM, Var, DI-Sync, R-F1, and R-tIoU; lower is better for FID, FVD, LPIPS, rPCC, R-ATD, and R-FID as displayed in the paper tables.

On RealTalk, GLARE reports the best or strongest overall balance across most metrics, including FID 35.697, FVD 142.454, LPIPS 0.454, DI-Sync 0.245, R-F1 0.594, R-tIoU 0.704, R-ATD 57.388, and R-FID 15.192. On Seamless, GLARE reports PSNR 14.245, SSIM 0.473, FID 27.288, FVD 193.281, LPIPS 0.371, R-F1 0.460, R-tIoU 0.582, R-ATD 102.113, and R-FID 28.857.

Per-class results compare GLARE to DyStream and show consistent R-F1 gains across all listed reaction classes on both datasets. The paper's interpretation is that laughing and smiling tend to be easier because they are more expressive or data-rich, while surprised is harder because it is sparse and ambiguous.

The ablations are important for builders: prosody conditioning alone improves some visual and occurrence metrics but worsens R-ATD in the module table, while reaction loss gives the major gains in DI-Sync, R-F1, R-tIoU, R-ATD, and R-FID. The reported best trade-off for reaction loss coefficient is lambda_react=0.05.

• Central datasets/benchmarks: RealTalk and Seamless.
• Central baselines: L2L, DIM, ViCo, ListenFormer, and DyStream.
• Human evaluation: 100 generated RealTalk videos, 165 reaction events, 23 participants, and 3,795 reaction-participant annotations.

fig:realtalk

Caption: Qualitative comparison of our approach with state-of-the-art methods on the RealTalk dataset. Reaction labels are displayed at frames where reactions are detected. Our method generates more natural listening motions and reactions that are more temporally aligned with the ground truths than other methods.

Why It Matters: It provides qualitative support for the paper's reaction timing and naturalness claims.

🔮 Conclusion

The paper concludes that listening-head generation benefits from treating reactions as explicit, temporally localized events. The combination of reaction-labeled data, GLARE's prosody-conditioned flow-matching architecture, temporal reaction loss, and reaction-centric metrics produces stronger listener behavior than baselines on RealTalk and Seamless.

For builders, the practical takeaway is not simply that GLARE is a better model row in a table; it is that listener-avatar evaluation needs to measure whether reactions happen, what type they are, when they occur, and whether reaction segments look realistic. The paper's results support explicit reaction supervision as a more reliable lever than generic prosody conditioning alone.

• Explicit reaction supervision is the most important component in the main-body ablations.
• Reaction metrics complement visual metrics and expose behavior quality that FID/FVD may miss.
• The dataset and annotation pipeline are central to the method's value.

🛠️ Future Research Improvements

A clear next research direction is multi-valid reaction evaluation. The paper acknowledges that appropriate listener behavior may not match a single reference timestamp, but the metrics still use ground truth as the reference for matching. Future work could evaluate sets of plausible reactions or use human/contextual preference models.

The prosody ablation suggests that speaker prosody alone is insufficient for precise reaction timing. A stronger model could combine prosody with transcript semantics, discourse structure, humor detection, turn-taking signals, and speaker intent to avoid over-triggering immediate listener reactions.

The class imbalance visible in the reaction statistics suggests room for targeted data collection or reweighting, especially for surprised and other sparse, context-specific reactions.

• Develop multi-reference or preference-based reaction metrics.
• Add semantic context beyond audio prosody.
• Improve sparse-class handling for surprised and subtle reactions.
• Validate robustness outside RealTalk and Seamless-style dyadic videos.

🏭 Potential Industry Use Scenarios

The most credible industry use case is conversational avatar generation where listener behavior matters: AI companions, virtual meeting participants, interview training systems, social coaching tools, tutoring avatars, and telepresence agents. In these settings, a static or mistimed listener can feel less believable than a lower-fidelity but socially responsive one.

The paper's dataset and metrics may also be useful as an evaluation harness for commercial avatar vendors. Teams can run R-F1, R-tIoU, R-ATD, and R-FID alongside their existing visual-quality metrics to detect whether model updates improve actual listening behavior.

Deployment uncertainty remains high because the main-body evidence is limited to RealTalk and Seamless, plus a RealTalk human study. Production systems would need domain-specific testing for culture, language, interpersonal norms, latency, and privacy constraints.

• AI meeting listeners and virtual agents that need nodding, smiling, laughter, or frowning.
• Training simulators for interviews, therapy practice, sales calls, and education.
• Avatar QA pipelines that evaluate reaction timing and type rather than only visual realism.
• Dataset bootstrapping for reaction-aware listener modeling.

💬 Critical Analysis

The paper is strongest when it argues that listener reactions deserve their own data and metrics. The dataset scale, reaction taxonomy, event-level evaluation, and ablation evidence all support the claim that a reaction-aware training signal improves listening-head generation.

The weaker point is appropriateness. The metrics verify type and timing against annotations, but conversational appropriateness can be socially and semantically multi-valid. A listener might smile, nod, remain neutral, or delay a reaction depending on personality and context, and the automatic metrics may punish plausible alternatives.

The model also depends on a reaction detector both for annotation and evaluation. This is practical for scale, but it risks detector-shaped supervision and detector-shaped measurement. The human evaluation partially offsets that concern, reporting overall HAR values of 0.899 for naturalness, 0.894 for contextual appropriateness, and 0.935 for timing plausibility, but only on sampled RealTalk outputs.

For reproducibility, the main-body training recipe is reasonably concrete, but key annotation details are deferred outside the main body and the release is stated as future availability. Builders should treat the paper as a strong design pattern and benchmark proposal, then verify the released pipeline before depending on the exact reported numbers.

• Strength: clear task reframing from visual listener motion to reaction-aware listener behavior.
• Strength: evaluation metrics are aligned with the failure modes builders actually notice.
• Risk: single-reference reaction matching underestimates multi-valid behavior.
• Risk: detector-based annotations and detector-based evaluation may share correlated errors.
• Reproducibility question: code, annotation pipeline, and processed dataset are stated as planned release.