Abstract

We propose a novel reinforcement learning framework for post training largelanguage models that does not rely on human in the loop feedback. Instead, ourapproach uses cross attention signals within the model itself to derive a selfsupervised reward, thereby guiding iterative fine tuning of the model policy.By analyzing how the model attends to the input prompt during generation, weconstruct measures of prompt coverage, focus, and coherence. We then use thesemeasures to rank or score candidate responses, providing a reward signal thatencourages the model to produce well aligned, on topic text. In empiricalcomparisons against standard policy gradient methods and RL fine tuning withsynthetic preference models, our method shows significant gains in promptrelevance and consistency over a non RL baseline. While it does not yet matchthe performance of fully human supervised RLHF systems, it highlights animportant direction for scaling alignment with minimal human labeling. Weprovide a detailed analysis, discuss potential limitations, and outline futurework for combining cross-attention based signals with smaller amounts of humanfeedback.