PAIRSAT: Integrating Preference-Based Signals for User Satisfaction Estimation in Dialogue Systems

User satisfaction estimation in dialogue systems is a fundamental measure for assessing and improving conversational-AI quality and user experience. Current approaches rely on users' satisfaction annotations, referred to as supervised labels. Yet these labels are scarce, costly to collect, and often domain-specific. Another form of feedback arises when a user selects one of two offered responses in a conversation, usually called a preference signal. In this work, we propose PAIRSAT, a new model for user-satisfaction estimation that integrates both satisfaction labels and preference signals. We reformulate satisfaction prediction as a bounded regression task on a continuous scale, enabling fine-grained modeling of satisfaction levels. To exploit the preference data, we incorporate a pairwise ranking loss that encourages higher predicted satisfaction for accepted conversation responses over rejected ones. PAIRSAT jointly optimizes regression on labeled data and ranking on preference pairs using a Transformer-based encoder. Experiments demonstrate that our model outperforms baselines that rely solely on supervised satisfaction labels, demonstrating the value of adding preference signals. Further, our results underscore the value of leveraging additional signals for satisfaction estimation in dialogue systems.