Learning Reward Functions from Multiple Feedback Types
with Amortized Variational Inference

Rich human feedback comes in many forms — preferences, demonstrations, ratings, stops — and is increasingly easy to collect. But combining these types so their complementary information is actually leveraged has remained an open problem. The key insight behind MAVRL: every feedback type is a noisy observation of the same underlying reward. MAVRL combines them all by encoding every type into a shared latent reward and decoding each one back — so their complementary signals are used jointly, with no manual loss balancing.

Each feedback type induces a characteristic pattern in the inferred reward and uncertainty. Demonstrations  yield low-uncertainty estimates along expert trajectories but leave large regions underdetermined. Preferences  provide broader coverage but can over-reward frequently visited states. Ratings  reliably identify goals but offer limited landscape information. Stops  strongly constrain unsafe regions while providing little guidance on desirable behavior. Only their combination recovers the ground-truth reward.

Reward learning typically relies on a single feedback type or combines multiple feedback types using manually weighted loss terms. Currently, it remains unclear how to jointly learn reward functions from heterogeneous feedback types — such as demonstrations, comparisons, ratings, and stops — that provide qualitatively different signals.

We address this challenge by formulating reward learning from multiple feedback types as Bayesian inference over a shared latent reward function, where each feedback type contributes information through an explicit likelihood. We introduce a scalable amortized variational inference approach that learns a shared reward encoder and feedback-specific likelihood decoders, trained by optimizing a single evidence lower bound.

Our approach avoids reducing feedback to a common intermediate representation and eliminates the need for manual loss balancing. Across discrete and continuous-control benchmarks, we show that jointly inferred reward posteriors outperform single-type baselines, exploit complementary information across feedback types, and yield policies that are more robust to environment perturbations.

MAVRL formulates reward learning as Bayesian inference over a shared latent reward function. A shared probabilistic reward encoder and an auxiliary Q-value model feed feedback-specific likelihood decoders, each linking the latent reward back to one feedback type. Everything is trained jointly by maximizing a single evidence lower bound (ELBO). New feedback types plug in simply by adding a decoder — no loss balancing across modalities.

Across six environments spanning tabular grid worlds and continuous-control tasks, combining all four feedback types with MAVRL consistently achieves the strongest overall performance. No single feedback type dominates across all settings.

Reward functions learned from multiple feedback types degrade more gracefully when the environment changes at deployment. Under dynamics perturbations — increased stochasticity, altered physics, stronger wind — policies trained on multi-type rewards are generally more stable, outperforming single-feedback baselines in the grid environments and remaining competitive in continuous-control tasks.

MAVRL is also resilient to feedback misspecification: when one feedback channel is corrupted (e.g., noisier preferences or miscalibrated ratings), the remaining modalities compensate. In 3 of 4 corruption scenarios, MAVRL retains ≥90% of its well-specified performance, while single-modality baselines collapse.