New Multiscale Reward Hedging for Learning from Demonstrations

This research addresses the challenging problem of learning from correct demonstrations, particularly when multiple valid answers exist and the learner does not observe rewards or losses. The paper introduces a novel multiscale reward hedging approach that provides the first horizon-free guarantee for continuous reward classes, a significant advancement over prior methods limited to finite reward classes. The core idea involves hedging across a shared "vote" over tolerant optimality tests at various accuracy scales. A target reward maintains a proxy at each scale, and a prediction with a gap exceeding that scale causes the proxy to double. This mechanism yields a simultaneous tail bound and, through integration, a cumulative hidden gap bounded by a metric-entropy integral, independent of the number of rounds. The method achieves polynomial finite bounds for tasks like bounded linear contextual recommendation and fixed-radius rank-two recommendation, demonstrating its effectiveness even with improper prediction.