Nonparametric Bayesian IRL Infers Multiple Expert Rewards.

Inverse Reinforcement Learning (IRL) aims to deduce the underlying reward function that explains observed expert behavior. A common limitation of standard IRL is the assumption that all demonstrations originate from a single expert, leading to an averaged, potentially inaccurate, reward function when multiple experts with distinct preferences are involved. This paper addresses this by proposing a Nonparametric Bayesian IRL approach. The core of the method is the use of a Dirichlet Process prior over reward functions, which allows the system to automatically infer the number of latent reward types present in the pooled demonstrations, alongside the rewards themselves. Inference is performed using a collapsed Gibbs sampler, which combines a Chinese Restaurant Process update for assigning demonstrations to clusters with a Metropolis-Hastings update for reward weights, using soft value iteration for planning. Evaluated on an ObjectWorld grid, the method successfully recovered the correct number of reward clusters and significantly outperformed a Maximum Entropy IRL baseline in terms of Adjusted Rand Index. Furthermore, the researchers parallelized the Gibbs sampler across CPU cores using Ray, achieving substantial speedups, demonstrating its potential for scalability. This approach offers a robust way to understand and model diverse expert behaviors.