New Remedies Fix Multi-Task Bayesian Optimization Pitfalls

Bayesian optimization often leverages data from related 'source' tasks to warm-start a 'target' experiment, a process known as multi-task Bayesian optimization. The multi-task Gaussian process is the conventional surrogate model used for this purpose. However, a recent study reveals that this default approach can significantly misestimate cross-task correlation, even in straightforward scenarios like affinely related tasks where transfer learning should ideally excel. The research pinpoints two independent structural issues contributing to this failure. Firstly, per-task standardization, a common fix for affine slice ambiguity, introduces finite-sample alignment errors that propagate into the recovered correlation. Secondly, the marginal likelihood identifies correlation only at a per-sample rate, which is further diluted when Gaussian processes operate on non-overlapping designs. Based on this analysis, three conservative remedies are proposed: promoting per-task means and scales to model parameters, restricting the task covariance to non-negative correlations, and strategically co-locating parts of the source and target designs. These remedies successfully restore the target-only baseline performance on simpler instances and show promise on more complex problems, outperforming most rank-based and latent-context variants.