New Research Explores Adversarial Robustness in Programming by Example Systems

Programming-by-Example (PBE) systems, which infer programs from input-output examples, are typically evaluated for robustness against random noise in examples. This research introduces a new perspective: adversarial robustness, where an attacker intentionally corrupts examples to cause the most damage to the inferred program. The study formalizes this "fixed-set worst-case corruption" for finite PBE systems. The paper implements methods to search for such adversarial corruptions in a string-transformation domain. It also proposes a defense mechanism called Version-Space Partition Aggregation (VPA), which synthesizes programs from disjoint example groups and combines results based on semantic signatures. The findings indicate that PBE tasks with low margins are vulnerable to adversarial attacks that random noise evaluations miss. While VPA can help in some cases, its effectiveness is limited to situations where clean semantics maintain a clear partition vote margin, which often isn't the case in practical tasks. The research provides evidence from various DSL tasks and public benchmarks, demonstrating that even small, targeted corruptions can significantly degrade PBE system performance.