LLMs Drive Evolutionary Feature Engineering for Structured Data.

Large language models (LLMs) are increasingly being utilized as powerful search operators within evolutionary optimization frameworks. This research introduces Evolutionary Feature Engineering (EFE), a novel framework that leverages LLM-based evolution to automatically discover effective preprocessing transformations for structured data. These transformations are represented as Python programs with a standardized `fit/transform` interface, allowing for seamless integration into existing machine learning pipelines. During the evolutionary process, candidate programs are iteratively refined using contextual information from the dataset, summary statistics, and performance feedback from a validation set. The EFE framework is demonstrated in two distinct settings. EFE-Time is designed for time-series forecasting, where it learns invertible, dataset-specific normalizations. These normalizations have been shown to improve the performance of off-the-shelf time-series foundation models, reducing forecasting errors (MASE, WQL, MAE) by 3% or more on average across datasets, with improvements reaching up to 19% on specific datasets like COVID-Deaths. Notably, these gains are observed even with recent and advanced time-series foundation models such as Chronos-2. In the context of tabular prediction, EFE-Tab evolves compact feature programs that both add useful, interpretable features and remove redundant ones. This approach either matches or surpasses the performance of existing LLM-based feature-engineering methods. EFE-Tab proved particularly effective with classical decision trees, where small sets of evolved features yielded competitive accuracy while maintaining high interpretability. Overall, EFE highlights the significant potential of LLM-based evolution to enhance both the accuracy and interpretability of machine learning models when automatically processing structured data.