SymboLLM-FE Boosts Feature Engineering with LLMs and Symbolic Regression

Tabular data often presents challenges in machine learning due to features lacking sufficient discriminative power, which automated feature engineering (AutoFE) aims to address by generating and selecting new features. However, existing AutoFE methods frequently produce features that are difficult to interpret, while LLM-based approaches can be costly due to multi-round iterations and carry risks of bias or hallucination. To overcome these limitations, researchers propose SymboLLM-FE, a hybrid method that integrates symbolic regression with large language models. Symbolic regression is used to extract mathematically expressive formulas strongly correlated with the target variable, thereby enhancing model performance. These formulas are then refined by LLMs, leveraging their extensive prior knowledge to ensure the interpretability of the generated features. Empirical evaluations across six real-world datasets and four Kaggle competitions demonstrate that SymboLLM-FE surpasses current AutoFE techniques. The method effectively tackles the dual challenges of poor interpretability and excessive iterations by employing a statistically grounded LLM refinement mechanism, requiring only a minimal number of LLM calls.