LLMs Generate New Features for Improved Machine Learning Performance

Feature engineering is a critical, often manual, step in machine learning that involves creating new attributes from existing data to improve model performance. This new research introduces an automated pipeline that utilizes large language models (LLMs) to streamline this process. The proposed system takes a set of initial attributes from tabular data and a specific prompt, then instructs an LLM to generate novel features using mathematical and logical operations. Following this generation phase, a selection algorithm evaluates these newly created features and identifies the most effective combinations for a given task. Evaluated across eight diverse datasets, the method consistently demonstrated that LLM-generated features can significantly enhance classification results. This suggests that LLMs possess a strong capability to understand data relationships and construct valuable new features, reducing the manual effort traditionally required in feature engineering.