"Vibe Coding" Evaluated for Greenfield Software Engineering Tasks

The rapid advancements in generative AI are ushering in a new paradigm for human-computer interaction, particularly in software development. A growing trend, dubbed "vibe coding," involves using natural language prompts to create applications and coding infrastructure, often by individuals without extensive programming expertise. This approach represents a significant step towards higher levels of abstraction in programming, potentially eliminating the need for traditional code syntax. This research aims to assess the practical viability of "vibe coding" for developing new software projects from scratch, known as greenfield software engineering tasks. The authors also critically examine the benchmarks currently employed to gauge the software engineering capabilities of large language models. To provide focused insights, the paper introduces a new evaluation suite specifically designed to analyze an LLM's proficiency in handling simple, isolated greenfield programming tasks written in Python. This suite helps to scope the understanding of AI's effectiveness in this emerging programming style.