A Comparative Study on the Efficiency of EvoSuite and GPT-3.5-Turbo for Unit Test Generation

Abstract

Automated unit testing is essential for software quality, yet evaluating the true efficiency of generation tools remains challenging. This paper compares the test generation efficiency of a traditional search-based tool (EvoSuite) and an LLM (GPT-3.5-Turbo) on 15 Java files across varying structural complexities. We evaluate efficiency through statement/branch coverage, test pass rates, and the impact of iterative prompting. Quantitatively, EvoSuite demonstrates superior initial efficiency, achieving high coverage and nearly perfect pass rates, whereas GPT-3.5-Turbo requires multiple iterations to close the structural gap. However, our qualitative analysis of failed assertions reveals that this efficiency comparison masks a “pass rate illusion.” While EvoSuite relies heavily on defensive assertions for runtime stability, iteratively refined GPT-3.5-Turbo tests attempt to verify actual semantic correctness, despite exhibiting more Expected Result Errors. These findings suggest that evaluating test generation efficiency must go beyond traditional coverage and pass rate metrics to include assertion quality.