Researchers identify cross-project and inconsistent flakiness in OpenStack and reveal that flaky tests are not confined to individual projects but propagate across projects within software ecosystems. Credit: Kyushu University
In a study published in IEEE Transactions on Software Engineering, researchers from Kyushu University have found that "flaky tests," which are unstable software tests that seem to randomly pass or fail, do not stay confined to the projects they originate in and often spread across entire ecosystems.
After analyzing hundreds of interconnected projects in OpenStack, a widely used open-source cloud computing platform, the research team found that 55% of projects were affected by cross-project instability, resulting in a cumulative loss of 1,156 days of developer time.
Complex software systems, such as those used in cloud platforms, banking services, health care records, and government infrastructure, rely heavily on automated testing to ensure reliability. Each time a developer modifies code, automated tests run to confirm that nothing breaks.
This process is known as Continuous Integration (CI) and allows software to evolve quickly while maintaining stability. Without it, even small errors could disrupt critical services that are used daily by millions of people.
