Tyche: In Situ Analysis of Random Testing Effectiveness

Automated testing tools have adapted to increasing program complexity by reducing the user's role in the testing process. Approaches like property-based testing supplement traditional unit-testing with a mode declarative approach: rather than write traditional input-output examples, the user writes executable specifications of their programs. The testing framework then exercises those specifications with randomly generated values. However, more automated approaches to testing risk hiding too much from the user. Current property-based testing frameworks give insufficient feedback about the specific values that were used to test a given program and about the distributional trends in those values. In the worst case, this lack of visibility process may give users false confidence, encouraging them to believe their testing was thorough when, in fact, it had critical gaps. We demonstrate Tyche, an editor extension that recovers visibility into the property-based testing process. Tyche provides an interactive interface for understanding testing effectiveness, surfacing both "pre-testing"information about test inputs and their distributions and "post-testing"information like code coverage. The extension is designed to work out of the box with tests written in Python's popular Hypothesis framework, so users can immediately start using it to improve their testing.