Techniques for the automated testing of document analysis algorithms

This paper proposes a new approach to automate and manage the testing process for developing document analysis and understanding algorithms. A distributed test environment is proposed to assure visibility, repeatability, scalability and consistency during and between testing sessions. A variety of views are used to deal with multi-level operations in algorithm development. Tests are realized with dedicated test scenarios and events at different stages of the development cycle. Our main objective is to provide collaborating researchers with a flexible means of generating consistent ways to validate algorithm behaviour in a target environment. This is accomplished with a simulated environment and underlying resources for each test scenarios. A set of techniques to design test events for test scenarios (e.g. module, integration, regression) is proposed aimed at promoting a black-board style research approach. To demonstrate the functionality of this approach, we have implemented a prototype and trace an example algorithm through the development process.