"Succeeding When you have no Time for Testing"

Event Report

As the title suggests this evening's event focussed on how best to test software when project timescales have become so time-constrained, through either poor estimating or slippage, that there is insufficient time left to do the testing as thoroughly as it should ideally be done.

The speakers, Barry Varley and Tom Norris of Acutest, gave a very interesting talk with an interactive session to prove their points. Acutest focus on testing IT, processes and technology-centred change initiatives. They willingly shared their knowledge and experiences with the 35 - 40 members who journeyed to the Royal Holloway College at Egham, on Thursday 11th September 2003. After a short introduction, members took part in an exercise to identify the tests that would be required to debug software that had been designed to determine whether, from 3 varying input parameters, the result would be a triangle (equilateral, isosceles) or not. Members were also asked to estimate how many tests were required and how long the testing should take. The results were analysed during the rest of the presentation and feedback was provided at the end.

Acutest use an approach to testing that mitigates some of the risks caused by lack of time. They proposed that successful testing could be based on 3 basic rules: keen observation, precise tests and irrefutable conclusions.

• Keen observation is characterised by looking for failure from the outset, taking time for detailed reviews of the requirements and specifications in order to gain complete understanding by all parties, thus preventing misunderstandings that will be more costly, in both time and money, to correct later.
• Precise tests involve the identification of possible causes of failure, assigning likelihood and impact assessments to each, then testing those that were most likely and would have the worst impact. If more time were available at the end of these then testing could be done on the remainder in descending order of importance. The software could then be released on schedule knowing that there were no known bugs in the critical functions of the software and that the less significant areas would be tested and debugged in slower time.
• Irrefutable conclusions provide project managers with information about the testing process that give the necessary clarity and control needed to manage the project. They show what has been tested, what has passed and what has failed, as well as, for those areas not tested, the level of risk inherent in not doing so.

Overall, the process builds confidence regarding the quality of the software, identifies important areas for testing and ensures that any areas not tested are not likely to have a major adverse impact on the project's success. Informed agreements can be reached with the users/customer concerning the different amounts of testing that can be undertaken and the resultant impact they would have on project timescales.

Feedback was then given on the testing exercise and it was very interesting to note that, even among the project managers present who are aware of the testing dilemma, there was considerable (80%) under-estimation of the time needed for complete testing. The number of test required and the optimum time for each test also showed considerable variation - those members that proposed more tests also estimated that the test would take longer.

The event was therefore a salutary lesson for some project managers on the need to plan both sufficient time and resources for testing. However, where insufficient time is available an alternative approach, the use of the likelihood/impact matrix, could be used to good effect.

The slides for this event have been kindly made available by Barry Varley from Acutest, who with his colleague Tom Norris presented the lecture.

Also available are the results of the analysis of the questionnaire conducted at the meeting