The Art of Software Testing: Refining the Testing Palette

Dr Mark Hadley

Principal Software Safety Consultant, Atkins

Dr Mike Standish

Senior Scientist, DSTL (Portsdown West)

‘It’s been over 40 years (1979) since Glenford Myers published his book on ‘The Art of Software Testing’. In 1980 the Commodore VIC-20 was released which was an 8-bit computer with 5kb of memory, but only 3.5kb was for programmable use. Today Multi-Core Processors (MCPs), with 64-bit processing, are now common place and some form of micro-processor or controller is in our everyday appliances. Cloud Computing, Digital Twins, Machine Learning, Artificial Intelligence, and autonomous systems are no longer science fiction but are here today and will increasingly be involved in our everyday lives. The complexity and size of these systems does not compare to a VIC-20, so why do we have complete confidence in the testing techniques which first appeared in the book by Myers?

This presentation provides a summary of the commonly applied techniques defined in Myers and now common within standards in the safety-critical domain, e.g. DO-178C, 61508 etc. We explore the effectiveness of some of these commonly applied techniques (especially when used in isolation) and highlight the lack of empirical evidence to underpin them. There are opportunities to be smarter in our approach and apply an engineering perspective when selecting the appropriate testing techniques. This should be based upon the system properties to provide the required level of confidence and rigor’.

About Dr Mark Hadley

Mark has been involved in the safety critical software domain for almost 25 years with Atkins and the UK Defence Science and Technology Laboratory (Dstl) (and its predecessor organisations) working on a range of civil and UK Ministry of Defence (MOD) systems. Mark is a principal software safety consultant at Atkins and currently working in the energy sector. Mark was a senior principal consultant in software at Dstl and provided Independent Technical Evaluation (ITE) and Subject Matter Expert (SME) advice to a host of MOD Project Teams. He led research into a number of areas such as: multi-core processors, tool technology and the generation of diversity of evidence arguments to support the qualification of mission and safety critical systems. Mark completed his PhD in software testing at the University of York. He is a Chartered Engineer (CEng) gained via the Institution of Engineering and Technology (IET).

About Dr Mike Standish

Mike is a senior scientist in systems at the UK Defence Science and Technology Laboratory (Dstl). Mike has experience of all aspects of software and systems lifecycles, which has been gained in over 15 years within the defence sector. Mike holds a BSc in Software Engineering and an MSc in Strategic Information Systems. Mike recently gained an Engineering Doctorate (EngD) in Systems from the University of Bristol with a focus on how to adopt wider diverse evidence to mitigate shortfalls in software process-based safety assurance evidence. He is a Chartered Engineer (CEng) gained via the British Computer Society (BCS).