Most engineers would agree that plenty of early testing builds a better product. But the testing process is often short-changed due to time and budget overruns in design.
|Estimated vs. actual testing time. (Image Source: John Blyler (used with permission))|
As a rule of thumb, testing is estimated to take about one-third of the product development life cycle. However, that seldom happens. It is far more common that the development time far exceeds the projected development time, causing a substantial decrease in testing time. The risk of sufficient test time is high. For this reason, strategies like shift left testing have been gaining in popularity for many years.
Shift left testing is a methodology in which testing is performed earlier in the product lifecycle, that is, it is moved left on the project timeline. In the software world, the aim is to find and prevent defects early in the software delivery process. In the larger hardware-software space, shift-left testing benefits the development of the higher-level architecture.
The semiconductor chip design world has been promoting verification shift-left tools for many years. As the verification curve is shifted level, it forces bugs to be found in weeks instead of months. But enabling this shift in chip development requires the use and coordination of numerous tools including simulation, debugging, Verification IP, formal verification tools, FPGA prototyping, and emulation hardware.
Shift-left testing is also critical for the growth of the autonomous vehicle digital twin and the goal of more complete automotive simulation. But did you know there are four fundamentally different approaches to shift left testing? The approach you select will depend on where you want to go on the product lifecycle.
For engineers with a bend toward the automotive world, shifting left can be described in the racing vernacular of upshifting or downshifting, but through the systems engineering V-diagram instead of on the racetrack. To be clear, the analogy is that upshifting is like moving toward a higher gear or – in the case of the V-Diagram – a higher level of abstraction and validation. Downshifting moves one downward toward domain-specific testing and verification.
In the classical V-model diagram, the left side of the V represents requirements, architecture, design, and detailed implementation whereas the right side represents integration and testing. Here, the word "system" means either a pure software application or a multi-domain system.
According to the Software Engineering Institute (SEI), four variants can be derived from the classic V model:
- Traditional shift left verification – (Downshifting) Instead of emphasizing higher-level acceptance and system-level testing, traditional shift left to concentrate on unit and integration testing (e.g., using API testing and modern test tools).
- Incremental shift left testing – (Upshifting for Hardware/Downshifting for Software) Parts of the Waterfall testing is shifted left to become incremental testing on smaller increment. Incremental shift left testing is popular when developing large, complex systems, especially those incorporating significant amounts of hardware.
- Agile/DevOps shift left testing – (Downshifting) Agile software testing is typically restricted to developmental testing and does not include operational testing of the entire system
- Model-based shift left testing – (Upshifting) The previous models require software to exist before testing can begin. This effectively limits the use of testing to uncovering code defects. However, as Boehm and others have pointed out over the years, from 45 to 65 percent of defects are introduced in the requirements, architecture, and design activities.
Model-based shift left testing moves these activities to the left side of the V diagram by testing executable requirements, architecture, and design models. The benefit of this approach is that it enables us to begin at the very start of the project, instead of waiting a long time (traditional), medium time (incremental), or a short time (Agile/DevOps) is available to test. This approach will grow as more and more executable models, intellectual property (IP) reuse, and related simulation/testing tools become available.
All four shift left approaches move test creation and execution – but at different levels – to the left of the development V diagram life cycle, rather than creating tests after hardware and software development are complete.
Perhaps the model most in tune with the needs of the automotive industry and its safety mandates – e.g., ISO 26262 – is the model-based approach. Another advantage of the model-based approach is its use of simulation, IP reuse, emulation, and even prototyping, especially at the chip design level.
Modern use of simulation, emulation, and various prototyping technologies enables software developers to start almost in parallel with hardware development. Such co-development or effectively decouples the hardware and software design timelines. For example, simulation and emulation allow you to start software development as soon as you have the intellectual property (IP) models or the chip design register transfer language (RTL) code available, meaning you don’t have to wait for the availability of actual silicon. Thus, by parallelizing the software and hardware development, the overall development cycle shifts to the left.
John Blyler (used with permission)
|Traditional systems engineering V-Diagram with shift-to-left.|
John Blyler is a Design News senior editor, covering the electronics and advanced manufacturing spaces. With a BS in Engineering Physics and an MS in Electrical Engineering, he has years of hardware-software-network systems experience as an editor and engineer within the advanced manufacturing, IoT and semiconductor industries. John has co-authored books related to system engineering and electronics for IEEE, Wiley, and Elsevier.