Software testing essentials

Embedded software testing is a rigorous task that can be time and labor intensive. Manual testing is expensive, is prone to errors, and it can be difficult to reproduce manual tests. Tools that can automate testing can increase the efficiency of the process and save costs. Tool automation also allows tests to be run in a continuous integration environment, reducing the effort needed to identify and fix any bugs developed during software development.

As such, using a tool to automate software testing is a good step on the road to executing a robust testing plan as good tools save developers time and lower project costs. We’ve listed some important features to look out for when selecting an embedded software testing tool below.

1. On-target and on-host testing

Embedded software can be tested either on-target or on-host. In on-target testing, an application is tested on the hardware to be deployed (the target). In on-host testing, an application is tested on a host computer that has a different hardware environment to the final application.

On-target testing is important as software behavior often depends greatly on the hardware environment the software is hosted on. On-target testing is, however, often more expensive than on-host testing, and requires access to the target hardware. In some applications, however, access to the final target hardware may not be readily available and may only be available late during software development.

On-host testing is usually less resource intensive than on-target testing and provides a means of performing tests even when the hardware is not readily available. This has several benefits including allowing testing to be automated and run continuously during software development and allowing testing to be performed off-site.

When testing embedded software, the best option is to keep your options open so you can perform both on-target and on-host testing. Tools that are specifically designed for on-target testing such as RVS also support on-host testing, offering you the flexibility to use both testing strategies.

2. Unit, integration and system testing

Embedded software can be tested at various levels of granularity, representing environments that more or less closely represent the final system in which the software will be hosted. Among others, the terms unit testing, integration testing and system testing are often used.

Unit testing involves testing individual modules of the application in isolation i.e. without interaction with other modules or external dependencies such as databases. As unit testing only requires a subset of the software, it allows testing to begin early in software development. Unit tests are often automated to run throughout development as a set of regression tests that can identify errors introduced during development.

Integration testing involves checking if modules work correctly when they are combined. This provides a wider scope for testing, and bugs that are introduced arising from module interactions are commonly detected during integration testing.

System testing involves checking if the entire integrated software is working correctly. Performed on the fully integrated system, this lets engineers check whether the system meets its high-level requirements.

As unit, integration and system testing each have benefits during embedded software testing, you probably want to select a tool such as RapiTest, which lets you test systems at all levels of granularity.

3. Structural code coverage analysis

Structural code coverage is a metric used to establish the completeness of software testing. The higher the percentage of coverage achieved when testing a system, the less risk there is of the software having unintended functions, missing tests or requirements.

When testing embedded software, various code coverage metrics may be collected, which provide an increasing level of granularity about the software’s testing:

• Function coverage
• Statement coverage
• Decision (branch) coverage
• Modified condition decision coverage (MC/DC)

When selecting an embedded testing tool, the higher the granularity of coverage testing you perform, the more reassurance you can have that your software doesn't include unintentional functionality. As such, you may want to select a tool such as RapiCover, which analyzes coverage at all levels listed above.

4. Execution time analysis

Execution time analysis gives developers an understanding of the execution time behavior of their code in its target environment. This analysis is especially important when working with real-time systems or safety-critical applications.

This analysis can be used to test whether software executes within any timing budgets allocated to it, and to easily identify areas of the code that can be optimized to improve its timing performance.

When analyzing software execution time, various timing metrics are of interest:

• Overall execution time of the code
• High watermark execution time
• Contribution of each function to execution time
• Worst-case execution time (vital in safety critical applications)

As with code coverage analysis tools, the more metrics you can get out of your embedded testing tool, the better. Tools like RapiTime let you perform detailed timing of your software, helping you to optimize your code and make the most of your target hardware.

5. Scheduling analysis

Scheduling analysis provides an overview of when tasks are executed, allowing embedded testers to get a picture of the program flow and task response times. This analysis also lets engineers see how often specific events occur in their software. From this information, rare timing events such as race conditions or deadlocks can be identified, investigated and resolved.

To best understand the scheduling behavior of your system, you may want to look for tools with features that help you visualize large traces easily and view additional metrics such as CPU utilization and jitter, like those offered by RapiTask.

6. Continuous integration

Continuous integration servers are often used to keep track of software development over time. When an automated test tool can interface with the continuous integration server being used, tests can be automatically run with each new revision of the software. This helps with software quality control as it lets you track your testing progress over time and identify which versions of the source code caused failures.

When selecting an embedded software testing tool, you may want to select one that integrates with the continuous build environment you're already using.

7. Tool qualification

If you need to certify the embedded software you’re developing, it is good to have assurance that the testing tool you will be using produces accurate results. To provide this assurance, software testing tool vendors can qualify their tools, providing evidence that they produce dependable results through internal tests of their tool. This feature is essential, especially when working with safety-critical software e.g. avionics software and DO-178 certification.