What are the implications of multi-core devices on code coverage?

The use of code coverage (also called structural coverage analysis or SCA) when testing safety-critical systems is widely established. For example, in the aerospace domain, the DO-178B/C guidance requires specific types of coverage testing to be carried out. The depth of coverage testing required depends on the system's safety level, with the most critical requiring the use of modified condition/decision coverage (MC/DC).

Historically, coverage metrics (function, statement, decision, MC/DC etc) have all been based on the assumption of examining sequential source code executing on a single processor. However, increasingly the producers of embedded systems are turning to multi-core devices to address the continuing need for greater processing power. What implications does the adoption of multicore devices have on coverage testing?

Within our research on the EU-funded parMERASA project we have identified a number of additional coverage items that we think are important when considering the use of a multicore system in a safety-critical context. In this article, we discuss two of them:

• Core affinity.
• Coverage of synchronization mechanisms

Core affinity

The first, and perhaps most obvious metric concerns core affinity, that is, correlating sections of code with the core (or cores) on which they execute. At present, most multicore systems assume a fixed mapping where an application is assigned permanently to a specific core. In this case, it may be useful to verify that the expected sections of code always execute on the assigned core, and never on any others.

As the use of multicore and manycore devices increases, it appears likely that software will have to be parallelized in order to fully exploit the gains in performance. In a parallelized application, sections of code may be executed on a number of different cores. A feature offered by a multicore coverage tool would be to highlight which sections of code were exercised on each core in the system, and also indicate when sections of code were seen to execute on multiple cores. This information can be used to ensure that the effects of core migration are fully explored during testing.

Coverage of synchronization mechanisms

When multiple applications run on the same multicore system, it is necessary to protect access to shared resources (e.g. memory or IO) to ensure that mutual exclusion is maintained. Furthermore, one feature of parallelized software is the need for synchronization, to ensure consistency of data across an architecture where each core accesses a local memory or cache. These require the use of explicit critical sections or barriers within the application code itself. These synchronization methods can potentially be called from many different locations in the code, and in any number of different orders.

A coverage tool for parallel software could offer information on the coverage of synchronization and mutual exclusion mechanisms within the application. In particular, evidence can be generated to show that critical sections have been entered by all the possible routes in the source code. It can also be shown what proportion of the different combinations in which a barrier can be reached have been encountered during testing. This information can be used in conjunction with timing measurements to estimate the worst-case waiting time for the barrier. You can find out more about parMERASA at the project website.