How do I demonstrate the safe use of instrumented source code analysis?

In the second of two related blog posts, I describe one method to present a certification argument for the use of instrumentation in source code coverage analysis.

In a previous post I explained how the coverage gained from instrumented code can be used to support DO-178B/C Level A compliance for code coverage. RapiCover uses lightweight source code instrumentation in order to perform coverage analysis. This means that a copy of the application code is annotated so that when it is run we can capture trace data to see which lines of source, decisions, etc., were exercised.

The instrumented code is typically not deployed on the final system. As we want to ensure the results are accurate, we need to be sure that the instrumentation process has not changed the functionality of the original application. For example, if the instrumentation had somehow removed or altered a construct and hence misleading results were presented. As part of our tool qualification process we perform rigorous testing on our instrumenter to ensure that this doesn't occur, using specially designed test programs.

To provide additional confidence the applicant should also run their test suite on both the instrumented and uninstrumented source code and compare the results. If there are no discrepancies then this provides evidence of no changes to functionality. A safety argument representing this dual approach is presented using the Goal Structuring Notation (see GSN description). Note that this provides diverse evidence of the instrumenter's non-interference (StDiverse, SysTestWithWithout, EgProgWithWithout).

The strength of evidence will depend on the depth of testing of the different software (EquivTests). We also need a sanity check to ensure that all the types of construct that are instrumented on the application have been represented in our test programs (AllConstCov). To be complete, I should mention that a pathological case exists where the compiler may have inserted additional code related to the instrumentation which is somehow not tested. However, this is a much wider issue with compiler trustworthiness affecting all aspects of the application code, and is why additional certification requirements exists to examine the object code at Level A. Also, the instrumentation routine itself (generally written by the customer) could have nasty side effects. Our advice would be to avoid these!

[1] http://www.goalstructuringnotation.info/