Why is Software Tool Qualification Indispensable in ISO 26262 Based Software Development?
Software Tools, to a large extent, automate the tasks needed to be carried out during the ISO 26262 mandated safety lifecycle. However, they may also introduce errors in the form of fault-injection or non-detection of faults in the software.
For instance, a source code repository used in the project may get corrupted or a static analysis tool may create false positives for errors in the code. To avoid such risks getting introduced in the software, ISO 26262 standard recommends tool evaluation and qualification.
Clause 11 of Part-8 of the ISO 26262 standard comprises the tool evaluation and qualification methods. As per the standard, tool qualification is a two-phased process that begins with the tool evaluation where Tool Confidence Level is determined and is then followed by Tool validation.
A natural question that pops in the mind is, ‘Do all the tools require qualification for every project?’
Well, a simplistic answer to that question is yes! A more nuanced answer would be that it depends on the particular use cases, software requirements and project specific workflow. It might be possible that a tool requires qualification in one project and does not in another. The question to be asked here is, “Will the tool introduce an error in the software?” If there is a possibility of error introduction, the tool must pass through the scrutiny.
What is ISO 26262 Standard’s Approach to Software Tool Qualification?
As already mentioned, Part-8 of the ISO 26262 standard contains detailed guidelines on software tool qualification. The objective is to get evidence suggesting that the tool in question is suitable for safety-related software development. The different use-cases analysed during the tool evaluation process are documented as evidence.
A Snapshot of one such document is shown below:
Certain terms such as Tool Impact, Tool Error Detection and TCL can be seen in this snapshot. We will explain them in detail as we proceed further.
These use-cases help in the evaluation of the following aspects:
- Does a malfunctioning software tool and its wrong output violate any safety-goal derived during HARA?
- What is the probability of a software tool detecting errors in the output derived from the tool?
The analysis of these use-cases helps derive the Tool Confidence Level (TCL). The TCL along with ASIL assigned to the project is used to determine the right method for tool evaluation. We will discuss the methods while explaining the work products required for ISO 26262 software tool qualification.
The first step in the process of tool qualification is the planning of the usage of the tool. From the configuration of the tool to its use cases and execution environment, a plethora of information is required at this step.
A gist of information that must be available to ensure proper tool evaluation:
- Description of features, function, and properties of the tool
- Environment required for the tool operation
- Known malfunctions of the tool and associated safeguards
- Known expected behaviour of the tool under anomalous working environment
Now comes the analysis part. This process is also called Software Tool Classification Analysis
This analysis is required to determine the Tool Confidence Level (TCL). In order to derive TCL, tool impact and tool error detection need to be understood.
Tool Impact: If a tool can introduce faults in the software or fail in detecting the errors, the tool is considered to impact the final software. In such a case, the Tool Impact is denoted by TI 2. If the tool is not impacting the final software output, its falls in the category of TI 1.
Let us understand with an example. LabView tool is a test automation tool used to validate the functionality of a controller such as Torque-speed characteristic. As it is an important task, this tool impacts the final software. Hence, it is classified as TI 1.
Tool Error Detection: The use-case scenarios of the tool decide whether there are enough safeguards in the tool to prevent and detect malfunctioning and erroneous output. This capability of the tool decides the error detection confidence level.
|Degree of confidence||Tool error detection level|
The ISO 26262 Functional Safety experts take multiple factors into consideration while assigning Tool Detection levels. For instance, an IDE called CS+ may have a tool error detection level of TD 2 (medium) due to its incapability of detecting all the errors in the code. Another code generator for which the output code is tested according to ISO 26262 standard, can be given a TD1 level (High confidence).
There is another scenario where a different software tool is used to verify the output from a software tool. In such cases, the interdependence between the tools is also considered.
Tool Confidence Level:
When TI and TD are established, Tool Confidence Level can be determined using the following schematics.
TCL 1 is the lowest level of confidence. Tools assigned this level do not have high impact on the quality of the final product. Therefore, tool qualification is not required.
TCL 2 and 3 are medium and high level of confidence, respectively. A tool qualification is required to demonstrate tool reliability.
And when we have the TCL, we refer to the ISO 26262 standard Part-8 Table 4 and 5 to understand which Tool Qualification Method applies.
Tool Qualification Methods Mandated by ISO 26262
The tool qualification methods prescribed in the standard for TCL 2 and TCL 3 are almost similar, with a slight difference based on ASIL value.
The tables will make it clearer.
Let’s understand these methods and how they are applied for ISO 26262 tool qualification.
- Increased Confidence from use: The confidence is derived from previous use of the tool under similar development environment and use cases. There is an increased evidence only when the similar version of the tool has been used for the same purpose with similar functional constraints. Also, the data collected from the tool’s previous usage must be adequate. If these conditions are not met, this method is not applied. As the software tool versions upgrade very frequently, this method is not among the most used ones.
- Evaluation of the tool development process: This method relates to the appropriate standard applied during the development of the tool itself. In most cases, the tool evaluation team does not get in touch with the tool developers and assess their development process. However, the tool developers do get their tools certified by organizations such as TUV.
- Validation of software tool: Validation of the software tool is a method that is required for higher ASIL compliance (ASIL C and ASIL D). Also, the test suites must be designed to evaluate both functional and non-functional aspects of the software. The standard states that the tool validation can be performed by either the tool vendor or the tool’s end-users.
- Development in accordance with ISO 26262 standard: This method is rarely applied as a majority of software tools are PC based and are not developed as per ISO 26262 standard.
*For ASIL B, C, and D, an additional step of confirmation review as per Part2 of the ISO 26262 standard must be carried out. Different levels of confirmation review are applied as per the ASIL.
Tool qualification is a joint task for both the tool vendors and the users. Although tool qualification must be performed by the user with respect to specific projects and use-cases, the tool vendors can do their bit to make the process easier and less time-consuming.