GAIO CLUB ONLINE Vol.07:
DENSO Corporation Chassis Control Components Division - CoverageMaster winAMS Unit Test Automation Case Study

Please tell us what kind of development you have been involved in.

Mr. Makino:
Since the beginning I have been engaged in development environments around MPUs with a focus on functional safety MPU development. As one example, I developed tools to measure coverage using an MPU simulator, MC/DC measurement, and logging internal values of the MPU to conform to EPS (Electric Power Steering).

I'm now involved in safety concept construction and verification for an ASIL-D, the highest functional safety requirement level, MPU technology project related to redundant system with functional safety.

Mr. Watanabe:
Since 2005 I have been working in the EPS division, so it has been 13 years now. Initially I was in charge of developing control software for EPS-ECUs. At the time, I noticed that engineers around me performed a large part of unit testing by hand. I thought that this part could be made more efficient, and since I was interested in it, at some point I made and proposed a tool to automate by programming. In doing so, the engineers around me were pleased by my efforts. This became the start of my involvement in unit test automation.

Mr. Makino mentioned that he was engaged in MPU development involved in functional safety. Could you tell us what is a functional safety MPU?

Mr. Makino:
First of all, I would like to introduce the history of functional safety MPUs. Since 1990, I began developing software for ABS (Anti-lock Brake System) together with car makers. From that time on I was involved in "safety systems". Then I moved to the EPS division around 2002 and developed software for electric power steering. Utilizing that experience with my colleagues and OEMs, and I began planning and developing "Functional Safety MPUs" together with microcontroller semiconductors around 2004.

A "functional safety MPU" is an MPU that controls significant parts where safety is particularly important, such as "driving, turning and stopping" in automobiles.

At that time, there was no functional safety standard ISO 26262 for automobiles, so the meta standard IEC 61508 was one compass for making the the functional safety requirement. At that time, the single core was mainstream. But debate about the safety configuration with lock step type dual MPUs began, and I was deeply involved in how to make this architecture.

At that time, the thought that if a failure occurs, stop the system "fail-safes" were mainstream. But from around 2010, the thought that even if a failure occurs, continue control "fail operational" became mainstream increasingly. In DENSO too, from single channel system to dual channels system designing, then to "redundant functional safety". So multi-core MPUs, or 2 independent MPUs, are became popular safety requirements.

Mr. Makino:
Various new requirements are coming up for in-vehicle software. But in order to realize them, the reality is that they cannot be accepted in software development teams unless we inherit good points of traditional software. One of the main ideas is to realize these functions in hardware without imposing an overload on software for new requirements. For example, there are "Stack Separation" or "Memory Protection" as functions of safety design. If you try to write these functions in software, you would face the challenges of difficulty of design and performance.

Therefore, it is necessary to realize the functions in coordination with hardware. Thus, it is one of an important design technique to make a balance between realizing functions with "Fault-tolerant time", e.g., in case required to complete a reaction in short time, to use appropriate technique with hardware, and in other cases to create functions by software without timing constraint.I think that this is an interesting and worthwhile challenging part for engineers.  

Currently you are using CoverageMaster standardly in the EPS development team and other divisions as well. Could you tell me why it was adopted?

Mr. Makino:
As for software compliance of functional safety, we perform "Software Criticality Analysis", which is an analytical method for leveling risks of each functional element of software. Also, we perform analysis focusing on the completeness of the structure of software applying the "structure coverage" method. With regard to code coverage in particular, I began to think that the measurement of "MC/DC" proposed by NASA could be applied to DENSO EPS software.

At that time, I was considering a useful verification tool that has not only MPU but also peripherals as a simulator. However, MC/DC measurement cannot be realized only by the execution environment. It is necessary to embed a mechanism into the original source code and take out the execution result in the middle of the condition statement. I was looking for ways this could be done.

Then, I was introduced to "CoverageMaster" as it happens at a certain exhibition. I learned that it is a tool that can measure coverage from the actual machine code of the MPU and considered adopting the tool. This was in about 2007. After that, since an MC/DC measurement feature was added to the tool, it was linked with our in-house tool. During the time of our initial in-house tool, both the CoverageMaster ISS and the ISS made by microcontroller semiconductor were used. We confirmed that the execution results of both have had no inconsistency, and since consistency was sufficiently confirmed, currently we only use CoverageMaster in full-scale operation.

Could you tell me about the in-house tool, and how you automate unit tests?

Mr. Watanabe:
I am developing DENSO's in-house unit test automatic execution tool. The tool's name is "D-SPIDER". It is named “spider” because it catches bugs like a spider. The D is for "DENSO". Now I'll introduce the system mechanism.

First, the tester describes the name of the function to be tested, the input/output variables, the input values, and the expected values in the unique unit test data format decided by DENSO. Priority is given to coloring and ease of visibility, so an Excel XLSX file which has high format flexibility is used instead of a CSV file.

The user designates the created Excel file with the D-SPIDER UI and only needs to press the execute button. D-SPIDER calls CasePlayer2, analyzes the source code of the specified function, and generates embedded code for MC/DC measurement. It also calls the cross-compiler, compiles the embedded code for MC/DC measurement, and generates executable object code for CoverageMaster.

In order to compile the embedded code, it is necessary to modify the original makefile for the embedded code, but this mechanism is also automated.

In this way, D-SPIDER automatically generates an environment such as the object code for unit test execution, converts the test cases described in Excel to CSV for CoverageMaster, calls CoverageMaster, executes the unit test and performs MC/DC measurement.

The CoverageMaster execution result CSV file is written back to a DENSO format Excel file by D-SPIDER. This includes test case results that match with the expected values, test case results that do not match, the MC/DC coverage rate, and the coverage report. Using coloring in the Excel file, we devised it so that we can easily check the test results, such as by highlighting in red test cases where the expected value and the result were different.

From the user's perspective, it is a very easy interface for getting the test result, checking if the expected values match and getting the MC/DC percentage by automatically executing the unit test with a predetermined MPU simulator just by pressing a button. D-SPIDER is installed on the client PC of each tester, not the server, and it operates as a tool to drive the unit test.

How do you create test cases?

Each tester creates test cases by themselves based on function specifications. I have not automated this. DENSO unit tests have a long history since before automation systems like D-SPIDER have been in place, and we continue to operate using the same method for designing test data.

Even before D-SPIDER, I used an MPU simulator other than CoverageMaster. But, even if the mechanism of the tool changes, the testers will continue to use the same test design method. Even if the MPU is changed, it is the same. Therefore, the test case file is created based on the function specification, so that the same test cases can be continuously used.

What kind of merits were achieved by automating unit testing?

Mr. Watanabe:
If the source code has been changed, re-test by applying the same test cases to D-SPIDER. When testing with D-SPIDER, just push a button to automatically perform all the work, so retesting time is minimal. Through capitalization of test cases, we can conduct tests that check for degradation at the time of source code changes without burdening the engineer every time.

Mr. Makino:
D-SPIDER integrates all tasks performed by testers, such as preparing unit tests, execution, comparing results, etc. Until a while ago, there was no name for this tool, and in order to explain the tool, it was necessary to attach a long explanation such as "Tool to automatically performs unit test preparation to execution evaluation according to the user's environment". So, we attached the easy to remember name "D-SPIDER" named after a spider catching bugs.

It was completed as a tool for EPS around 2012, and gradually permeates throughout the company. Currently it extends to divisions other than EPS, with plans to expand to 6 divisions by 2019 March, and to 12 divisions by 2020 March.

Lastly, do you have any requests for GAIO?

Mr. Watanabe:
You may receive questions about CoverageMaster from in-house D-SPIDER users. Since CoverageMaster supports a variety of MPUs, it can be applied to many divisions. On the other hand, since it has many parameters using it can be difficult. For example, for users who only want to use certain features, something like an "easy UI mode" could make it very easy to use.

Also, it would be useful to have a display customization feature. For example, I think that it would become easier to spread to other divisions if the menus and buttons displayed were limited for users who want to use it more easily.

Since adopting GAIO's tool, user support has been satisfactory, without any particular points of dissatisfaction. GAIO responds to questions quickly and we are able to do our work without worry.

Mr. Makino:
In order to do unit tests, you have to think about test cases as the first input. This is the place where engineers' wits are put to the test, but there is the issue that the burden is still great. Therefore, I would like to incorporate an automatic generation feature that will help create test cases. For example, I would like to propose a feature to support the creation of test cases based on specifications, such as maximum and minimum values, boundary condition ±1 and intermediate values of equivalence partitioning for variables that are aspects of test cases. CoverageMaster already has a test analysis feature and an editor, but I would like to see a more advanced support feature for this.

Next, as a request from the viewpoint of governance, standardization is progressing in tools and development processes. To do this, tool chains between tools from different makers must be made. I think that if the interface portion of tools is standardized at each manufacturer it will be possible to develop new tools faster.

In addition, D-SPIDER realizes its function in cooperation with GAIO's tool. Once in the past, due to a CoverageMaster version update, there was a problem that the D-SPIDER system would not work. In order to prevent this kind of situation, I would like to get information on interface changes etc. at an early stage before the version update.