FeliCa Networks, Inc.

Mobile FeliCa is a contactless IC card technology developed for mobile phones. Simply holding a mobile phone over a reader/writer enables user authentication, electronic payments, and other functions, with expansion into various future services expected.
Because Mobile FeliCa is intended to be implemented across devices from all domestic mobile carriers and manufacturers, the number of deployed target devices is expected to be massive. Should a defect occur in the embedded software, it would not only impact a vast number of users, but the losses associated with addressing the issue would also be immense.
For this reason, rigorous quality assurance is required for Mobile FeliCa embedded software. This was one of the reasons why the Development Department at FeliCa Networks focused on module unit testing and path coverage testing to verify software quality.

The microcontrollers used for Mobile FeliCa are not limited to specific models, requiring compatibility with various platforms based on requirements. Although the software to be implemented is written in highly versatile C language, it is not guaranteed to operate seamlessly on every embedded microcontroller. Therefore, software verification, including unit testing, must be conducted on each individual microcontroller.
One of the main reasons for selecting GAIO's "CoverageMaster" as a module unit testing tool was its support for these multi-platform environments. "CoverageMaster" utilizes an Instruction Set Simulator (ISS) that supports a wide range of CPUs as its test engine, allowing test data assets to be used universally across all microcontrollers.
In module unit testing, turning test data into reusable assets for iterative use in subsequent development is key to maintaining software quality, which served as a major reason for selecting GAIO's tool.

Software bugs can be broadly categorized into normal-case and exception-case bugs. Most bugs in normal operations are surface-level and will eventually be discovered during the development process. However, many exception-case bugs occur only when multiple exceptional conditions overlap, making them undetectable through comprehensive testing.

For example, NULL pointer access is a typical bug in C software development. Even for a variable that would normally never be NULL, if it becomes NULL when a hardware failure occurs, it cannot be detected through comprehensive testing focused on normal operations.

In short, unless these scenarios are verified, a product could end up hanging merely from a partial hardware failure.

Module unit testing, which is a form of "white-box testing," comprehensively tests conditions without depending on system states such as normal or abnormal operations. By treating the NULL case mentioned above as a single test case, logic to handle exceptional hardware failures can be built into the software in advance.
By thoroughly conducting "module unit testing" upstream in development, the FeliCa Networks Development Department aims to establish a verification process that front-loads debugging—including exceptional cases that often become latent bugs—to complement subsequent comprehensive testing.

Now, let us discuss how the module unit testing tool is operated.
After deciding to adopt "CoverageMaster," the FeliCa Networks Development Department established operational standards for module unit testing across the entire development team.
Even with the tool adopted, an approach of simply distributing it and allowing developers to conduct tests based on their own personal standards would result in significant variations in tool usage, leading to inconsistent final software quality assurance criteria.
Therefore, standards were established for determining test cases for functions—such as setting at least three cases for conditional branches: "maximum value," "minimum value," and "threshold value"—thereby standardizing test items across all engineers.

A key characteristic of the operational approach at the FeliCa Networks Development Department is that developers themselves perform module unit testing. When developers create test data with "coverage" in mind for the functions they designed, it serves as an opportunity to review their own source code, frequently leading to the discovery of hidden bugs during test data creation. In particular, using CoverageMaster's "Coverage View" (shown below) to review execution paths within the source code for each test case enables verification of hard-to-spot errors, such as incorrect threshold values in if statement conditions (the boundary between TRUE and FALSE), ultimately enhancing source code reliability.
Compared to an approach where a third party mechanically performs unit tests and merely feeds back the results for developers to fix the code, unit testing conducted by developers themselves—integrated into the development process at FeliCa Networks Development Department—delivers many times the effectiveness.

The following is an overview of the software and test data developed in this project:

・Microcontroller: RISC / CISC
・Object code size: 50 KB
・Number of evaluated functions: 850
・Number of test cases: 7,500 (approx. 9 cases per function)

Focusing on conditional branches such as "if" and "switch" statements, three test cases ("minimum value," "maximum value," and "threshold value") are set for each branch condition within a function. On average, approximately 9 test cases are defined per function.

Regarding coverage criteria, C0 coverage (verifying whether source lines are executed at least once) is measured. Simultaneously, for conditional branching, validation requires that expected values for the three aforementioned cases are met correctly.

All subfunctions called by the evaluated functions are replaced using the stubbing feature of "CoverageMaster." This prevents execution from entering nested function calls, ensuring that testing remains completely self-contained within the function under evaluation.

Unit testing for this project was conducted in two phases. In the first phase, a preliminary trial test was conducted to establish standards for stub configuration methods and finalize the testing approach before executing simultaneous testing across the board.
The headcount and duration required for testing are as follows. The testing personnel worked on a full-time basis rather than handling dual responsibilities.

1. Preliminary trial testing: 5 people, 5 days
2. Simultaneous test execution: 7 people, 15 days

The unit testing results in this project are as follows:

・ C0 coverage rate: 99.9%
・ Branch condition testing (3 cases): 100%

The C0 coverage rate did not reach 100%, leaving a small portion of unexecuted code. The FeliCa Networks Development Department investigated the source code that was omitted from evaluation. This was due to the default case of a switch statement, where corresponding data could not be set based on the code's logic. By analyzing and fully understanding all such causes, it can be said that virtually 100% C0 coverage was achieved.

The number of defects detected prior to integration testing through this unit testing effort is as follows:

・ Number of defects: 90 (including those found during development reviews)

The FeliCa Networks Development Department has compiled these defect causes into a database and analyzed their classification. Most of them were defects in sections corresponding to exceptional conditions, many of which were highly likely undetectable during integration testing.

In this development project, the goals were to introduce "CoverageMaster" to establish a foundation for multi-platform module unit testing and to turn test data into reusable assets—both of which were successfully achieved.
One of the major future challenges is determining how to leverage this valuable test data asset, which enables checking for exceptional conditions, in future development. The next goal is to carry out development while maintaining software quality through efficient operations, such as automating the testing process. Another challenge is the test data management method. Managing test data as an asset is one of the essential requirements for maintaining software quality.
GAIO has received these items for consideration, including potential feature updates.

In this article, we presented a case study where adopting GAIO's module unit and coverage tool achieved results in a short period. We will continue striving to improve our products based on your valuable feedback. (Photo: Members of the Development Department, FeliCa Networks, Inc.)