Architecting a Modular Test Software Framework

The rapid advancement in product design tools and growing trends for global design to manufacturing (GDM) are creating new opportunities for test managers and engineers to develop next generation test systems that greatly contribute to accelerating the product development cycle. The approach to test system development is shifting from building application specific, turn-key test systems to focusing on incorporating essential technologies for test to architect a modular test framework. The next generation test framework provides the ability to support the test requirements of multiple products and easily facilitates the insertion of new test technologies for additional test coverage in the future. Developing a robust, open test software framework is a key component of architecting a next generation modular test framework. This paper will discuss the importance of defining a modular test software framework, guidelines for developing your test software framework, and essential software technologies for automated test.

Industry Trends and Challenges

There are many factors (see Figure 1) to consider when defining the essential technologies and framework for your next generation test systems. Some of the factors are quantitative including cost, size, yield, and throughput while others are qualitative relating to growing business needs for increasing integration of design and test and overcoming the complexity of developing and testing the products of tomorrow on a global scale.


Figure 1. Key Factors Influencing Automated Test

According to a recent survey performed by National Instruments, test engineers and managers reported that the ratio of development costs versus capital costs is rapidly increasing with development costs often 2 to 10 times more than capitol costs. The breakdown of the average costs associated with building test systems today is shown in Figure 2. These results indicate more than 50% of the cost associated with building today’s test systems is related to software development (30%) and system integration (27%). The widening ratio of development costs versus capital costs is an interesting trend that indicates the strategic challenge to test managers and engineers to develop an improved test software framework to facilitate the insertion of new technologies and test coverage.


Figure 2. 2004 Test System Cost Analysis Survey Results (ni.com/daq)

The rapid rise in development costs is causing a profound impact on the ability for many test engineering groups to keep up with the accelerated product development cycle and globalization of design and test. The current approaches to test system development are generating far too many projects for today’s engineers to keep up with. According to a survey performed by Test & Measurement World magazine, the average test engineer is working on 7 – 10 projects a year with the average project lasting approximately nine months. The results from this survey pinpoint the cause of the pressures you may be experiencing today resulting from the challenge of working on multiple projects at any given time. Test managers and engineers must identify new approaches to test system development to effectively keep up with the increasing complexity of today’s products under test and the accelerated product development cycle.

Essential Technologies for Streamlining the Test Development Cycle

The key to streamlining test development and reducing cost is focusing on improving the efficiency of the design iteration, validation, and manufacturing test system development phases (see Figure 3). In order to drive improvements in these areas it is important to understand the latest technologies and strategies that can assist you in standardizing on a common test platform. Leveraging an integrated test platform across the product development cycle allows you to quickly test your designs in the prototype phase and minimizes the cycle times associated with supporting new products in the manufacturing test system development.


[+] Enlarge Image
Figure 3. Common Product Development Cycle

Defining a common test platform that can be used across design and manufacturing requires a technology oriented approach to system requirements. National Instruments recommends PC-based virtual instrumentation systems using a modular software and hardware architecture as the core platform for your common test needs (see Figure 4). Using open software environments and industry standard hardware platforms is also extremely important to ensure integration with other simulation and enterprise systems. The ability to support hybrid systems comprised of existing GPIB, VXI, and PC-based test equipment is also important for supporting specific test system requirements and reducing cost. Lastly, size does matter when defining a common test platform. Anything larger than a 3U form factor is difficult to share and effectively use within a design validation and prototype environment.


Figure 4. Recommended Technologies for Defining a Common Test Platform

Adopting Next Generation Test System Development Practices

Implementing next generation development practices is as equally important as adopting the essential technologies for test in your common test platform. Some of the general next generation development practices are listed below:

· Assign design, product engineering, and manufacturing test teams as equal stakeholders in product success
· Give equivalent attention and priority to the strategic planning in design and test of product
· Treat the success of the test strategy and implementation as an integral component of the overall product success
· Increase focus on necessary test coverage required for each phase of product development cycle

The underlying focus of these recommended practices revolves around increasing the strategic planning and accountability of test as an equal shareholder in the overall success of a new product’s introduction. National Instruments and other leading Fortune 500 electronic manufacturers have already begun treating the test strategy as equally important as the actual design plans for the new product being developed. This greatly encourages the design and test teams to work more closely together and increases the awareness of issues that might have otherwise gone undetected until further down the product development cycle when mistakes become very costly. Test engineers working more closely with the design teams can also help determine which tests are really required through the different testing phases (see Figure 5). This can greatly reduce the amount of development and maintenance efforts required across the entire organization.


[+] Enlarge Image
Figure 5. Planning Test Coverage for Product Development Cycle

Role of a Modular Test Software Framework

The test software framework determines the effective lifetime of your test system and is a crucial component for architecting a common test platform. A test software framework is the application of a modular test software architecture designed around the specific technical and business requirements of a given test system or organization. An example of a real-world test software framework is shown below in Figure 6. This example highlights the "Ultimate II" test software framework developed by National Instruments test engineers for use in our global manufacturing facilities.


[+] Enlarge Image
Figure 6. Ultimate II - Example Test Software Framework

The Ultimate II framework illustrates a strategically defined test software architecture that is used to test more than 100 different products. The common test framework is used for validation, manufacturing, and service testing. NI TestStand and LabVIEW are the core software tools used to develop the framework. Design and manufacturing engineers developing tests for the framework need only develop the test files and test code modules. The software framework provides commonality across all test systems and is maintained by the test engineering group.

Components of a Modular Test Software Architecture

The general components needed to architect a modular test framework are shown in Figure 7. A modular test software architecture consists of test management software, such as NI TestStand, to manage and execute the validation and manufacturing tests while communicating the test results back to the enterprise systems through built-in database integration. The test management software leverages popular application development environments (ADE) such as LabVIEW and LabWindows/CVI to create test modules that interface with test equipment through the use of measurement services which act as the driver interface to the equipment. Lastly, the switch management software, such as NI Switch Executive, coordinates the configuration and automation of multiple switch devices through an interactive route editor and single API for controlling all of the switch devices in the system.


[+] Enlarge Image
Figure 7. Core Components of a Modular Test Software Architecture

Guidelines for Architecting a Modular Test Software Framework

1. Leverage modular test software components to reduce initial development costs and increase the reuse and scalability of your test software framework. NI TestStand’s open software architecture is a popular choice for test management software (see Figure 8) and greatly reduces the effort required to implement a scalable test software framework.


[+] Enlarge Image
Figure 8. NI TestStand Test Management Software Architecture

Test management software, such as NI TestStand, provides a core architecture for your test system that eliminates the development of common test executive tasks. An important component of the test management software that you chose is the underlying architecture. NI TestStand provides a completely modular and open architecture that can be used “as is” off-the-shelf or used as individual components for designing your completely customized TestStand based test systems. The TestStand Engine is a powerful, multithreaded test engine with a complete API featuring more than 1,400 exported functions. The Sequence Editor provides the main test sequence development environment. The module adapters provide an open language interface to automate tests written in any language. The process models provide superior modularity between the test code and the system level functions that must be performed. Lastly, the operator interfaces are provided in source code in multiple programming languages for rapid customization to match your exact needs. TestStand test management software is a core technology for building today’s modular test system frameworks.

2. You can greatly simplify the development of test procedures under your new test software framework by separating the system-level and unit under test (UUT)-level test operations (see Figure 9).


[+] Enlarge Image
Figure 9. Separating UUT- and System-level Tasks

Separating UUT- and System-level tasks removes the redundant portions of code from the test modules and allows the Process Model to manage these tasks (See Figure 10). The resulting code modules are greatly simplified delivering decreased development and maintenance efforts.


[+] Enlarge Image
Figure 10. Separation of Tasks Using NI TestStand Test Management Software

3. Developing your tests as reusable modules rather than compiled applications is one of the most important guidelines. An immediate benefit of architecting your own modular test software framework with test management software and the programming language of your choice is the simplicity of developing individual test code modules for plugging into your test sequences. Once all of the system and UUT-level tasks are defined within the test management software environment, all that remains in the test code modules is the acquisition and measurement analysis. Avoid the temptation to build all of your tests into executables (*.exe’s) and instead save your test code modules as modular pieces of code such as *.VIs, DLL functions, etc. You should also remove all of the limit testing, report generation, and database I/O code from your test code modules. TestStand will handle all of these tasks for you and allow you to focus your test module development on acquiring the actual measurements. It is also recommended to store all of the parameter information at the test management software level and simply pass the data values for the inputs and outputs of your test functions to the test code modules at runtime. This further increases the modularity of your test code as shown in Figure 11. The TestStand module adapters provide direct interface I/O to transfer parameters between TestStand and the languages directly supported by the module adapters. This greatly increases the reusability of your test code.


[+] Enlarge Image
Figure 11. Example Test Code Module

4. It is also important that your new test software framework take into consideration the needs of the various users who will operate the test systems. A core component of a modular test software framework is the ability to provide multiple operator interfaces for the same test system. This requires developing your operator interfaces as clients to the test management engine rather than building elaborate graphical user interfaces that are attached to your specific test code modules. Developing your operator interfaces in this manner provides much greater scalability and consistency among test systems for the operators.

TestStand provides you with several built-in options for providing user interfaces specifically for engineers, operators, and technicians. The Sequence Editor shown in Figure 12 is the most commonly preferred user interface for test engineers developing test sequences. The Sequence Editor provides test engineers to all the functionality and tools needed to develop the most sophisticated automated test systems. The Sequence Editor also provides user management services preventing some users from using restricted functionality based on the privileges defined by your TestStand administrator.


[+] Enlarge Image
Figure 12. NI TestStand Sequence Editor

The operator interface is the preferred view for most operators and technicians working with test systems in a manufacturing environment. The Ultimate II operator interface shown Figure 13 is a slightly modified version of an example operator interface that is included with TestStand. All of the fully functional example operator interfaces that TestStand provides also include the built-in user management services. This allows operators to simply load and execute a test sequence while allowing technicians to login and debug the tests using breakpoints and the ability to actually step into the test code if needed. By architecting your modular test software framework around TestStand test management software, it is relatively easy to provide such services to the end users of your automated test systems.


[+] Enlarge Image
Figure 13. Ultimate II Operator Interface

Summary

Rather than defining a test system architecture around a specific product to be tested, today’s electronic companies are focusing on building a standardized test framework based on a one-time investment to provide the scalability and extensibility for use across multiple products, multiple manufacturing sites, and within different organizations in their company. The modular test software architecture provided by NI TestStand test management software saves these companies millions of dollars in combined development costs and allows them to develop their test systems in a fraction of the time while reusing tests across the organization.

To learn more about architecting your own modular test software framework using NI software and hardware products, contact your local NI field engineer and schedule an appointment today.

Reader Comments | Submit a comment »