Production Test Management Software -- Build or Buy?: A Financial Comparison Using TestStand 2.0

Software for running a production test system is the control center for automated manufacturing organizations. This software, often referred to as a test executive, must meet demanding requirements for performance, flexibility, and connectivity to be effective. Today, especially in the highly competitive consumer electronics, telecommunications, and automotive markets, many manufacturing organizations realize that a test executive can be a key competitive edge. As manufacturing technologies improve to meet larger production volumes and quality standards tighten to deliver higher yields with less rework, the test executive software controlling the entire system has become a critical component. Weighing these factors, production managers and test engineers do not make the test executive software decision lightly. This paper outlines some of the specific factors involved in defining a test software strategy and reviews the advantages and disadvantages between building your own test executive system and buying a software product to handle these requirements.

Test Executive

A test executive is a high-level management system for organizing, executing, and reporting software modules that perform functional tests on products during the manufacturing process. Test executives generally have the following two modes or functions:

• A development mode for defining testing process or test sequences
• A run-time mode for executing sequences, displaying results, and generating reports on these test sequences

In most cases, the actual test programs that control the instrumentation hardware are separate software modules written in either a test programming language, such as LabVIEW or LabWindows/CVI, or a general-purpose programming language, such as Microsoft Visual Basic or Visual C++. By following a model that separates the test modules, which are often specified or developed by product design engineers, from the execution system, which is developed and deployed by test or production departments, you have a modular, flexible architecture that can be supported and maintained.

Text Executive Functionality

The following list outlines specific tasks handled by a test executive.

• Unit Under Test (UUT) identification (for example, serial number scanning)
• Test result displays (for example, pass/fail banners)
• Sequence execution display (for example, operator interface)
• Sequence execution control with the following options:
  - Breakpoints
  - Single-step execution
  - Step into/Step over
  - Pause/Continue
  - Interactive looping
  - Test preconditions/Conditional branching
• Test result evaluation with the following options:
  - Numeric test limit comparison
  - String value comparison
  - Pass/fail by test
  - User-defined comparison/evaluation
• Reporting in the following formats:
  - Local files
  - Databases
  - Distributed/Web-enabled displays
• Analysis (for example, SPC/Yield statistics)
• User management (for example, password-protected access)
• Sequence development environment with the following features:
  - Editor for specifying tests in a sequence
  - Branching/looping logic
  - Editor for specifying test limits
  - Ability to integrate test code modules in variety of forms, such as EXEs, DLLs, libs, objs, and VIs
  - Source code control
  - Parallel execution models
  - Direct instrument connectivity
  - Database connectivity

  Build Versus Buy
  

The Case for Building
A test executive controlling a production line must be integrated with a variety of different subsystems throughout the plant. Manufacturing and testing processes can vary widely between different companies, between different facilities within the same company, and even between different products within the same plant. For these reasons, it is challenging to find an off-the-shelf, ready-to-run test executive that exactly meets your requirements. Many operations, like the ones in the following list, are specific to a particular production line and cannot be addressed by a shrink-wrapped package.

• How are UUTs identified? Bar code readers? Vision systems? Operator entry?
• Which UUT handler systems automatically move products through the line?
• How advanced are the operators running the production floor?
• Which type of user interface is required?
• How much debugging/troubleshooting will be done on the production line?
• Which access levels are available for different users in the plant?
• How should the test results be stored?
• Which database formats are used?
• What should the test result reports look like?
• Who should have access to the reports electronically, and how?
• In which development tools or file formats were the tests developed?
• How should the test executive interact with the individual test modules?

Because each of these questions typically has a different answer for each company, test engineers or production specialists need to be directly involved in the development of a custom test management solution. Because of the importance of a test management system, many engineers feel that they cannot afford to give up control over the development of such a critical component.

The Case for Buying
Most test executives used on production lines today are the products of custom, in-house development groups. Many companies are now realizing that these home-grown test executives are difficult and expensive to support and maintain. With home-grown test executives, companies are trapped when key developers involved in the projects leave or move on to other projects. Another problem is that in-house development teams spend millions of dollars designing and implementing test management software frameworks, instead of focusing their resources on testing products. Because of these problems, companies are looking for an off-the-shelf product, with standard support, training, and continuous upgrades and improvements, that they can buy rather than build themselves.

TestStand – Ready-to-Run, Yet Customizable

Off-the-shelf test executive products cannot adequately meet the wide variety of testing requirements in the industry. In-house-developed test executives redirect many valuable resources away from core competencies of the company and can be difficult to support and maintain. Custom-developed test executives from system integrators or outside development companies can leave a manufacturer hostage to these integrators and exposed to exorbitant fees when updates and upgrades are needed.

TestStand from National Instruments is a ready-to-run, yet customizable test executive, designed to deliver the cost benefits of a purchased product while maintaining the flexibility of a custom-developed solution. TestStand is a high-speed, multithreading test engine capable of parallel execution of test modules written in any test language, such as LabVIEW, LabWindows/CVI, Visual Basic, and Visual C++. The engine is designed to meet the most rigorous throughput requirements of high-volume production systems. In addition, with the TestStand sequence development environment, users can define advanced sequences with conditional looping, branching, and execution flow to handle the most complex testing scenarios.

The TestStand execution engine calls a number of separate code modules, or callback routines, for handling specific tasks during the testing process. These modules include code for serial number prompting, pass/fail banner display, report generation, database storage, and system initialization and shutdown. These modules are included in source code with the standard product. Using this modular approach, a test developer can customize specific components of the TestStand execution system without writing the entire system from scratch. Users can modify the code included with the TestStand product or completely replace modules with their own code.

Test Executive Implementation Cost Comparison

Many developers undertake the task of building their own test executives before they fully realize the magnitude of the task. The following table outlines specific development tasks required for building a test executive. An estimate of the time required to implement each of these tasks is included, compared with the time it takes to implement the same task using TestStand. The actual development time estimates are not important. The important points here are the identification of the development tasks required, and the significance of the development investment required to complete these tasks.

Note: The development time estimates for custom solutions shown below are based on developing a specific solution or implementation. Test engineers developing a test executive will have specific ideas and objectives that they are trying to meet with their implementation. In most cases, a specific solution is just that, a very specific, minimal implementation to get the job done, leaving out any additional flexibility or functionality that is not immediately evident. TestStand is implemented with a very general, flexible approach, enabling many different engineers to customize it to meet their specific requirements. Therefore, the time estimates shown below for custom solutions in no way reflect the development effort invested in TestStand.

Task Description	Development Days
	Custom	TestStand
Execution Engine
Custom Operator Interface built with Visual C++ Load, display, and run sequences; controls for executing and debugging sequences; pass/fail banners, operator prompts, and so on	40	5 Use the example operator interfaces in TestStand as your starting point
Localization of the User Interface and Sequence Editor for Global Standardization	15 This number assumes that experienced translators are available	5 TestStand exports all strings to .ini files for easy translation
Parallel Execution Engine Architecture for loading and running test programs in parallel; passing/retrieving data for test programs, executing branching/looping/sequencing logic, storing internal values during breakpoint/single-step execution	150 This number will vary based on the number and variety of compatible test languages/file formats	0 TestStand calls LabVIEW VIs, LabWindows/CVI DLLs, Visual Basic Automation Servers, HT Basic subroutines, and executables
Remote Execution Capability Ability to run and monitor tests on remote machines locally and globally	50 This number assumes detailed knowledge of developing distributed computing	0 Built into the architecture of TestStand
Developing Step Types Call steps of certain implicit functionality independent of language being used	15	0
Result Evaluation Built-in limit comparisons, user-defined evaluation capabilities, user-defined parameters for test code	15	0
User Management Password-protected access to test executive capabilities, environment for setting up/managing user information	5	0
Reporting
Generating Test Results Various formats (ASCII, XML, HTML, Excel)	15	3 Use the example code in TestStand as your starting point
Database Connectivity Select results, store results in various formats for Oracle, SQL Server Access, and other ODBC & ADO databases	10	1
Off-Line SPC Analysis Yield Statistics such as Pareto and Cpk	10	2 TestStand includes basic SPC routines and display capabilities
Front-End Data Collection
Serial Number Scanning	2	1 TestStand has internal variables defined. Add serial number acquisition code
Sequence Loading	5	0
Test Parameter & Limit Loading Dynamically load test limits from an external file or database	5	0 TestStand loads limits and parameters from ASCII files, Excel files, and databases
Development Environment
Sequence File Editor Tools for specifying test programs; defining looping, branching, and conditional execution logic, specifying parameters to be passed to tests, specifying test limits, and so on	100 This could vary significantly depending on the complexity of the sequence logic and variety of compatible test languages available	0
Source Code Control and Workspace Window Architecture for viewing complex projects containing numerous files and subjecting these to standard source control methods	25	0 TestStand uses standard MS SCC API for source control
Documentation Both the development tools and the operator interface must be documented with hard-copy manuals and online help	50	2 TestStand is completely documented. Users may need to customize the documentation for the operator interface if they make changes to the standard operator interface. The operator interface documentation is included in Word source files for easy editing.

Cost Analysis
When evaluating the cost of developing custom software, it is important to realize all of the costs associated with an employee. For instance, even though the cost of an engineer may be calculated by the hour, or day, it is rare that this includes intangibles, such as benefits, health care, and pension, or the cost of office tools such as a computer, a desk, and office space. When all of these aspects are included, the average cost of a capable software engineer can easily reach $150,000 per year. However, in the analysis that follows, we will assume that the cost of an engineer is $100,000 per year, or roughly $400 per working day.

Development Option	Development Time (Days)	Development Cost at $400/day
In-House Development Generic System In-House Development Application Specific (approximately 50% of the cost of a generic system)	512 256	$204,800 $102,400
TestStand Purchase Customization Total	31	$2,995 $12,400 $15,395

Developing Application Modules

Once the shell of the test executive is complete, the engineers need to develop the application modules that stimulate and measure the UUT. Once again the choice of the development environment is critical in reducing the cost of testing. In many case studies, it has been proven that test-specific rapid application development tools, such as LabVIEW and Measurement Studio, can save substantial amounts of time and effort. With these tools, studies have shown that it is possible to develop from five to 50 times faster than with conventional programming languages. In addition, these tools are often adopted in research and development and validation, so that code can be reused from one department to another.

Associating a value with these tools is difficult. But assuming an improvement of only five times over traditional code creation on a single project that may take 25 weeks (or 125 working days), you would see the following savings:

Hence, we find a saving of at least $40,000 on a single project, which can grow very quickly as engineering resources are added or the project expands.

Instrumentation Costs

When defining the cost of a test system, the software cost is only part of the equation. Hardware is required to measure the UUT, provide stimuli, and power the device. In the past, every test head had its own set of instrumentation; however, with TestStand this is no longer true. TestStand has the ability to run numerous tests in parallel using the same set of instruments, by coordinating the usage with switches and advanced synchronization tools. With this capability, it is possible to dramatically cut costs by sharing expensive test equipment, such as GSM or UMTS test boxes, across a number of UUTs and augmenting them for simpler measurements with less expensive instruments such as DMMs and oscilloscopes.

Maintenance and Support Concerns

Developing your test management solution is only the beginning of the task. Software is not a one-time cost, but an ongoing effort. As new technologies emerge, adjustments must be made to the software to take advantage of these technologies. To remain viable, ready-to-run packages purchased from a software vendor must be updated by the vendor to run on the latest operating systems and computers. Custom-developed software must be updated and maintained by the end-user or third-party developer on a contract basis. Many times, this maintenance effort can become a major undertaking and consume months of development time simply to port the same features to a new operating system. Today, most companies are moving toward Windows NT or Windows 2000 as the operating system for running advanced production test systems. However, porting software from an older operating system such as Windows 3.1 or DOS can be a painful experience. The operating system shift from 16-bit to 32-bit has ramifications throughout the application; the user interface is completely different, and the low-level device driver model is completely different as well.

Again, associating a definitive cost on maintenance and support is impossible because every case is different. However, we assume that the overhead of maintaining an in-house test executive will require the effort of 15 to 20 percent of the test development department, whereas the overhead for TestStand would be substantially lower, at around five percent of the department, because fewer customizations are needed.

Assuming that a typical test department at a global organization may involve 100 people, we can develop the following cost reduction:

Hence, it is possible to see the ongoing cost of test, which is often hidden or neglected, and the value that an externally supported test executive can offer.

Training Costs

Training is another expense that many companies overlook. Many companies are looking for a standard production test package that they can use throughout the world, in many different manufacturing facilities. When standardizing on a software package, you also must provide detailed training and documentation to all users. A custom-developed test executive requires more than just software development if it is to be used throughout several facilities. It also requires printed and online documentation, standard training classes, technical support via the telephone and the Web, and local experts who can answer questions when problems arise. As a standard, ready-to-run package, TestStand includes all of these services. Although the actual cost of training varies between organizations, we can see the cost of developing and delivering training by making the following assumptions. In this example, we will assume the organization needs to train 20 engineers.

Summary

Building a custom test management software architecture is an extremely difficult and expensive undertaking. The system must satisfy a wide variety of needs across many different high-volume manufacturing industries. TestStand is an off-the-shelf product that provides an architecture for building a highly customized solution that is extensible and easily upgradeable. As we have seen in the examples in this paper, by saving hundreds of thousands (or even millions) of dollars at manufacturing organizations, TestStand can reduce costs and increase profitability.

Reader Comments | Submit a comment »

Learning curve
Because of course, C++ and Perl are trivial and anybody can develop a test sequencer in 500 hours starting from scratch! I think NI's comparison accounts for TestStand learning time in the integration cost.
- May 22, 2007

Cost Analysis
TestStand is well known for having a steep learning curve, yet you do not address this in your cost analysis.
- May 16, 2006