Guide to Effective Test Software Development with NI TestStand

Welcome to the Test Management Software Developers Guide. This guide is collection of white papers designed to help you develop test systems that lower your cost and increase your test throughput and can scale with future requirements. This paper provides guidelines for effective test software development with NI TestStand.

To view the other topics, visit the Test Management Software Developers Guide.

Introduction

Test management software is a set of tools designed to manage the many components that make up a test application. Test management software ultimately handles measurements from an external Unit under test (UUT). Measurements are analyzed and compiled into reports that describe the integrity of a particular UUT. In modular test system architectures, test management software uses a set of layered components. The most immediate layer consists of code modules written in application development environments (ADEs) such as LabVIEW, LabWindows/CVI, and Microsoft Visual Studio. In order to perform instrument control or data acquisition, code modules access one or more driver services such as NI-DAQmx, NI-488, IVI, and OPC. Each service accesses an interface with a specific hardware platform or bus including GPIB, serial, data acquisition, modular instruments, vision, and motion. The low-level layer consists of the sensors that are reading the measurements from the UUT.

Modular test system architectures ultimately yield faster delivery of products to the market, achieve greater product quality, and reduce development and production costs. A major benefit of modular software development is code reusability. Individual components of a test system, for instance a set of code modules that return measurements for a particular UUT, may be reused throughout multiple test applications. Code reuse reduces the amount of development time necessary for future projects. Developing a modular system also reduces the complexity of a project because each module can be produced and tested individually. Additionally, modular architecture permits more concurrent development, decreasing the overall production time. Finally, test management software takes advantage of the capabilities of multiple ADEs to provide easy integration of multiple development platforms.

Modular Test Systems Implemented in NI TestStand

NI TestStand, as shown in Figure 1, directly implements a modular test system architecture. NI TestStand is a management tool that interacts directly with code modules written in almost any ADE. An investigation of the relationship of NI TestStand with ADEs reveals how a modular test system is realized.

Initially, NI TestStand plays the role of a test system automation controller. All test system operations are automated through test sequences composed of steps. Steps execute sequencing operations or invoke code modules. Each code module performs a task consisting of measurement acquisition or nontest-related I/O. Nontest-related I/O includes tasks that do not directly relate to the integrity of a particular UUT such as reading a bar code scanner for a UUT serial number. NI TestStand is capable of invoking code modules developed in several ADEs. Each step invokes code modules directly through the set of NI TestStand module adapters. The adapters handle communication with specific module types such as LabVIEW VIs, C/C++ DLLs, ActiveX automation servers and .NET assemblies.

Additionally, NI TestStand separates UUT-specific test operations from system-level operations by distinguishing between UUT test sequences and process model sequences. Process model sequences perform all non-UUT specific actions including result tracking, report generation, database logging, and parallel execution. A process model also controls the execution of UUT-specific test operations and provides for looped UUT testing and concurrent UUT test executions.

ADEs are primarily used to develop code modules executed from NI TestStand. Code modules implement tasks such as measurement I/O, advanced data processing, and test-specific dialogs. The ultimate goal of a modular test system is to develop and maintain as little code as possible. In that respect, significant emphasis should be placed on developing highly modular and reusable code modules. The guidelines enumerated in this article provide a rubric for achieving a modular test system.

An additional task of ADEs is to develop operator interfaces for a deployed NI TestStand system. Operator interfaces provide a deployable test application that will execute on test stations. The application acts as an interface for any test operator executing tests. Operator interfaces are built from the open NI TestStand API and user interface components, each exposed through a set of ActiveX automation servers. An NI TestStand operator interface can be developed in any ADE capable of ActiveX functionality.

NI TestStand Features

NI TestStand includes several tools that facilitate modular test system development.

• NI TestStand Sequence Editor

The NI TestStand Sequence Editor is an application program in which you create, edit, execute, and debug sequences. The sequence editor gives you access to all NI TestStand features, such as step types and process models; it also features debugging tools familiar to other ADEs such as LabVIEW, LabWindows/CVI (ANSI), and Microsoft Visual Studio .NET. These debugging tools include setting breakpoints; stepping into, out of, or over steps; tracing through program executions; displaying variables; and monitoring variables and expressions during executions. Using the NI TestStand Sequence Editor, you to start multiple concurrent executions – you can execute multiple instances of the same sequence, or you can execute different sequences at the same time. Each execution instance has its own execution window. In trace mode, the execution window displays the steps in the currently executing sequence. If the execution is suspended, the execution window displays the next step to execute and provides debugging options.

• NI TestStand Operator Interfaces

NI TestStand includes several operator interfaces, each of which is a separate application program. These interfaces (see Figure 2), which are developed in LabVIEW, LabWindows/CVI, Microsoft Visual Basic .NET, C#, and C++ (MFC), are available in both source and executable formats. The NI TestStand Operator Interfaces are fully customizable. Like the NI TestStand Sequence Editor, with the operator interfaces you can start multiple concurrent executions, set breakpoints, and single-step. However, in the operator interfaces, you cannot modify sequences, and they do not display sequence variables, sequence parameters, step properties, and so on. The source code of the operator interfaces provides a starting point for customization or as a model for your own operator interface.

• NI TestStand User Interface Controls

The NI TestStand User Interface (UI) Controls are a collection of ActiveX controls for creating custom user interfaces in NI TestStand. These controls simplify common user interface tasks, such as displaying sequences and executions. The NI TestStand UI Controls facilitate the quick generation of complete, full-featured operator interfaces. You can use these controls in any programming environment that can host ActiveX controls.

• NI TestStand API

The NI TestStand API is a set of DLLs that are exported through an ActiveX automation server. The NI TestStand Sequence Editor and User Interface Controls use the NI TestStand API, callable from any programming environment, that provides access to ActiveX automation servers, including code modules you write in LabVIEW, LabWindows/CVI, and Microsoft Visual Studio .NET. All NI TestStand operations including sequence execution and development, user management, and report generation can be controlled programmatically through the NI TestStand API.

• Variables and Properties

NI TestStand stores data values in variables and properties. Variables are properties you can freely create in certain contexts. You can have variables that are global to a sequence file or local to a particular sequence. You can also have station global variables, which have values that are persistent across different executions and even across different invocations of the sequence editor or operator interfaces. The NI TestStand Engine maintains the value of station global variables in a file on the computer on which it is installed. You can use NI TestStand variables to share data among tests written in different programming languages, even if they do not have compatible data representations. You can pass values you store in variables and properties to code modules. You can also use the NI TestStand API to access variable and property values directly from code modules. Each step in a sequence can have properties. For example, a step might have a floating-point measurement code property. The type of step determines its set of properties.

• Report and Database Configuration Wizards

The Report and Database configuration wizards provide easy modification of both the format and data of test results. With the report configuration wizard you can select common formats including as HTML, ASCII, XML, and ATML. Additionally, multi-UUT tests can be compiled into a single report or separated into individual reports. With the database configuration wizard, you can select any database with an ActiveX Data Object interface. Moreover, the target structure and data are completely customizable from the wizard.

• User Management Wizard

With the User management wizard, you can control the privilege level of any user. Users may be added or removed from a list that is global to an entire station. Users can be quickly generated from a set of common profiles such as operator, developer, and administrator.

• Deployment Utility

The NI TestStand Deployment Utility is a wizard that simplifies test deployment. The deployment utility will automatically determine dependencies for test sequences and code modules. You can also build an installation kit that makes distribution to several systems easy. Moreover, the installation kit can include National Instruments drivers such as NI-DAQmx, NI-488, and NI-VISA, as well as run-time engines including LabVIEW and LabWindows/CVI. With the distribution kit, you can also deploy user profiles, station globals, and station settings.

• Process Models

NI TestStand installs with a sequential model and two concurrent execution models. Each model includes functionality for report generation, database logging, flow control, and multi-UUT or single-UUT testing. Additionally, the supporting code module source is provided with the process models. The source code provides an excellent example for developing process models or as a starting point for customization.

• Example Programs

NI TestStand is shipped with a large set of example sequences and code modules. The example programs cover topics such as controlling parallel executions, dynamic user management, report modification, database customization, and customizing sequences. The examples that are shipped with NI TestStand are an excellent resource for learning the NI TestStand API. Additionally, examples are implemented in multiple development environments for easier adoption by developers with differing backgrounds.

Guidelines for Effective Test System Development

Due to the many layers available in NI TestStand, it can sometimes be difficult to determine the practices that will minimize development effort while producing a scalable, reusable test system. For instance, should range checking be accomplished in a NI TestStand sequence or in a code module? At which level should new threads be spawned? Each decision yields different results.

Measurement I/O
Measurements may be acquired from code modules or in steps from a NI TestStand sequence. In either case, the measurements will be handled by NI TestStand. However, implementing measurement I/O in code modules provides more modularity. Initially, modifications to one code module will affect all steps that invoke the code module. In cases where a code module is reused several times, this feature can save development time. Additionally, the same code modules can be reused in future tests. It is much more difficult to find and copy specific steps that perform measurements. Finally, multiple related code modules can be used together in different combinations yielding more flexibility.

A common template for instrument I/O is a set of code modules. One module initializes the instrument, returning an instrument handle. The second module reads a measurement from the instrument. The third module releases the driver resources using the instrument handle. The instrument handle is stored in NI TestStand and passed to subsequent modules to access the instrument drivers. Separating the code module operations provides scalability. Multiple measurements can be made using the measurement module while still using only one initialization and one release module. If the initialization and release code were in the same module, the instrument would have to be reinitialized every time a measurement is made. NI TestStand provides a cleanup step group where the resource cleanup can occur. In case of a run-time error, this step group will still be executed. This template of measurement I/O produces robust and safe test systems.

Performing measurements in a step should only be done for one-time measurements. Using steps provides for quick test development. Additionally, performing tests inside of NI TestStand does not require an external ADE for development or run-time engine for deployment.

Limit Testing
Limit testing checks that a measurement fits within a certain range. This operation can be performed in a code module or a NI TestStand step. Checking limits in NI TestStand provides more flexibility for a test system. Limit checking in a code module requires either hard coding the range of a test or passing parameters that determine the range. In NI TestStand, limits can be stored with a particular step. Limits can be easily modified from within the Sequence Editor. Moreover, a single code module can be used with several different limits over many different steps. NI TestStand also provides tools to dynamically import limits for every limit test step from a file or database. Limits can be easily modified in a text editor without requiring changes to an existing test system.

Switching
Hardware switches provide a method of routing signals. Switches are often required for large test applications to ensure that the proper signals are read. With switches, you can also read several sensors on a relatively limited set of hardware. NI TestStand provides top-level control of switch hardware through the embedded use of NI Switch Executive. It is possible to route signals from any step using NI Switch Executive. Developers are provided with more flexibility by using the NI Switch Executive configuration in NI TestStand. External code modules often require recompilation resulting in a rebuild of a particular test system. With NI TestStand and NI Switch Executive, changes are relatively easy.

Information Display to Operators
Operators execute test applications on test stations. Test applications that display information to operators are often more powerful and flexible. However, it is necessary to consider where information display should occur. An operator interface is a test application that hosts the execution of test sequences. Should the information display occur in the sequence or in the operator interface? This often depends upon the type of test application. Test-specific items should be avoided in an operator interface, especially particular steps or sequences required for an operator interface to run correctly. This removes flexibility from the test system. Moreover, the operator interface cannot be reused for future test systems. Instead, the operator interface should act as a test application framework capable of executing many different test sequences.

The sequence layer is preferred for displaying test-specific information to the operator. In most cases, the information displayed will be test specific such as UUT information or test flow decisions. Loose coupling between the operator interface and sequence provides all the benefits of modular test applications, including concurrent development and reuse. In cases where it is necessary for the operator interface to display test information, NI TestStand provides user interface messages, which act as a communication protocol between the sequence layer and the operator interface layer. Operator interfaces should not rely on receiving specific user interface messages, however. This would ultimately tie the operator interface to a limited set of test sequences.

Reporting and Database Logging
In NI TestStand, reporting and database logging are handled in the process model. The process model is meant to separate system level operations from test-specific operations. Reporting and database logging are automatically implemented regardless of the test application. Development effort can be focused on specific test application problems, rather than recoding the same infrastructure. Moreover, modifications to the structure or data included in a test report are easily made through simple configuration wizards. NI TestStand also provides the source code for the three default process models included with the product. Therefore, customization of the existing framework requires relatively little effort compared to generating a process model from scratch. Customizations are also reusable for future test applications.

User Management Tools
User management sets restrictions on the functionality of a test application to ensure that test applications are executed in the intended manner. The NI TestStand Sequence Editor provides a user management tool for creating users, deleting users, and modifying privileges. The user management tool is the easiest method for modifying user information. Moreover, the tool generates a distributable file so that user information can be shared across several test stations.

NI TestStand also includes a set of functions for modifying users through the NI TestStand API. User information can be modified dynamically through the sequence layer, operator interface layer, or code module layer. Implementation at the sequence or code module layer is advantageous for automated user modification. Adding user management code to the operator interface offfers features such as adding new users or removing existing users.

Multithreaded Execution
Multithreading provides the ability to use resources and execute tests concurrently. In NI TestStand, new threads can be generated at the process model layer, sequence layer, or code module layer. NI TestStand provides two concurrent execution process models. The creation and destruction of threads is automatically handled by the process model so the developer can focus on test-specific items. Each thread generated in the process model is considered a test socket. Test sockets are threads that test a single UUT. With the process model, test sockets run concurrently, which can save total test time. The management of multiple reports and execution flow is automatically handled in the process model. Moreover, the number of test sockets is easily scaled through a model configuration wizard.

Threads can also be generated at the sequence layer in NI TestStand. Sequence layer threads provide more control, but require more overhead. Execution flow is handled completely by in the test sequence. Generating threads at the sequence level is excellent for handling concurrent executions that perform different tasks. Sequence level threads are also important if more control is needed over the lifetime of a specific thread. Generating threads from a code module written in external ADEs places the full burden of multithreaded applications on the developer. The developer is fully responsible for forking and destroying separate threads, which is potentially dangerous if a thread is not properly destroyed before the test application is completed. The top-level control of the sequence layer or process model layer provides more flexibility. Changes are easier to make at the test management level rather than individual code modules. Ultimately, generating threads in code modules requires more development effort and does not use the existing tools in NI TestStand.

Relevant NI Products and White Papers

National Instruments, a leader in automated test, is committed to providing the hardware and software products engineers need to create these next-generation test systems.

Software:

• NI TestStand Test Management Framework
• LabVIEW Graphical Programming Environment
• SignalExpress Interactive Measurement Software
  

Hardware:

• Modular Instruments (Oscilloscopes, Multimeters, RF, Switching, and more)
• Multifunction Data Acquisition
• PXI System Components (Chassis and Controllers)
• Instrument Control (GPIB, USB, and LAN)
  

White Papers
NI offers a Designing Next-Generation Test Systems Developers Guide. This guide is collection of white papers designed to help you develop test systems that lower your cost, increase your test throughput, and can scale with future requirements. To read the entire developers guide, you can: Download the PDF (90+ page) version or view the web-version of the Designing Next-Generation Test Systems Developers Guide.

Reader Comments | Submit a comment »