Evaluating a Test Executive -- Feature Comparison Matrix

National Instruments TestStand combines a high-performance test execution engine with off-the-shelf features and incredible flexibility for use in a variety of applications from high-volume consumer electronics, telecommunications, and automotive testing, to complex ATE systems for the military and aerospace industries. This document is intended to assist you in making the important decision of whether to buy an off-the-shelf test executive or build your test executive software in house. This document will help you conduct a head-to-head comparison of available commercial test executive software. The comparison matrix highlights features to look for and questions to answer when evaluating test executive software. Each table in the document addresses a group of test executive or test system features. You can use this matrix both as a tool in your test software evaluation process and as an overview of some of the more important features of TestStand. This comparison matrix covers the following feature categories:

Test Execution

Does the Test Executive Meet the Following Requirement?	TestStand	Competitive Test Executive or In-House System
Platforms
Windows 3.1	No. Windows 3.1 does not support multithreading.
Windows 9x	Yes
Windows 2000	Yes
Windows NT	Yes
Windows XP	Yes
Mac OS, HP-UX, Solaris 2	No. However, sequences and tests developed on these platforms with the LabVIEW Test Executive toolkit are upgradable to run under TestStand on any Windows platform. This is also true for LabWindows/CVI Test Executive toolkit on Solaris 2.
Test Code Module Formats
Dynamic Link Libraries (.DLL)	Yes
LabVIEW Virtual Instruments (.VI)	Yes
ActiveX Servers (.DLL and .EXE)	Yes
Object Files (.OBJ)	Yes
Static Libraries (.LIB)	Yes
Stand-Alone Executables (.EXE)	Yes
HT Basic Files (.PRG and .BAS)	Yes
Development Environments
LabVIEW	Yes
LabWindows/CVI	Yes
Visual C++	Yes
Visual Basic	Yes (ActiveX servers)
Visual J++	Yes (ActiveX servers)
Borland C++ Builder, Delphi, or any other environment that can generate dynamic link libraries or ActiveX servers	Yes
HP VEE	Yes, through DLL interface. HP VEE examples available online.
HT Basic	Yes. Version 8.0 and above (Windows only).
Java Scripts	Yes, through DLL interface. Java examples available online.
TCL Scripts	Yes, through DLL interface. TCL examples available online.
Perl Scripts	Yes, through DLL interface. Perl examples available online.
Execution
Does the Test Executive handle multithreaded execution?	Yes. TestStand can easily run any number of executions in parallel using either Batch or Parallel models. You can independently debug or terminate each execution. Each execution runs in a separate operating system thread.
Can an execution programmatically launch other executions that run in parallel?	Yes
Can test sequences synchronize with each other when they run in separate threads?	Yes. TestStand includes a set of synchronization tools to notify other threads of their requirements.
How fast is the Test Executive?	Execution speed depends on the types of code modules you call and the number of options you enable. To help you determine the execution speed you can expect in your application, TestStand is shipped with a set of benchmark sequences. There is also a Profiling tool available online to help you identify which tests are taking the most time to execute.
Sequence Files
Can the test executive contain multiple test sequences?	Yes
Can the test executive contain sequences that automatically run when the sequence file loads and unloads?	Yes
Can the test executive contain variables shared by multiple sequences?	Yes
Can you automatically link sequence files to specific process models?	Yes
Sequence Flow Control
Can you establish pretest conditions?	Yes. Test steps can specify the conditions that must be true before they can execute. You can base these conditions on previous step results or the values of variables and step properties.
Can you establish post test actions?	Yes. Test steps can specify an action to take based on step results or any other condition. Actions include branching, breaking, terminating, or calling a subsequence.
Can you specify looping?	Yes. You can configure test modules to loop based on any condition.
Is a label step provided as branch target?	Yes
Does the test executive offer explicit Goto branching?	Yes
Variables
Can test sequences define variables which share information between test steps?	Yes. You can define any number of variables at the sequence, sequence file, and test station levels. Variables can be numbers, Booleans, strings, arrays, or user-defined objects and can be shared across threads.
Can you access variables from test code modules?	Yes. You can access all TestStand variables and properties within any test code module, regardless of the language in which the module is developed. You also can pass variables directly to DLL functions, ActiveX server methods, and subsequences.
Debugging
Can you set sequence breakpoints?	Yes
Can the test executive break based on test results?	Yes
Can the test executive step into subsequences and code modules?	Yes
Can the test executive step out of subsequences and code modules?	Yes
Can the test executive step over subsequences and code modules?	Yes
Can the test executive interactively set the next step?	Yes
Can the test executive run or loop selected tests?	Yes
Does the test executive include a variable browser/viewer and watch expressions?	Yes
Can the test executive break, resume, or terminate a selected sequence execution or all active executions?	Yes
Can the test executive step into LabWindows/CVI DLLs and source files?	Yes
Can the test executive step into LabVIEW VIs?	Yes
Does the test executive honor breakpoints in Visual C++ and Visual Basic?	Yes
Remote/Distributed Execution
Can test sequences call subsequences on remote machines?	Yes. TestStand uses Microsoft DCOM to support remote execution of sequences. You can call any number of sequences on any number of remote machines.
Can you pass parameter values to and return parameter values from remotely executed sequences?	Yes
Test Limits
Can you store and edit test limits in the test steps to which they apply?	Yes. This is the default if you do not use external limit files.
Can you import and export test limits to external files?	Yes
Which limit file formats are acceptable?	TestStand can import, export, and view comma and tab delimited text limit files and Microsoft Excel spreadsheet limit files.
Can you dynamically load test limits from external files at run time?	Yes
Can you import and export test limits to databases?	Yes
Can you dynamically load test limits from a database at run time?	Yes
Error Handling
Are hardware and software initialization and cleanup steps separated from test steps?	Yes. Each sequence has a Setup and Cleanup step list to handle initialization and cleanup. Each sequence also has a Main step list where you place the steps that test your UUT.
Do test sequences automatically cleanup properly if an error occurs?	Yes. Even if an error occurs, TestStand runs the cleanup steps for the active sequences to return the system to a known state.
Are access violations and other fatal system errors in test code modules trapped and converted into step execution errors?	Yes
Can you configure test steps to ignore errors?	Yes
Can you break on errors for debugging?	Yes. You can enable the Break on Run-time Error option to assist you in your sequence debugging.

Sequence Development

Does the Test Executive Meet the Following Requirement?	TestStand	Competitive Test Executive or In-House System
Sequence Editing
Does the sequence editor offer standard Windows editing features?	Yes. The TestStand sequence editor offers drag-and-drop editing, right-click context menus, cut/copy/paste, multiple item selections, and toolbars.
Can you view multiple files simultaneously and drag and drop steps and variables between files?	Yes. The TestStand sequence editor presents a standard multiple document interface similar to applications such as Word and Excel.
Can you add comments to sequences and sequence steps?	Yes. In TestStand, you can place comments in sequence files, sequences, steps, variables, and properties.
Types of Sequence Steps
What types of steps can you use in a test sequence?	TestStand allows you to define your own types of test steps. However, TestStand comes with a set of predefined step types that are sufficient for many tasks. These defined step types are as follows: Action – Calls a test module that performs an action and optionally returns error information. Numeric Limit Test – Calls a test module that returns a numeric measurement and compares the measurement against the limits you specify. Multiple Numeric Limit Test – Calls a test module that returns multiple numeric measurements and compares the measurements against independent limits for each measurement. String Value Test – Calls a test module that returns a string value and compares the string value against the string value you specify. Pass/Fail Test – Calls a test module that returns a Boolean pass or fail result. Sequence Call – Calls other sequences as subsequences, separate threads, separate executions, or remotely. You can pass values to and receive values from the sequences you call. Statement – Performs calculations with variables. Message Popup – Displays user-defined operator prompts. Call Executable – Calls an external executable and allows you to pass command line arguments. Property Loader – Loads test properties from an external file or database at run time. Goto – Branches to a step you specify. Label – Specifies destination for Goto steps or post action branches. You do not have to branch to label steps, but doing so makes your sequence easier to maintain. Database – A set of steps for low-level manipulation of data and databases. Synchronization – Facilitates communication and control between application threads and executions. IVI – Instrument control steps which expose full functionality of standard instruments.
Reusable Test Sequences
Can any number of parent sequences call a test sequence?	Yes
Can sequences accept and return any number of parameters?	Yes
Can you call sequences reentrantly (required for multithread safety) and recursively?	Yes
Can a sequence dynamically select which subsequence to call at run time?	Yes
User Management
Does the test executive offer user login and logout?	Yes
Does the test executive offer optional user privilege enforcement?	Yes
Does the test executive offer a user template editor?	Yes. New user templates provided for operator, technician, developer, and administrator. You can modify these templates or define your own.
Can the test executive automatically use Windows system/domain login?	Yes
Can the test executive define new privileges?	Yes
Can the test executive modify or replace user login and logout modules?	Yes
Can the test executive check and enforce user privileges in custom operator interfaces?	Yes
Can the test executive access user privileges in step preconditions and in test code modules?	Yes
Source Control
Does the test executive support source code control (SCC)?	Yes. TestStand implements source code control through the Microsoft SCC API. This provides built-in integration for Microsoft Visual SourceSafe, Perforce, MKS Source Integrity, and Rational ClearCase. You can easily check files in and out from the TestStand Workspace and Project window.
Can you differentiate sequences for code tracking?	Yes. TestStand has an advanced graphical differencing tool that shows all differences between any two sequences.
Backward Compatibility
Is the current version backward compatible with previous versions of the software?	Yes
List applicable migration paths.	Sequences and tests from the LabVIEW and LabWindows/CVI Test Executive Toolkits can migrate to TestStand. TestStand provides sequence converters that upgrade the sequence files from these previous packages to TestStand sequence files. Third parties can easily plug-in additional converters to support migration from other legacy environments.

Reporting/Result Management

Does the Test Executive Meet the Following Requirement?	TestStand	Competitive Test Executive or In-House System
Reporting
Does the test executive generate ASCII text reports?	Yes
Does the test executive generate HTML reports?	Yes
Does the test executive generate XML reports?	Yes. TestStand is shipped with a full XML example.
Can you configure the report content?	Yes
Can you configure the location and naming of report files?	Yes
Can you customize or replace report generation on the test station as a whole?	Yes
Can you customize or replace report generation on a per sequence file basis?	Yes
Can test code modules add additional information to the test report?	Yes
Can reports contain graphics and hyperlinks to diagnostic or repair procedures?	Yes. You must use the HTML or XML report formats.
Result Collection
Are results automatically collected for each step?	Yes. TestStand automatically collects the results for each step, including user-defined steps that contain extended result information.
Are results accessible from sequences, code modules, report generators, database loggers, and other tools?	Yes. You can access TestStand results in a sequence or from components written in any language.
Can you disable result collection?	Yes. You can disable result collection at the step, sequence, or test station level.
Can code modules add additional result information?	Yes. A code module can add any amount of additional information to the results for a step. This information is accessible in report generators and other modules.
Database Connectivity
Can you automatically store test results in database tables?	Yes
Which databases are supported?	All ADO and ODBC compliant databases are supported.* * The default schema that the TestStand process model uses to store test results is compatible with Oracle, Access, and SQL Server. You may have to change the default schema before you can use other databases to store test results. Refer to the Database chapter in the TestStand User Manual for more information.

Customization Features

Does the Test Executive Meet the Following Requirement?	TestStand	Competitive Test Executive or In-House System
Operator Interfaces
Can you customize or replace the operator interface program?	Yes. TestStand includes operator interfaces written in LabVIEW, LabWindows/CVI, Visual Basic, and Delphi. Each operator interface comes with full source code.
Which languages and environments can you use to develop operator interfaces?	You can use any language or environment that can call the TestStand ActiveX API. This includes LabVIEW, LabWindows/CVI, Visual C++, Visual Basic, and Visual J++.
Can different operator interfaces execute the same test sequences?	Yes. Operator interface programs call the TestStand API to execute sequences. Thus, any operator interface program can execute any sequence file.
Can users select the viewer application that the operator interface programs use to display report and limit files?	Yes
Test Executive Engine API
Does the test executive engine export a user-callable API?	Yes. TestStand is an open system that exports an ActiveX API that you can call from test code modules, operator interface programs, and third-party components. The TestStand API is similar in structure to the APIs exported by applications such as Word and Excel.
What languages can call the test executive API?	Any language that can call an ActiveX server. This includes LabVIEW, LabWindows/CVI, Visual C++, Visual J++, and Visual Basic. TestStand includes C and C++ header and source files for calling the TestStand API from LabWindows/CVI and Visual C++
Does the Test Executive API come with online help?	Yes
What functionality does the test executive API export?	All the functionality required to build operator interface programs such as the initiation, control, and monitoring of sequence executions. All the functionality required to enable custom tools to load, save, create, inspect, and edit sequence files. All the functionality required to enable test code modules to get or set the values of variables and properties, to detect termination requests during lengthy operations, to report status messages or progress indications to the operator interface, and to inspect or alter the state of execution.
Definition of the Testing Process
Does the test executive hard code a model of the testing process?	No. TestStand defines the testing process model as the sequence of actions that surround the execution of a test sequence. This includes operations such as identifying the UUT, displaying pass or fail notifications, generating test reports, or logging test information to a database. In TestStand, the occurrence and order of such actions in the testing process is defined by a sequence file that is designated as the station process model file. You are free to modify or replace this file.
How many standard test executive models are supplied?	Three – Sequential Model, Batch Model, and Parallel Model.
Can you modify or replace the actions that the test executive automatically performs when it runs a test sequence?	Yes. The process model defines actions such as identifying the UUT or generating a report. You can add to, change, or replace the actions that define your testing process by editing the process model sequence file. In addition, you can redefine in each individual test sequence file an action in the process model file that is marked as a callback. For example, you can change or replace how a sequence file report is generated without altering the reports for other sequence files.
Custom Sequence Editor
Can the developer customize the Sequence Editor?	Yes. Certain aspects of the TestStand Sequence Editor are available to customize. The TestStand API also provides the ability to create your own sequence building/editing environments in the same way you create operating interfaces. A development system license is required for any environment that creates, edits, or builds sequence files.
Custom Step/Test Types
Can developers create reusable types of test steps that appear as if they are built into the test executive?	Yes
Can custom step types define their own data, results, and run-time behavior?	Yes. You can create steps that accept custom inputs, return multiple results, and perform special processing. You also can create the editing dialog box with which a user configures instances of your step type.
In which programming languages can you develop custom step types?	Any language or environment that TestStand supports for test code module development.
Can custom step types define source code templates to aid users in creating test code that is called by the step?	Yes. You can create templates for each code format TestStand supports.
Can you customize the step menu and appearance in the test sequence?	Yes. TestStand provides a menu editor utility to customize your development menus and allows you to insert custom icons for each step.
Localization Provisions
Are user-viewable text strings stored in separate files for each language?	Yes. TestStand stores all strings used by the sequence editor and operator interfaces as a set of ASCII files. You can add new language files without recompiling or reconfiguring any TestStand components.
Can you view and edit string files in a standard text editor?	Yes. All string files are in a standard .ini format.
Can third-party components define new strings and redefine existing strings?	Yes. Third parties can supply their own string files so that their components can display localized strings.
Are multibyte character sets possible?	Yes
Is it compatible with Japanese Windows?	Yes
Are product manuals available in multiple languages?	Some manuals are available in Japanese.

Documentation

Does the Test Executive Meet the Following Requirement?	TestStand	Competitive Test Executive or In-House System
Documentation and Online Help
Which manuals and online help files are provided?	Getting Started with TestStand Manual (printed and online) TestStand User Manual (printed and online) TestStand Programmer Help – API Reference (online) Converting from LabVIEW Test Executive 2.0.1 to TestStand (online) Converting from LabWindows/CVI Test Executive 5.0 to TestStand (online) TestStand Run-time Operator Interface Manual (online and source) Session Manager Help (online) TestStand Process Models (online) IVI Step Types (online) TestStand Course Manuals (printed, available separately) TestStand Application Notes (Developer Zone) Online Example Programs (Developer Zone)
Documentation Generation
Can you automatically generate test sequence documentation?	Yes
What documentation formats are available?	Text and HTML.
Can you modify the documentation format?	Yes. The documentation generator is a component that calls the TestStand API to access the contents of sequence files. The source code is provided. You can modify or replace the documentation tool.

Support and Training

Does the Test Executive Meet the Following Requirement?	TestStand	Competitive Test Executive or In-House System
Support, 7 a.m. to 7 p.m. (central)
Is telephone technical support available?	Yes. Free telephone support. Free e-mail support. Free online support.
Are technical discussion forums available?	Yes. National Instruments maintains a free TestStand technical discussion forum monitored by product developers. Join the forum at exchange.ni.com. An independent forum exists at testexec@listserv.tamu.edu.
Is there a knowledge base and file library on the web?	Yes. ni.com/support/teststandsupp and on the Developer Zone at zone.ni.com.
Who is responsible for documentation, testing and long term software maintenance?	National Instruments maintains and supports all documentation and releases new versions when necessary. Our large user base results in more complete product testing.
Training
Are training courses available?	Yes. There are two training courses on TestStand taught regularly in Austin, Texas, and around the United States. Courses in local languages are available throughout the world. An Advanced System Design course is also offered throughout the year. Information on the TestStand training courses is available online at ni.com/custed.
Are courses available onsite?	Yes. You can book onsite TestStand courses anywhere in the world.
Are course materials available separately?	Yes
Is extended training available?	Yes. NI Consulting Services (NICS) can work with your developers to define system architectures and train them on implementation issues.
Are certification levels available	Yes. There are two levels of TestStand certification: TestStand developer and TestStand architect. You can find more information about obtaining TestStand certification online at ni.com/custed .

Open Industry-Standard Architecture

Does the Test Executive Meet the Following Requirement?	TestStand	Competitive Test Executive or In-House System
Third-Party Access
Can you integrate third-party or in-house developed tools into the test executive?	Yes. TestStand is a platform for interchangeable, transportable test software components. Third parties can build add-on tools and integrate TestStand into full test systems.
Hardware Access
Is the test executive compatible with industry-standard instrument drivers	Yes. TestStand accesses instrument and hardware I/O drivers through IVI steps, direct DLL calls, or through test modules written in any of the supported code formats.
Does the test executive work with GPIB, RS-232, VXI, PXI, and plug-in instruments?	Yes, through test modules. LabVIEW, LabWindows/CVI, and Measurement Studio can handle all major instrumentation hardware
Does the test executive work with switching devices?	Yes. TestStand offers robust switching capabilities through seamless integration with the NI Switch Executive. TestStand also offers a built-in IVI Switch step type for controlling any IVI-compliant switch directly from TestStand. You also can perform your switching needs through code modules called from TestStand.
Vendor Expertise and Experience
Is the company experienced in the test market?	National Instruments has been a part of the test and measurement market since 1976 and has supplied the test executive software market since 1993.
Are trained integrators available?	NI's Alliance Program includes expert TestStand integrators throughout the United States and the world.

Purchasing Options

Does the Test Executive Meet the Following Requirement?	TestStand	Competitive Test Executive or In-House System
Packaging
Which configurations are available?	TestStand Development System – Comprehensive, customizable test executive. Includes customizable operator interfaces and sequence development environment. TestStand Base Deployment Engine – Includes TestStand execution engine and support files for deploying your system on a target computer. TestStand Debug Deployment Environment – Includes TestStand Deployment Engine and extended sequence debugging capabilities. Also includes a license to use LabVIEW or LabWindows/CVI, or Measurement Studio on the target computer for debugging purposes only. Developer Suite, Professional Test Edition – Comprehensive suite of tools for building a test application, including the following elements: - TestStand Development System - LabVIEW – Graphical Programming - LabWindows/CVI – C development environment and compiler - Measurement Studio – Tools for Visual C++ and Visual Basic - NI Switch Executive – Configuration-based switch management software - IVI Instrument Driver Library

12 ratings | 3.08 out of 5

Print

Reader Comments | Submit a comment »

Calling TCL Scripts from TestStand
I can't find any document or examples about the TCL script language support. For example, there only has the Perl DLL, but no TCL one. Do NI support the TCL script language? BTW, I found there have a TCL lib called "GPIB-Tcl" on URL of "http://gpib-tcl.sourceforge.net/" . Will NI hardware support this lib?
- mike, Taiwan. mikeduh@vntek.com.tw - May 25, 2006