Best Practices for NI TestStand Process Model Development and Customization

Process model development and customization is probably one of the most powerful features in NI TestStand because you can generalize concepts in multiple test sequences and promote heavy code reuse to reduce development and maintenance time. This document outlines best practices for customizing the process model. This document is most useful to those who have a good knowledge of basic process model development. Not understanding fully how to effectively customize process models can negatively affect system development, potentially during the later stages of the development cycle. Refer to the NI TestStand Reference Manual for more information about the different aspects of process models. If a project includes complex requirements, consider using outside expertise when the risk of the learning curve is not acceptable.

Process Model Architecture Overview

Testing a UUT requires more than just executing a set of tests. Usually, the test system must perform a series of operations before and after it executes the sequence that performs the tests. Common operations that define the testing process include identifying the UUT, notifying the operator of pass/fail status, logging results, and generating a test report. The set of such operations and their flow of execution is called a process model.

Even with this detailed description, the idea of a process model might seem abstract. Another way of understanding how a process model relates to the test itself is to compare the process model-test relationship to a sporting event. Picture a game or race among athletes that ends with a specific result as a test that runs on a specific unit under test. Advertising the sporting event, selling tickets to the event, broadcasting the event on TV, compiling complex statistics from the event, and writing a summary of the event are not part of the race or game play itself but are still part of experiencing the sporting event as a whole. Similarly, the process model includes auxiliary actions that occur at large during testing but are not part of the core testing itself. To extend the analogy, compiling statistics and writing event recaps represent database logging and reporting, respectively.

Process models set TestStand apart from commercial and home-grown monolithic test executives. Without a process model, each test sequence would need to provide the mechanism for common tasks, such as database logging and reporting, and each new test sequence created would need to repeat this code. You would need to repeat in each test sequence any changes to the operation of the common tasks. Using a process model to perform common tasks allows for increased modularity and reusability because you can make modifications to common operations in only one location. For example, in a system that tests multiple types of Outs with multiple test stations, each test sequence implements the actions specific to a UUT, and the process model includes requirements common at the system or station level.

Advantages of a Process Model Architecture

You can use the TestStand process models and related features to create an extremely powerful and flexible test application with minimal effort. The TestStand implementation of the process model simplifies development by using a modular architecture that minimizes the amount of code changes you need to make to affect every part of a test system. For example, several different test sequences can use a single process model to minimize development effort and improve maintainability in the future. Additionally, you can run a single test sequence within several different process models to gain increased flexibility when running in different environments. You can use the same test sequence for validation and verification tasks during development and then use the test sequence in a different process model for testing UUTs on the factory floor. The test process can change, but the tests executed remain the same. Using the TestStand process model architecture to develop a complete test system can save time as well as development and maintenance costs.

Process Model Components

The process model is a special type of sequence file that uses the following different types of sequences:

• Model Callbacks
• Engine Callbacks
• Execution Entry Points
• Configuration Entry Points
• Normal

Use the Model tab of the Sequence Properties dialog box to configure the types of sequences a process model file can contain.

Model Callbacks

A model callback is a special type of sequence that implements default behavior or acts as a placeholder. You can customize the behavior of a callback for each client sequence file that uses it. The action of customizing a callback behavior in a client sequence file is called overriding a callback. You define a model callback by adding a sequence to the process model file, marking it a callback in the Sequence Properties dialog box, and then calling it from the process model. You can use the Sequence File Callbacks dialog box to create a sequence of the same name in the client sequence file to override the callback in the model sequence file.

For example, the default TestStand process model defines a TestReport callback that generates the test report for each UUT. Normally, the TestReport callback is sufficient because it handles many types of test results. However, you can use the Sequence File Callbacks dialog box to define a different TestReport callback in a particular client sequence file to override the default TestReport callback.

Engine Callbacks

TestStand specifies a set of Engine callback sequences it invokes at specific points during execution. Use Engine callbacks to make TestStand call certain sequences before and after the execution of individual steps, before and after interactive executions, after loading a sequence file, and before unloading a sequence file. You can override Engine callbacks in the StationCallbacks.seq for Station Engine callbacks, the process model file for Process Model Engine callbacks, or a regular sequence file for Sequence File Engine callbacks. Use the Sequence File Callbacks dialog box to define Engine callbacks in a process model.

Execution Entry Points

Use Execution entry points to run test programs by calling the MainSequence callback in the client sequence file. The default TestStand process model provides two Execution entry points—Test UUTs and Single Pass. The TestStand User Interfaces include Execution entry points to execute sequences. By default, the Execute menu lists Execution entry points only when the active window contains a sequence file that uses the process model.

Configuration Entry Points

Configuration entry points configure a feature of the process model and usually save the configuration information in a .ini file the <TestStand Application Data>\Cfg directory. By default, the Configure menu lists the following Configuration entry points – Report Options, Database Options, and Model Options.

TestStand Sequential Process Model

This document uses the TestStand Sequential process model to demonstrate best practices. Some of these concepts also apply to the Parallel and Batch process models, which play significant roles in manufacturing environments. Refer to Appendix A, Process Model Architecture, of the NI TestStand Reference Manual for more information about the three default TestStand process models.

The Sequential process model uses the Single Pass and Test UUTs Execution entry points. Both Execution entry points call the MainSequence sequence of the client sequence file to execute tests for one UUT at a time. The entry points also share other actions, such as generating test reports and storing data results in a database. Additionally, the Test UUTs entry point allows users to identify the UUT in the PreUUT Callback and runs the same test on multiple units until users decide to stop testing. The Single Pass entry point tests only a single unidentified unit.

Figure 1 uses the main model callbacks to demonstrate the similarities and differences in the execution of both entry points.

Figure 1 – Single Pass and Test UUT Execution Entry Point Flow in Sequential Process Model

Common Process Model Modifications

Project requirements determine the modifications you must apply to a process model, but the most common changes include modifications to reports, test retry strategies, error handling and logging, station calibration routines, and UUT selection mechanisms.

Report Customization

Report customization is a typical process model modification. For example, an organization might want to add UUT subassembly serial numbers to the reports generated for most of the test sequences run on an assembled UUT. This modification can be made in a new process model callback that differs from the default report generation behavior.

Test Retries

In an automated testing environment, you might want the test system to automatically retry a test on a UUT when certain types of failures occur. Another use case is to limit the number of retries per operator before requiring supervisor intervention to determine the need for further retesting. You can implement these modifications as various conditions or looping on the MainSequence Callback file in the process model.

Error Handling and Logging

Test requirements determine error handling strategies. Typically, highly trained operators in R&D environments want to extract as much information as possible from errors and want control over system behavior when errors occur. In other use cases, subcontracted, fully automatic environments typically require start/stop inputs and red/green light outputs with appropriate debug logging to trace previous system and test error activity. In most cases, however, test framework requirements fall somewhere in between and use the same process model in different error treatment contexts.

You can use the ProcessModelPostStepFailure and the ProcessModelPostStepRuntimeError Engine callbacks defined at the process model level to generically handle errors from all client sequence files without requiring extra effort from the test sequence programmer. You can define the callbacks to use separate configurable error management modules to perform error logging and determine the test system reaction to the error.

Station Calibration Routines

You might need to confirm station calibration validity before executing test sequence files on a test station. These routines typically check for test equipment calibration validity based on an expiration date. Expiration of calibration triggers an error condition, warns the operator, and prevents the test from executing. You typically implement station calibration routines as part of the process model before calling the client sequence file.

UUT Selection Mechanism

The default TestStand UUT selection mechanism tracks only a serial number, but you can customize the process model to use a more complex UUT selection mechanism. For example, the test system might need to track a part number to provide custom testing based on part number. Refer to the Customizing the UUT Management GUI in Proligent™ section of this document for more information about a system that uses this kind of customization.

Process Model Development and Customization

The following sections outline some general guidelines to consider when you develop and customize process models.

Start with a Copy of the Original

Use a copy of the process model as a starting point for developing new process models or customizing existing process models so you can reuse large amounts of fined-tuned logic already implemented in the default process model.

If you do not need some of the default behavior of the process model, consider changing that behavior through configuration options instead of by modifying the process model itself. The default Configuration entry points are Report Options, Database Options, and Model Options, which you can use to disable reporting or database logging. For more control or to remove behavior in the process model, skip steps in the entry point instead of deleting the steps so you can more easily revert the change if needed.

When you start from scratch to build a process model or make significant modifications to a copy of an existing process model, you might lose default error management and other useful hidden actions in the process. However, creating a process model from scratch is a good learning experience to understand the structure, complexity, and details of a process model.

Overriding vs. Customizing Callbacks

Determining when to override callbacks in a client sequence file or customize the callback in the process model directly depends on the breadth of sequences that need the changed functionality. Override callbacks when the functionality in the callback is necessary only for part of all the products tested in the station. When you override a callback in a client sequence file, only the sequences in the sequence file related to the products execute the functionality. Customize the callback in the process model directly when the test system must guarantee that all products run the same functionality. For example, test systems that require stringent validation can easily validate that functionality for all products by implementing it directly in the process model.

Customizing Callbacks vs. Process Models

Customize the content of an entry point by adding or modifying callback sequences rather than by inserting steps directly into the entry point or by using a subsequence so you can isolate and keep track of the modifications made to the default process model. With this strategy, you can define a common behavior for all sequence files and still override each client sequence file. Customizing an entry point by using a non-callback, or normal, subsequence hides implementation details from the client sequence file.

You can use the Copy Step and Locals when Creating an Overriding Sequence option on the Model tab of the Sequence Properties dialog box to copy the original content of the process model callback into the client sequence file to help test sequence developers start with a default implementation rather than from scratch.

During development, you might think you need a second customized process model to implement customization differently from the current customized process model. Typically, it is more efficient to add more entry points to the current customized process model than to create a second process model sequence file. Adding entry points results in less copied code to maintain and fewer process model files to deploy. However, allowing multiple developers to share modifications to the same process model file requires proper source code management.

Callback Implementation Methods

You can implement a callback in the process model in the following ways. Selecting the method to use requires forecasting how client sequence files will use the callback.

• Implement a placeholder with no default functionality so the client sequence file can insert functionality if necessary. PreUUTLoop is an example in which a client sequence file can override the callback if needed. Typically, the client sequence file does not require data from the process model, which therefore passes no parameters.
• Implement a placeholder with no default functionality but pass parameters that control how the model behaves. ModifyReportHeader is an example that does not provide a default implementation but allows calling the client sequence file as needed. In this case, parameters that contain report-related data are passed to the overridden callback to make overriding easier and more obvious.
• Implement default behavior and expect overriding callbacks to implement altered functionality. TestReport is an example of implementing the default report generation for the whole system. However, client sequences can override and implement report generation differently, such as when you need a specific report format for a specific sequence file.
• Implement default behavior so the client callback can invoke the model callback to run the process model default implementation and the client callback can implement additional functionality. By default, PostUUT implements banners to display the result of the test sequence. A client sequence file can implement custom PostUUT behaviors and keep the banner by overriding the callback and still calling the process model implementation.

Using the Sequential Model for Parallel Testing

The main advantages of the Parallel and Batch process models are the GUI that controls the execution and the ability to generate cumulative reports for multiple sockets. In a production environment, however, a custom user interface can manage the execution, and the capability of joining multiple UUTs in a report might not always be an important requirement for the test station. In such cases, you can launch multiple executions of the Sequential process model instead of using the Parallel and Batch process models so you can take advantage of working with an easier-to-understand process model. A drawback to using this approach is that you need to properly propagate context, such as the socket index, so that each UUT knows which socket it is on. The Parallel and Batch process models implement this type of information passing by default. You need to add this kind of information to the parameter of a new entry point in the Sequential process model. You can add specific measurements to the sequences to complement the context for the reporting, such as UUTs tested concurrently, test socket, and other information related to the concurrent execution.

Analyzing and Prototyping Before Implementing

Avoid quick code-and-fix strategies because the process model is the heart of a test solution. Before you make a change, use breakpoints and watch the data during different executions to make sure you understand the purpose of a few previous and next steps to avoid interfering with the current implementation. You should also prototype your changes to validate the analyzed concepts.

For example, you might make a common mistake when you attempt to customize the report path. The Determine Report File Path step, which appears before the MainSequence sequence call step, uses information in the Locals.ReportOptions container to create a file path. TestStand stores the file path in Locals.ReportFilePath, which TestStand uses to write the report in a step called Write UUT Report. However, many people try to change the Locals.ReportOptions.ReportFilePath in the MainSequence instead, not realizing that this is not the actual path TestStand uses.

Isolate Process Model Changes

When you make changes to components in the process model, limit the scope of these changes to the callback or step where they are performed. For example, instead of modifying local variables in the callback, pass parameters to the model callback. Isolating modifications helps reduce future maintenance efforts.

Implementing Features in UIs vs. Process Models

Although it is common software architecture practice to avoid logic implementation in user interface modules, as a framework programmer, you might face the dilemma of implementing logic in user interfaces or in process models. The selection of the client sequence file, depending on the product flavor or UUT process state, is an example of functionality you can implement in the GUI instead of in the process model. In this case, you need to implement UUT management in the GUI because TestStand launches the process model on a specific client sequence file. You can pass serial numbers and other product-context information, such as product type, product revision, and socket indexes in multi-UUT environments, as parameters to the process model entry points and then change the PreUUT Callback to automatically fill the parameters with the supplied values.

Strongly Encourage Documentation

It is critical to fully document the requirements, architecture, design, and implementation of the process model because it is of utmost importance in an enterprise wide TestStand deployment. In addition to traditional documentation, use entry point and callback comments, step comments, and proper step and variable names to document the implementation. Also, use a source code management tool to track version numbers to match the TestStand sequence file version and to track comments between revisions.

Customize Entry Point Name Expressions

When you use multiple entry points, a common mistake is to forget to customize the entry point name expression. The entry point name is the name of the sequence itself. The entry point name expression is the name that appears in the Sequence Editor or in a user interface while using the entry point. Use the Entry Point Name Expression text box on the Model tab of the Sequence Properties dialog box to edit this value. The Entry Point Name Expression text box is visible only when the sequence you select is an Execution entry point. A default value, such as ResStr("MODEL", "TEST_UUTS"), uses the ResStr function, which features TestStand localization. Replace the value with another localized value or use a constant string expression that describes the entry point from the point of view of a user. Refer to the Localization section of Chapter 9, Creating Custom User Interfaces, of the NI TestStand Reference Manual for more information about localization.

Use TestStand Sequence File Versioning

Use and increment the sequence file version property as you make revisions to a process model file. Do not rely on the file disk information, such as file size or modification date, to track file revisions. You can also document changes in an external document or with each source code control revision. Use the Version text box on the General tab of the Sequence File Properties dialog box to view and edit the sequence file version value.

Diff Process Models when Upgrading

When you upgrade to a newer version of TestStand, check the differences between the former and new default process models. This task is more crucial when you base the customized process model on a copy of the original process model. If all the modifications were properly centralized and confined, it should be a relatively straightforward task to import process model customizations into the newer version of the process model.

Using StationCallback.seq

StationCallback.seq is a special type of sequence file that you can use to define global model callbacks and Engine callback behaviors. Callbacks defined in this Station Callback override any matching callback at the process model or client file level. The Station Callback level defines shared behavior among multiple process models. Because the Station Callback hides the implementation of a matching callback in the process model, call the matching process model dynamically to avoid hiding a process model callback implementation. The Customizing the UUT Management GUI in Proligent™ section of this document includes an example of using the Station Callback.

Customizing the UUT Management GUI in Proligent™

The NI TestStand Bridge for Proligent™ implements many of the best practices this document outlines. Proligent is a professional, enterprise-level solution that integrates many of the functions required for manufacturing and test, including advanced data management, system configuration, test automation, and real-time web reporting. Combined with TestStand, Proligent guarantees test process control, assembly tracking, and visibility over manufacturing. The NI TestStand Bridge allows TestStand to interact bi-directionally with Proligent.

A key element of Proligent is to link a serial number to a product type that has manufacturing processes with multiple operations, such as assembly, board-level test, pre-functional testing, and system test. In this context, identifying the serial number is more complex than using a simple text box, as in the default TestStand implementation. A modification for bridging Proligent with TestStand was to override all the default TestStand UUT selection GUIs with a custom Proligent GUI. The current UUT management GUI in TestStand uses DLLs written in LabWindows/CVI™. The source is available in the process model directory.

Averna created new, customized process model files – one for each of the default Sequential, Parallel and Batch process models in TestStand – by starting from copies of the default implementations because the intent of the bridge was to add functionality to the current implementation, not to start from scratch. By modifying the default process models and assuming that most people implement their own process models starting from the default ones, Averna provides an easier migration path to integrate with Proligent.

In the Sequential process model, the PreUUT callback has two parameters – ContinueTesting and the UUT container. The current implementation writes values to the ContinueTesting Boolean parameter to indicate if the user clicked the Stop button and writes the serial number entered in the text box in the UUT.SerialNumber string parameter. Implementing the same behavior with a different dialog box window was a straightforward task. Averna modified the PreUUT callback at the process model level to implement a generic implementation for all client sequence files.

In the Parallel and Batch process models, the interaction between the process model and the UUT dialog box is more complex. To override UUT management in those process models, the first step was to study the LabWindows/CVI source code located in the process model directory. The default GUI receives messages from the process model through function calls and replies by using the NI TestStand Synchronization Manager. Making significant alterations to this mechanism probably involves massive changes to the process model and presents more pitfalls than advantages. However, it is relatively easy to take the source code of the initial GUI and replicate its behavior in a custom GUI. Averna implemented this messaging replication to create a new GUI for the Parallel and Batch process models.

Also, the Proligent system needed to integrate as seamlessly as possible with any customer process model implementation. Additionally, it was necessary to implement context loading for a specific UUT and to store the result after the test sequence executed. Averna used the PreUUT and PostUUT callbacks in the StationCallbacks.seq file to implement this functionality by dynamically calling the original process model behavior to avoid hiding the process model implementation. Using this strategy, the default TestStand models work with Proligent out-of-the-box, without any further modifications. For some Averna customers, there is often a good chance that Proligent works out-of-the-box, even with a customized process model.

When National Instruments releases a new version of TestStand, Averna compares the process models for differences and updates the Proligent process models as needed.

Proligent process models also use methods outside the scope of this document to meet complex requirements. In fact, it is common to encounter unforeseen issues or more complex requirements during development that might require you to deviate from the guidelines in this document. When these situations arise, periodically refocus and rethink about the pertinence and coherence of customizations. It is also possible to implement what might become future best practices, as long as they prove to be flexible, robust, and maintainable well into the future.

Conclusion

TestStand process models implement generic behavior that you can override as needed. Test developers might take the process model behaviors for granted but should instead take the time to truly understand the non-trivial default process models. The best practices in this document provide an experienced point of view of typical customizations to make the modification process as smooth as possible.

Refer to Chapter 10, Customizing Process Model and Callbacks, and Appendix A, Process Model Architecture, of the NI TestStand Reference Manual for more information about TestStand process models. Refer to the NI TestStand Advanced Architecture Series for more articles on advanced TestStand topics.

About the Author

Mathieu Daigle – Averna

Software Engineer Mathieu Daigle has acquired over five years of experience in implementing best practices for clients in diverse industries. Mathieu is fully versed in LabVIEW™, NI TestStand™, Microsoft .NET, and Proligent™, The Test Engineering Platform™, as well as other technologies. He is a Certified TestStand Architect, a Certified LabVIEW Developer, and a Certified Professional Instructor. Mathieu holds a bachelor’s degree in computer engineering from Université de Sherbrooke.

Reader Comments | Submit a comment »