NI TestStand Type Management Best Practices

NI TestStand types are reusable data structures you can use to define variables, properties, and steps. Types facilitate the creation of type instances by providing a structure that you can use as a starting point for creating new elements instead of creating elements from scratch every time. TestStand uses types to define elements that define the behavior of a test system, such as steps, and elements that store data, such as result properties.

Although TestStand types facilitate the development of test systems, the shared and modular nature of types requires you to follow correct type management practices to avoid unintended type version propagation or unexpected changes to step instance values due to step type value changes. This document describes how types function and suggests best practices for type management.

Categories of TestStand Types

TestStand uses different categories of types to store information and define the behavior of steps.  TestStand uses data types to store information during the development and execution of a test system, and step types define the behavior of a step and the results the step collects during execution.  Both data types and step types can be built-in and ship with TestStand, or they can be custom types that the user develops.

Data Types

Data types define the structure and default values of station global variables, sequence file global variables, sequence local variables, and step properties.  Each data type consists of a unique name, the type’s data structure, and the default values for the type properties.  Some data types can be very simple and consist of only a single property.  For example, a type that defines file paths can consist of a single string property that stores the path to a file.  Data types can also be more complex and contain a number of different properties and containers.  For example, an error data type can consist of three properties: a Boolean property in the type that can specify if an error occurred, a string value property that can contain the error message, and an integer property that can define the error code.

If you modify the structure of a data type, all the instances of that data type in sequences loaded in memory update to reflect the new data type structure.  However, changing the value of a type property in one type instance does not, by default, change the value in all instances.  You can think of the values in a data type as the default values for new instances.

Standard Data Types

Standard data types are a particular set of custom data types that ship with TestStand and that define a standard format for storing common types of data, such as error, path, or waveform.  You cannot modify most standard data types. 

Step Types

Just as you can create a variable or property from a custom data type, you can create a step from a step type.  Each step type consists of a unique name, built-in and custom step type properties, and default values for properties and step type operations.  All step types share a common set of properties that define the basic operation and data for all steps.  Use the Step Type Properties dialog box, as shown in Figure 1, to edit these common properties.

Figure 1. Step Type Properties dialog box for the Pass/Fail Test

You can add custom step type properties to store custom settings or results on the instances of the step type.

Step types can include three different categories of properties: instance properties, instance properties with a shared flag, and type-only properties. Each category determines whether the property values of a step instance reflect changes to step type property values.  The most common category of properties is instance properties, which only exist in step instances. Instance properties’ respective step types define the  default values of the properties.  Changes to the values of instance properties in a step type will not be reflected in any instance of the step type.  Two examples of instance properties are the default module and status expression.

In certain situations you might want instance properties to update their values when the step type values change.  Instance property values can be tied to step type values by enabling the shared flag for the property, as shown in Figure 2.  Instance properties with a shared flag, much like instance properties, exist in the instance of a step, but instead of storing their own value, instance properties with a shared flag consist of an alias to the step type property value.

Figure 2. Step Type Properties dialog box for the Pass/Fail Test step

Type-only properties differ from instance properties and instance properties with a shared flag in that type-only properties exist only in the step type and not in the step instances.  For example, the default name format property is a type-only property because TestStand uses it to define the initial name of a step while the actual step name is stored as a property of the step object.

How TestStand Stores Types

TestStand can store types in a number of different files.  The following is a list of the files used to store types.

1. Users file (Users.ini)
2. Station globals file (StationGlobals.ini)
3. Type palette files
4. Process model files
5. Front-End callbacks file (FrontEndCallbacks.seq)
6. Sequence files

The Users.ini and StationGlobals.ini files store types to facilitate distributing these two files.  If these files did not include type information, distributing them would require that you deploy a type palette file as well as the users or station globals file.  Type palette files contain type definitions to create steps, variables, or properties in a particular test station. Sequence files, including those that make up the Front-End callbacks and process models, contain the type definitions for all the types used in the file. Because TestStand stores types in the sequence file that uses the types, you do need to distribute a type palette file to distribute a sequence file.

How TestStand Loads Types into Memory

A TestStand application loads files that contain types in the order shown above in the list of files used to store types.  When a file is loaded, all the types defined in the document are loaded into memory as well.  The TestStand Engine loads sequence files into memory only if a user directly opens the sequence file or if an execution dynamically loads the sequence file.

Each type is uniquely identified by name and version number.  The Type Properties dialog box, as shown in Figure 3, defines a version’s type. 

Figure 3.Type Properties dialog box showing Version properties

A particular file might contain a different version of a type than the version of the type currently loaded in memory.  Because the TestStand Engine can only have one version of a type in memory at a time, the engine selects the version of the type to use and updates all instances of the type to match the one the engine selected.  By default, types are configured so that the engine selects the type with the highest version number. However, if either type is marked as modified (edited since the last time the version was changed), or if the type is configured to always prompt the user to resolve conflicts, TestStand prompts the user to select which type version to use. TestStand also prompts the user to decide the version to use when two types have the same name and version, but different structure or default values. When the user selects the version of the type to load, all instances of the type update to the version the user selected.

Type Conflicts

When TestStand tries to load a type with the same name but different structure than one that is already in memory, the TestStand Engine selects the version to use based on the type version, as described earlier in this document. If the engine cannot automatically determine the version to use, a type conflict occurs. At run time, a type conflict can result in a run-time error. At edit time, TestStand displays the Type Conflict in File dialog box, as shown in Figure 4, in which you can resolve the type conflict.

Figure 4. Use the Type Conflict In File dialog box to resolve type conflicts

Resolve the type conflict by selecting to load one of the two types, renaming one of the types, or cancel opening the file.  When you select the version to use, TestStand converts all instances in memory to match the type you selected. If you rename one of the types, TestStand modifies the instances in memory that refer to that version of the type to refer to the renamed version of the type.

Unwanted Type Version Propagation

Automatically updating types to the newest version facilitates the deployment of new types to users, but can also lead to unwanted type propagation.  One example of unwanted type propagation occurs when you create a new version of a step type that is not compatible with previous versions of TestStand.  When a sequence file that contains the new type loads in a previous version of TestStand that contains an older version of the type in a type palette, the new version of the type automatically propagates to the type palette and all files loaded in memory on that station.  Because the type is not compatible with the previous version of TestStand, you might experience undesired behavior when you run sequences that use the new type.

Another situation in which you might experience unwanted type propagation is when two types with completely different functionality share the same name.  When the TestStand Engine loads the two types, the engine assumes that the types are different versions of the same type and loads the type with the highest version number.  The type with the higher version number might have completely different functionality than the type with the lower version number, which can affect the functionality of the sequence files that contain instances of the second type.

A third type of unwanted type propagation can occur when two different developers independently customize the same type with different changes. For example, if one developer customizes the CommonResults data type and increases the version to x.01,a second developer customizes the same type differently and increases the version to x.02, and the TestStand Engine loads these two types, the type from the second developer is loaded because it has the higher version number.  Every step type uses the CommonResults data type, so steps that do not expect the changes to the CommonResults data type might operate incorrectly.

Best Practices for Type Management

Use unique organization IDs for type names

When you develop new types, you should prepend a unique organization ID to the type name to guarantee unique type names.  For example, all the names of the IVI step types that ship with TestStand begin with NI_.

Because the TestStand Engine can load only one version of a type with a particular name at a time, type conflicts occur with unrelated types that share the same name.  One developer might create a type called Wait that waits for one second and then ends execution.  A second developer might create another step called Wait that accepts as an input parameter the amount of time the step waits in milliseconds.  If a sequence that contains the first developer’s type comes into contact with a type palette that contains the second developer’s type, TestStand replaces one of the types with the other in memory based on the type version. 

Consider issues caused by the “save as previous” functionality of sequence files

TestStand allows developers to save sequence files in previous versions of TestStand.  Because sequence files contain types, a custom type might need to run on a version of TestStand that is previous to the one in which the custom type was developed.  Not all types run correctly on previous versions of TestStand due to incompatibilities. You can mitigate the issues caused by executing types in previous versions of TestStand by using the compatibility directories and by setting the earliest version of TestStand that can use a type.

One way to avoid incompatibility issues is to run types in earlier versions of TestStand than the one in which the types were developed.  Another way to avoid incompatibility issues is to create different versions of the type to run on different versions of TestStand.  When you save a sequence file to a previous version, TestStand searches a set of compatibility directories to find the type version that is compatible with the previous version in which you want to save the sequence file.

Store previous versions of custom types in the compatibility directories so sequence files saved as previous versions of TestStand include the correct type versions.  Compatibility directories are located in <TestStand>\Components\NI\Compatibility\<Version> and <TestStand>\Components\User\Compatibility\<Version>.  Add custom step types to the correct version directory in the User directory.

A second method to avoid incompatibilities is to limit the earliest TestStand version on which a type can be used.  If you do not set the earliest version of TestStand that can run a type, future versions of a type can easily propagate into type palettes and sequence files running on the previous version of TestStand. This unwanted type propagation is facilitated by the fact that types developed in newer versions of TestStand will most likely have higher version numbers than the types in previous versions.  The TestStand Engine automatically uses the types from the newest version of TestStand due to their higher version number.

To set the earliest version of TestStand that can use a type click the Version tab of the Step Type Properties dialog box, enable the Set the Earliest TestStand Version that can Use this Type option, and specify the earliest TestStand version that can use the type.

Use caution when customizing the CommonResults and CustomPrivileges data types

Developers should be very cautious when customizing the CommonResults and CustomPrivileges data types.  The two data types are used in every step type and user type, respectively, thus increasing the potential for unwanted type propagation and type conflicts.

Do not customize the CommonResults data type if you are creating step types that will be distributed as third-party products.  There is no way to predict whether the systems on which a third-party custom step type is used contains the customized CommonResults data type.  If a third-party custom step type customizes the CommonResults data type, the customization conflicts with any customizations  that have been deployed in the organization because only one version of the type can be loaded in memory.

If you customize the CommonResults and CustomPrivileges data types, assume that only one version of the type will be used in all test systems across your organization.  In addition, be aware that there might be type management issues if sequences or type palettes from your organization are used in other organizations.  If your sequences or type palettes are distributed, other organizations will either inherit your type customizations or conflicts occur if the other organization's version of these types is loaded into your files.

Developers must centrally manage any updates to both the CommonResults and CustomPrivileges data types.  Because a change to either of these types impacts every step type, a central entity or architect that understands the organizations’ test architecture must sign off on the changes to reduce type management issues.

Finally, developers must guarantee that any changes to the CommonResults and CustomPrivileges data types are backwards-compatible.  These two types propagate across all step types and user type instances in an organization, so legacy sequences use the new type, thus requiring the changes to the types to be backwards-compatible.

Use type-only properties to implement functionality and properties for all step instances should share

You should use type-only step properties to implement functionality that all steps of a certain type must share.  In particular, the three methods you can use to invoke a code module from a step type are the default module, Pre-Step subtep, and Post-Step substep.  Pre- and Post-Step substeps are type-only properties, while the default module is an instance property.  Because substeps exist only in the step type, all instances of the type refer back to the substeps in the step type.  Changing substeps of a step type automatically updates all existing instances of the step type.

NOTE: If you want steps to run only the substep code modules and you do not want users to specify their own modules, enable the Designate an Adapter option on the General tab of the Step Type Properties dialog box and select the <None> adapter.

Use the default module option only when the potential exists for each instance of a step to call a different code module.  The default module setting exists separately in every instance of the step, and existing step instances do not update by default if you change the setting of the step type.

NOTE: Do not set the shared flag on the default module.  TestStand does not support this use case.  If you want to share a code module across all step instances, use the Pre- or Post-Step substeps.

Some of the property values that you might want to share across all step instances are instance properties, such as the Status Expression and Load Option properties.  If you want to share instance property values across all step type instances, enable the shared flag on the properties.

Prompt users to resolve type conflicts if you want to guarantee that a sequence is using the correct type version

TestStand updates type versions automatically, so users might not be aware of the type updates in sequences and type palettes.  By prompting users every time, TestStand attempts to load two non-identical types with the same name, so you can alert users to unwanted type propagation and they can prevent incorrect behavior in their test systems.  When you create a type, consider enabling the Always prompt the user to resolve the conflict option on the Version tab of the Type Properties dialog box.

Make type palette files read-only for sequence developers and deployed systems

To prevent saving type palettes that have incorrectly updated type versions, set type palette files as read-only for developers who are not involved with type development and deployed systems.  Set the TypePalettes directory as read-only or select each type palette file independently and use the Windows File Explorer Properties dialog box to set the file attribute to read-only.

Although type palette files might update in memory with undesired versions of types during development or sequence execution, using the read-only setting prevents TestStand from saving these changes to disk.  Additionally, using the read-only setting gives you more control when deploying types because developers must choose to remove the read-only setting to import new types into the type palettes.

Disable automatic type conflict resolution for types which you do not want to propagate automatically.

By disabling automatic conflict resolution, you can help avoid unwanted type version propagation as described earlier in this document. On the Version tab of the Type Properties dialog box for a type or the Step Type Properties dialog box for a step type, select the Always prompt the user to resolve the conflict option to disable automatic type conflict resolution. To avoid a type conflict dialog box when using this setting, ensure that all of your files are saved with the version of the type you want to use. Select Tools»Update Sequence Files to launch the Sequence File Converter tool, which you can use to update your files.

Centralize type management

The tracking, management, and ownership assignment of TestStand types should be centralized in an organization.  One person or group in an organization should be responsible for keeping track of what types are being used throughout the organization.  This same entity should be responsible for determining the versions of types to use in development, and later to deploy to production.  Each type should also have a documented owner.

Keep track of what types are currently used in an organization and control the distribution of new types to reduce type conflicts and unwanted type propagation.  The central type management entity will be in a position to understand and control the outcome of adding new or updated types to test systems.  This central entity will also help reduce type conflicts by assigning an owner to each type who can better evaluate the consequences of modifying the type.

Conclusion

TestStand types facilitate test system development by providing components, such a step types that can be reused to implement common functionality.  The modular and shared nature of TestStand types poses certain challenges, such as those discussed in this document.  Follow the best practices outlined previously to help mitigate unwanted type propagation and type conflicts.  . Refer to Chapter 11, Type Concepts, Chapter 12, Standard and Custom Data Types, and Chapter 13, Custom Step Types, of the NI TestStand Reference Manual for more information about using types in TestStand.  Visit the NI TestStand Advanced Architecture Series to read other documents.

Reader Comments | Submit a comment »