NI TestStand Shared Drive Deployment Reference Architecture

The deployment of test systems is one of the most important pieces of test framework development, and one which is often overlooked. Deployment is especially critical to an effective test strategy in organizations where there is a high mix or high volume of products. To deploy an NI TestStand system, you must identify the different components to deploy, determine their dependencies, and then package them into a deployable solution. Identifying test system components and their dependencies can be challenging due to the fact that not all components and their dependencies are explicit.

Once you create a deployable solution, you have several options for deploying that solution to one or more test stations. This document discusses a deployment strategy based on a shared network drive. This strategy can reduce software maintenance costs, especially for test systems spread out at multiple or international locations, large numbers of deployed test systems, or test systems requiring frequent updates. In addition, deploying systems using a shared network drive can result in cost savings on new test systems for new products due to code reuse and standardization. .

Authors: Roberto Piacentini, Hjalmar Perez - National Instruments

Shared Drive Deployment Architecture

Shared drive deployment architecture concentrates file storage on a shared network drive rather than a local hard drive. In this architecture, test stations map to a folder on a single network drive which contains the necessary files to execute a test solution. In this approach, test station configuration is standard and it does not vary between test stations. All test files are stored on the shared drive, and access to them is controlled by regular network access.  Figure 1 illustrates the major components of shared drive architecture.

Figure 1: Shared Drive Deployment Architecture

The Master File Repository contains the current version of each file along with a copy of older versions of each file. You can use a source code control application to provide file management functionality.

The Development Side includes the components required to support the needs of the test development group to develop, test and debug test files. The Development Shared Drive is updated by the Master File Repository, which provides the version of the files that test developers need.  Test developers use local Development Machines to develop and debug test files, and then update the Master File Repository with new files or new versions of existing files. The shared drive allows test developers to test their changes and make them available to other developers without committing them to the Master File Repository until they are verified.

The Production Side includes components which support the execution of files on Production Machines. The Production Shared Drive is updated by the Master File Repository to use a specific version of the files, usually the most current version. Production Machines can then access this Production Shared Drive to run these test files. On the Production Side, changes to the files cannot be loaded from the Production Shared Drive into the Master File Repository. This reduces the risk of accidental changes to a file that can cause tests to fail.

Depending on the resources available, it is possible to develop other models which combine the functionality of multiple machines into one. For example, you can combine the Master File Repository and the Development Shared Drive or you can set up test stations  to access the current version of all files directly from the Master File Repository.

The design of a shared drive deployment strategy does not end with deciding to use a shared drive for deployment. You must also make tactical implementation decisions based on your specific business needs. Some key business considerations include whether  all test systems are running the same version of the test software, what test file resources must be deployed to test stations and which resources should be stored on the shared drive, how to implement change management control, how to handle incremental updates, and how to manage required differences between development and production test stations.

This document discusses the concepts and best practices for a deployment architecture based on a shared network drive.

Introduction of Terms and Concepts

Before we address the best practices for shared drive deployment, we will discuss common terms and concepts used throughout this document.

Main components of an TestStand Test System

A test system is a set of hardware and software components used to test a particular product. In this document we will focus on the software aspect of a test system. The main components of a TestStand test system are engines, drivers, TestStand configuration files, TestStand components, the user interface, and test code. In shared drive architecture, these components can reside either on either the local hard drive or the shared network drive. 

It is important to understand the characteristics of each component and determine whether that component should reside locally or on the shared drive. For example, drivers are tightly related to operating system files, so they cannot reside on the shared drive.  For components without restrictions, important considerations include load time, change control management requirements, code reuse opportunities, and maintenance costs.

Engines

Engines are software applications that drive the execution and functionality of a program. They act as translators by converting high-level instructions, which are easier for humans to understand, into low-level machine code. A TestStand system can use a number of different engines.  With the exception of the TestStand Engine, you must install engines on the systems that utilize them.  Table 1 describes some of the most common engines used in TestStand systems. 

Engine	Associated Files
TestStand Engine	Sequences (.SEQ)
LabVIEW Run-Time Engine	Virtual Instruments (.VI) Virtual Instruments Libraries (.LLB) Dynamic  Link Libraries (.DLL) Executables (.EXE) created with LabVIEW
LabWindows™/CVI ™ Run-Time Engine	Dynamic Linked Libraries (.DLL) Object Files (.OBJ) C files (.C) Executables created with LabWindows/ CVI
Microsoft .NET Common Language Runtime	Assemblies and Executables created with .NET Framework ADE (.DLL, .EXE)

Table 1: Engines Used in an NI TestStand System

Drivers

Drivers are software libraries that instruct the operating system and programs how to interface with and control physical hardware. They allow developers to communicate with test instruments and units under test (UUTs).   Some drivers must be installed on the stations which utilize them. Other drivers, such as LabVIEW instrument drivers, consist of code libraries that do not need to be installed on a system.  Determine the best location for this type of driver according to the dependencies between the driver and the code that calls the driver.  Refer to the File Dependencies section of this document for more information about file dependencies.

TestStand Configuration Files

Use TestStand configuration files to customize the settings of the local installation of TestStand.   These files describe and drive the behavior of a particular test system.  TestStand applications read and write to some configuration files, such as TestExec.ini and StationGlobals.ini during execution. 

NOTE: Because most configuration files must be stored together  in the same folder, and because some configuration files need to have read and write access, configuration files must be located on the test station instead of the shared server.  The Users.ini file is an exception and can be stored in a location other than where the rest of the configuration files are stored. 

Table 2 describes the configuration files in a TestStand system.

Name	File Name	Description
Station Options	TestExec.ini	Stores most system settings, including configuration of execution, error handling, process models, and language options.
Station Globals	StationGlobals.ini	Stores the definition and value of station global variables.
User Information	Users.ini	Stores user names,, privileges, and passwords
Templates	Templates.ini	Stores step, sequence and variable template definitions
Tool Menu	ToolMenu.ini	Stores the configuration for the tool menu found in the TestStand user interfaces
Process Model Options	TestStandModelModelOptions.ini	Stores the TestStand process model configuration options
Process Model Report Options	TestStandModelReportOptions.ini	Stores the report configuration options for TestStand process models
Sequence Editor Configuration	SeqEdit.xml	Stores the user interface configuration for the TestStand Sequence Editor

Table 2: TestStand Configuration Files

TestStand Components

TestStand components are features that implement the high level execution of the test system and are decoupled from the test sequence to improve the modularity of the TestStand architecture. TestStand components implement features such as login and logout functionality, reporting, database logging, and serial number entry.  Table 3 describes the components of a TestStand system.

Name	Folder Name	Description
Front -End Callbacks	<NI Components>\Callbacks\FrontEnd	Front end callbacks implement features to log users in and out of a test system.
Station Callbacks	<NI Components>\Callbacks\Station	Station callbacks store callbacks that are global to the station and take priority over sequence file callbacks
Compatibility	<NI Components>\Compatibility	Contains type palette files TestStand uses to save sequence files compatible with earlier versions of TestStand
Icons	<NI Components>\Icons	Contains icon files for module adapters and step types
Language	<NI Components>\Language	Contains string resource files in language-specific subdirectories
Models	<NI Components>\Models	Contains the default process model sequence files and supporting code modules.
Obsolete	<NI Components>\Obsolete	Contains components TestStand no longer uses but installs to maintain backward compatibility.
RuntimeServers	<NI Components>\RuntimeServers	Runtime servers and engines used to execute code modules in systems that do not have the development environment for these code modules installed
Step Types	<NI Components>\StepTypes	Code modules and dependencies necessary to execute step types
Tools	<NI Components>\Tools	Tools used by TestStand to perform different operations and tasks
Type Palettes	<NI Components>\TypePalettes	Type Palettes for the default types in the system

Table 3: TestStand Components

User Interfaces

End users view and control the test system through a user interface. User interfaces can be simple or highly customized, depending on your needs.  

You can store user interfaces on a server or on a test station.  Although the user interface relies on the TestStand engine to execute, the binary file can be located on a remote server.

Test Code

Test code for TestStand systems can be either the actual test code modules or TestStand sequences that call the code modules. Test code can be located on either the shared drive or test machine. 

The deployment effort varies depending on the ADE (Application Development Environment) you use to develop code modules. In order to debug your code on deployed stations, you must deploy the code as well as the  ADE you used to compile it. The following section discusses the most common ADEs and their dependencies. 

File Dependencies

When deploying a TestStand system, you must account for any file dependencies in the code called by TestStand sequences. Each programming language requires you to deploy a specific run-time engine to each test station in order to execute your code. In addition, programming languages produce different output files which may require you to deploy additional libraries or engines to target machines. The challenge in determining file dependencies is to identify code dependencies you might not be aware of. For example, when you use LabVIEW to develop your code, there is significant functionality that references files inside the VI.LIB, which is a standard LabVIEW library that contains most of the non-native functions of LabVIEW. Therefore, when you deploy your test VI, you must include any VIs from that library. You can use the LabVIEW project window to identify all the dependencies of your VI.In the case of .NET assemblies and DLLs, you must pay special attention to dependencies located in operating system caches and default search folders. For example, the System and GAC folders for DLLs and .NET assemblies, respectively, can contain a significant number of hidden dependencies since those locations are global operating system repositories. Tools such as dependency walkers help identify the dependencies for DLLs and .NET assemblies.   

Table 4 lists the most common programming languages and the file extensions you should include for each language.

Programming Language	File Extension
LabVIEW	.VI, .LLB, DLL, EXE, .LVLIB, LVPROJ
LabWindows/CVI	.DLL, .EXE, .UIR, .C, .H
Visual Studio C++	.DLL, .EXE, .C, .H
Visual Studio .NET	.DLL, EXE, .CS, .VB, .CPP
Visual Basic 6.0	.EXE, .DLL

Table 4:  File extensions for most common programming languages in automated test

Absolute Paths, Search directories, and. Relative Paths

Absolute paths, search directories, and relative paths are methods TestStand uses to  search for code modules at run time. When a test developer uses absolute paths, TestStand loads the code module from the specified path.  If the physical location of the module changes, developers must modify the sequence file to reflect the new location.   Search directories leverage the TestStand search paths list to locate code modules by searching a set of directories.  This method allows you to modify the module without further changes to sequences which use that module, though you must add the new folder location for the code modules to the search directory list.  Relative paths locate a file relative to a specified search directory.  If the location of a code module changes, you must either update the sequence file that calls the code module with a new the relative path to reflect the new location, or you must add a search directory to point to a folder relative

Best Practices and Considerations for the Shared Drive Deployment Architecture

This section steps through a series of important considerations and best practices for designing and implementing a shared drive strategy for deploying files to production machines. As illustrated in Figure 1, there are two perspectives to consider—Development and Production.  The following subsections present a challenge developers might face when implementing the architecture, followed by a discussion of the possible solutions, highlighting the tradeoffs of the different options and pointing out any differences that you should consider when planning for a shared drive for your Development or Production needs.

File Storage Location

Shared drive deployment architecture allows developers to store files in either a shared drive or on a test station.  With the exception of a few cases, the developer must decide where to store the files. The following are guidelines for selecting whether to store files on a shared drive or production machines.

The first step in making a decision about file location is understanding whether there are any  technical constraints that limit or dictate where you can store certain files. For example, TestStand imposes certain restrictions on the following types of files:

1. Configuration files must be located in the local hard drive
2. Front-End and Station callbacks, as well as Icons, must be located on the test station. You can mitigate the local drive restriction by adding sequence calls inside the local Front-End or station callback file that refers to files on the shared drive.
3. Engines and drivers must be installed on the local computer, with the exception of the TestStand Engine

For files that do not have any restrictions on where they can be stored, the developer must understand the tradeoffs of storing these files on the shared drive versus storing a copy on each test station.  Storing a copy of a given file on each test station can decrease the load time for the file, while story a file on the shared drive can reduce the effort of updating the file when it needs to be modified. Storing the file on the shared drive also ensures that all test stations use the same version of the file. However, storing large files on a shared folder may cause the test station to hang when loading the files.  The user concern due to the long load times can be mitigated by displaying a message to the user alerting them of the long load time as the file is being loaded.

There is also a hybrid approach for storing files. In this approach, you store a copy of the file on the test station in order to decrease load time, and then use  additional tools to compare the version of the file on the test station against the one on the shared drive each time the test station is started. If the versions match, then the file on the test station is used and execution continues. However, if the versions are different, the test station is updated with the latest copy of the file before executing.

Network and Shared Drive Dependency

Storing files on a shared drive means that the operation of the test stations will no longer depend only on these machines being up and running but also on the availability of the shared d and the network connection. If the Development Shared Drive or network is unavailable, test developers cannot perform their test development tasks. If the Production Shared Drive or network is unavailable, manufacturing operators cannot run test files from the test station until both the Shared Drive and the network link to it are restored. Each of the network connections in the system has a different level of importance and their downtime affect production differently.  The network connection that links the shared drive with the test stations must be up at all times to support continuous production.  The connection between development machines and the shared drive is critical in deploying new test systems, but will not stop production if it is down.

You must work with your IT department to make sure they can meet the uptime requirements for both the shared drive and the network. For example, IT might use server redundancy in order to meet uptime requirements for the shared drive.

In addition to the uptime considerations, you must also work with your IT department to plan for periodic backups of the shared drive in order to reduce the cost of rebuilding the server if it is corrupted. Your IT department should also be able to monitor bandwidth and network usage, as well as monitor available space on the share drive for storing files.

Change Control Management

A Change Control Management (CCM) strategy encompasses your organization’s processes and tools for ensuring that every test station is executing the right version of each file.  From a production perspective, a CCM strategy ensures that all local production machines are running the same version of a file so that you can test products consistently on all test stations. From the development perspective, a CCM strategy ensures that changes made by test developers are submitted for tracking. In addition, a CCM strategy reduces the risk of test developers accidentally overwriting changes to shared files .

CCM processes and tools help reduce the cost of maintaining shared drives and test stations as well as the risk of making accidental changes that can unfortunately end up in production and potentially stop production or ship products tested incorrectly. This section discusses how your CCM strategy, file location, and file organization impact each other.

The Challenge

We can break up the CCM challenge into two parts. First, we need to allow test developers to make the change to the correct version (the “master”) of the file that needs to be updated. Then, we need to make the new version of the file available to each test station. The following aref key questions to address when making a change to a file and deploying it.

1. Making the change to the file
   • Where is the master copy of the files to update?
   • How will you update files when changes need to be made?
   • How will you enforce who can make changes?
   • How will you document and track these changes?
2. Deploying the updated file
   • How will you ensure that each test station gets access to the updated version?
   • How will you revert changes if you find issues with an updated file?

Addressing the Challenge

The specific shared drive model you select will impact the answers to the questions above. Assuming that we chose the model on Figure 1, the master copy of the file to update will be located on the Master File Repository. Therefore, starting on the Development Side of the model, test developers can sign out the master copy of the file and place it on the Development Shared Drive using the specific functionality provided by the source code control application used to interact with the Master File Repository.

The test developer could then use a development machine to access the file from the Development Shared Drive, make the necessary changes and verify that those changes work as intended. Once the test developer is done with the changes, he can check the updated file back into the Master File Repository. You can use the functionality provided by your source code control application to grant edit privileges to folders based on the person(s) who own them so that test developers can only change and check in files located under folders they own.  You should consider implementing a communication and approval mechanism to make sure that all the impacted parties are aware of and agree to the change. Finally, the source code control application can also provide functionality for documenting and tracking changes.

Once the developer has checked the updated file  back into the Master File Repository, the next step is to deploy the updated file to the Production Side. This process depends on whether the file resides on the Production Shared Drive or on each test station. If the file resides on the Production Shared Drive, production software must be stopped to ensure that file references are released and that the Production Shared Drive is updated with the new file. Once this process is complete, all test stations using the file will be ready to use it the next time the machines are started.

Having these changes available to all test stations with such ease is one of the biggest advantages to using a shared drive approach. In the case where you have multiple Production Shared Drives spread across different production sites, you may need to set up additional processes to synchronize the changes across all these shared drives.

On the other hand, if the updated file resides on each test station, the deployment process is complicated by the fact that all affected test stations must be updated individually to replace the old file with the new one. This may not be a huge undertaking if the number of test stations is small. However, for a large number of test stations or machines spread across multiple sites, the effort may be significant. In addition, you must factor in the risk of missing some test stations and leaving them running the old version of the file, thus testing products incorrectly. One way to deal with this is to consider the hybrid approach mentioned earlier, which provides an automated method for checking and updating the file on each test station. However, you must factor in the cost of developing and maintaining such a tool.

In addition to updating files, you must also be able to revert changes. This may become important, for example, if you discover that a change produces an unforeseen side effect which causes a bigger problem. The ability to confidently identify and revert changes to one or more files is a key feature to consider when evaluating a CCM strategy.

Setting up Test Stations

Often, setting up a test station is  a very complex and time-consuming process. To set up a test station in a very specific way and avoid human error, you could create an automated installer that initially prompts the user for basic information about the setup required for the test station. Then, the installer can ensure that the necessary steps are followed. This would guarantee that similar stations are set up identically, resulting in station standardization, which in turn reduces machine maintenance costs. However, this automated installer is an additional software application that must be created and maintained, adding to the cost of the overall strategy.

Even if an automated installer is made available, the installation process may be longer than what your organization can afford. In this case, you can consider creating images of the hard drives of the test stations. You could then use these images to rebuild machines that crash or to build new ones as throughput needs increase. However, you must create and maintain a process and  tools in order to support an image library for hard drives.

Folder Structure for Test Code

Most organizations require certain level of file sharing among test developers. This may involve sharing tools used to display messages in a standard screen, sharing low-level test code to standardize how a particular type of measurement is made, or sharing common tools for interacting with a corporate database. 

These file sharing practices reduce development costs and support test standardization across machines. In addition, these practices can support the implementation of CCM processes and tools by designing the folder structure for your test files around the structure of your development organization. For example, you can use folders to group files according to the test developers or teams that own and edit them. For example, your test development organization may be broken down by product lines and that each product line has its own test development team. In this case, a high level folder structure can be defined where each product line has its own folder.

Figure 2 shows an example of two product lines, DIO and MIO. Each product line contains folders which store different versions of the file. Notice that each product line follows a standard structure which includes Documentation, Resources, Sequences, and TestCode. Although the test developer is free to customize the folder structure at this level, standardization allows you to implement error checking tools for deploying changes efficiently and effectively.

Figure 2: Example folder structure for test system deployed components

In a similar manner, you could create a folder under Products called Common to store all of the shared tools used by test developers.

Once you agree on a standard folder structure, you can grant access privileges to a folder for test developers responsible for that product line. In the case of the “Common” tools folder, you may have a central group responsible for maintaining tools so they work across all product lines, such as a Frameworks Development” group. This group would be the only one with edit privileges for this folder, limiting unauthorized or accidental changes to files.

With regards to the folder structure between the Development Side and the Production Side, they are identical up to the level of the Documentation, Resources, Sequences, and TestCode folders for all product lines. This extends the standardization of this high level folder structure from the Development Side and into the Production Side. Below this standard folder level, each test development team may decide what to make available when their product line folder hierarchy is deployed to the Production Side. For example, a development team may chose they do not want some ‘development only’ files deployed when synching their product line folder to the Production Shared Drive, but only when synching to the Development Side. They may chose to do this to save space or to prevent anyone from even viewing these ‘special’ files from a Test station. A different development team may decide that they want the Development Side and the Production Side to always contain all the files in their entire product line folder.

Although not directly related to the topic at hand, another benefit of having as much of a standardized folder structure as possible is that it reduces the learning curve for new test developers who join or move between product groups. 

Development Machines vs. Test Stations

As we have pointed out, there are at conceptually two types of test stations-- development machines, which test developers use to develop files, and test stations, which run the files during production to test products. You may conclude that there is no value for your business to differentiate between the two types of machines and it is more cost effective to support both development and production needs in one generic machine. However, evaluating your development and production needs separately is a good exercise to ensure that your system meets all of your requirements.,

Test developers typically need development machines equipped with all the applications necessary to develop, test and debug their test files, such as  TestStand and LabVIEW and/or Visual Studio. Test developers also need the ability to make file changes quickly and easily in order to develop and debug their programs efficiently.

Test stations are only used to run the files, and do not necessarily require source code to be available. Therefore, it is typically sufficient to install run-time engines for most applications. Test stations must be tightly controlled to protect against unauthorized file changes, since these stations are used to test products that will be shipped to customers.

Maintaining both production and development machines increases your overall system cost, due to the need for additional documentation, training, or automated installers. However, increases in development efficiency and a reduced risk of shipping incorrectly tested products may justify these increased costs.

Conclusion

Effective deployment architecture can improve the efficiency of automated test systems by facilitating the distribution of new test software and reducing the time to market for new products.  In order to create effective deployment architecture, developers must understand the different pieces that make up a TestStand system, their dependencies, and the best ways to package them to facilitate deployment.  Shared drive architecture facilitates deployment for organizations with a high mix and high volume of products.

For more information about deploying TestStand systems, refer to Chapter 14, Deploying TestStand Systems, in the NI TestStand Reference Manual.  Read the NI TestStand Advanced Architecture Series for more articles on advanced TestStand topics.

About the Authors

Roberto Piacentini - National Instruments

Roberto Piacentini is group manager for Frameworks and Calibration groups at National Instruments. Equipped with 12 years of experience in electrical engineering and software development, he leads a global team of developers that creates integrated and innovative frameworks meeting the evolving test and calibration requirements of internal and external customers.

Hjalmar Perez - National Instruments

Through his 15+ years of experience, Hjalmar Perez has held multiple leadership and technical roles within Test Engineering in R&D as well as Manufacturing. He architected the first integrated production test framework used at National Instruments and is currently responsible for driving the vision and architecture of the test frameworks needed to support growing global operations.

Reader Comments | Submit a comment »

Corrections and suggestions
See in section -> File Storage Location - > 4th Paragraph -> See the word 'story' in this sentence: "while story a file on the shared drive...". I think it should be 'storing'. See in section -> The Challenge -> 3rd line -> See the word 'aref' -> "The following aref key.....". It should be 'are'. I could not find which configuration (basic) files are required when creating the deployment utility. Even this is not specifically mentioned in the Teststand reference manual. Appreciate if a link could be provided for doing this.
- Nirmal Sharma, MindTree Ltd.. nirmal_sharma@mindtree.com - May 1, 2008