NI TestStand Verification and Validation Best Practices

An on-going challenge of today’s test world is ensuring that test systems are designed correctly and that they continue to function properly. While developing a test system, engineers create test procedures and set measurement limits to correctly detect defective products. Once the test methods have been determined, the test system is developed to confirm that the software and hardware performs the tasks correctly. When the system is completed and implemented, changes might arise in the company, product, or test environment that must be incorporated into the test system. Verification and Validation (V&V) processes formally ensure that the test system is developed correctly and accomplishes its intended purpose.

V&V primarily affects businesses governed by ISO or FDA procedures or good practices that manufacture products such as pharmaceuticals or medical devices, or products for automotive and aeronautical use. Since such products are highly critical to health and safety, these industries are subject to formal oversight, including well-defined V&V processes. Some companies voluntarily invest in formal V&V processes to reduce costs, or for competitive reasons. If a company’s competitiveness is based on quality or reliability, investing in a rigorous V&V process might pay for itself.

NI TestStand software helps engineers develop test systems by providing tools to create test sequences that call code modules written in many different programming languages. TestStand also helps test engineers manage test limits and configuration. TestStand components are critical to any V&V effort.

This document discusses V&V as it applies to test systems developed with TestStand. First, this document introduces the concepts of verification, validation, and impact analysis as applied to TestStand. Second, this document examines the components of TestStand and describes some best practices to help streamline V&V for each component. Finally, this document discusses some general purpose best practices for V&V.

Validation and Verification                                

The governing principles of V&V are well-defined for many industries, and are outlined by disciplines like Good Manufacturing Practices (GMP) or by regulation such as ISO-9000, FDA's 21 CFR, or IEEE Standards.  Each V&V system is similar but uses slightly different terminology to explain the generic requirements of the two processes.  Specific requirements are usually not defined.  This document explores V&V processes for automated test systems.

Learning the difference between verification and validation is an important step in preparing to develop and certify a test system. 

Verification is one or many formal audits to determine if a test system is built according to specifications provided in a design, drawing, statement of work, or other similar guideline.  Verification allows an auditor to test at the end of a development phase if the item being produced meets the conditions imposed on it.  Verification tests can be performed at several milestones of a product development, and can be performed on assemblies and subassemblies.

Validation is the process to evaluate if the system, having already passed verification to one or more specifications throughout several phases, is complete and now accomplishes its purpose or intent.  Validation might include a formal pass/fail test procedure, or it might be a subjective form of a usability study performed with customers, users, or patients.  Validation often involves some subjective requirements, such as rejects defective products or has an easy to use interface.

To illustrate the difference between verification and validation, suppose a test department builds a simple test fixture for measuring the electrical current consumption of a unit under test (UUT).  The test system must compare the current of the UUT current against test limits that require the UUT to consume less than 500mA.  If the system performs the measurement correctly with acceptable repeatability, the system passes verification.

If a known failure mode exists in manufacturing, such as a diode installed in reverse, that causes some parts of the circuit not to activate, the current consumption might be excessively low, perhaps 150mA.  Unfortunately, in this example the UUTs pass testing on the verified fixture, and might be shipped.  Though the test system was built correctly according to the specifications, the system does not serve the purpose it was commissioned to fulfill and therefore fails validation.  The specification and the test system must be modified to incorporate an upper and a lower measurement limit, such as 400-500mA. 

Performing system verification can be relatively easy based on a well-written specification, drawing, or statement of work, and test methods can be very straightforward so that defects are easy to find, but validation  can be more challenging, as shown by the previous example. 

In comparison, validation requirements that relate specifically to medical devices in the FDA’s 21 CFR are vague, including phrases that specify that a medical device must be validated to conform to user needs and intended uses, so the quality system manager must define the needs and oversee validation testing.  Selecting the validation methods is not necessarily very easy.  For test systems, one method of defining a validation test might be as simple as keeping track of known failure modes, and having good and bad product samples available to help ensure the system detects known defects.  Another method might use a trusted manual test procedure or include another automated system to validate the results of the new test system.  

Some instances of validation can be extraordinarily thorough, such as those that surround the aeronautics, pharmaceuticals, and medical device industries, because the validation processes involve extremes cases of safety, quality, or cost.  A thorough system validation might take weeks or months to define and perform.  For example, if a test system uses a switch matrix in a 16x32 configuration, the test engineer might be required to test every possible combination of connections using a continuity tester and ensure that no restricted connections are ever made.  Another example might consist of validating a communication system in which every possible command and sequence of commands must be tested and validated.  Although such validation processes might seem extreme, is imperative that no damage, injury, or incorrect results arise under any circumstance.

No written procedures exist to explain what must be verified or validated, or to define how testing must be accomplished.  The same is true for reverification or revalidation if changes are made to a test system.  The organization must appoint someone to make recommendations about test procedures and review and approve them.  Although each company must decide and define how to implement design controls and change management in their products and test systems, this document provides some ideas and best practices to help with defining such policies.

Impact Analysis

Impact analysis concerns the understanding of consequences of any change that alters the way in which a test system performs, such as replacing a failing instrument or modifying an algorithm or setting.  Changes might be due to maintenance or repair, or trying to improve or correct the performance of the system.  A change might cause the system to operate improperly in a noticeable or even unnoticeable manner, and might result in product recalls, production stoppages, or other interference with business.  A change might also cause the system operate properly but affect the outcome or test results, and cause incorrect decisions about tested products. 

The cost of a missed change or an incorrect validation can be extremely large, such as shipping bad products to customers.  In some industries, shipments of faulty products can result in a product recall.  The FDA reserves the right to take regulatory action.  One company reported that they take extra measures beyond typical V&V processes because even a tiny reduction in stock price might cost them billions of dollars.

Assume that every change you make will require revalidation. No simple rule exists for deciding what you can change, how you can make changes, or what effects will arise from any change. Understanding the components of TestStand and the impact of changes made to the components can help make changes easier to plan for, easier to recognize, and can minimize the scope and difficulty of testing the changes.

Components of TestStand

TestStand is not a traditional programming language, and therefore possesses distinct advantages that facilitate V&V processes.  TestStand includes a very modular architecture made up of basic building blocks that allow your test system to operate however you want.  The following sections describe the TestStand components as methods to facilitate design decisions to better manage V&V processes.

Sequences & Steps

The most fundamental part of a TestStand system is a sequence.  In a sequence, you define the overall behavior of the system and the settings and limits applied to each test.  Even in the simplest sequence, much of the work involves creating and configuring steps in the sequence.  You can add sequencing actions, such as looping properties, preconditions, and other configurations that directly influence which steps execute and in what order.  For each step, you can select a number of settings, such as how to record results, what expressions to evaluate, or what actions to take if the step fails.  Both built-in and custom steps have options that determine if a step passes or fails, and if the UUT passes or fails.  Some steps determine failure from measurement limits that you set. For example, for the numeric limit step you can require that a measurement be within a range.

Changes to the steps in a TestStand sequence fundamentally modify how a test system operates.  Changes include renaming steps, modifying the preconditions that allow a step to run, or even deleting an obsolete step.   Such changes might be critical and require revalidation of the system. Other changes might be somewhat minimal and require minimal validation.   Each step has some amount of interaction with the steps before and after it.  Determining the effect of a change requires a specific study of how those steps work together and where each on gets information it needs to run.

The following are some best practices to help you make changes to sequence files easier to recognize, understand, and test:

• Use Case steps instead of preconditions.  Case steps are more visible in the TestStand environment, are expandable, and can be modified to run different options and include more than one step per condition.  Preconditions are partially hidden and can only determine whether to run a single step.
• Use Switch Steps instead of using the switching options for a step.  Switch steps are more visible in a sequence and make the sequence somewhat more self-documenting.  The switching options for a step are convenient but are partially hidden and can only be set to start and stop in relation to a single step.  Create written procedures and TestStand sequences that relate to each other directly.  For example, if a written procedure requires that you first power on a device and wait for the device to boot, create two instruction steps in the procedure to correlate to two steps in TestStand.
• Name every step carefully as a form of documentation within the sequence.  The step name should describe what the step does and why it performs an action, if possible.  For example, instead of naming a step Wait, name the step Wait 2 second for system to boot.  If the name requires more information, use the Description property of the step to specify additional details. 
• Organize the sequence so that the Main sequence is composed almost entirely of Sequence Call steps.  Each Sequence Call step can relate to a single test that can be given a name, such as Power and Current Test sequence call followed by the Audio Test sequence call, and so on for each test in the test procedure.  This helps logically organize the sequence similar to how someone might describe the sequence verbally, saying “The procedure starts by doing a power and current test, then it does an audio test.” 
• Create each subsequence to stand alone and prepare itself to run, including the setup of all instruments and devices for the subsequence.  For example, if you make changes to the Audio Test, you might want to  run the subsequence over and over before validation to ensure it runs properly and to avoid having to run the entire sequence.  The test subsequence should power on the device, set up all necessary switches or instruments, and execute with noticeably good or bad feedback from the Audio Test results.  If each test runs independently, testing and retesting can become much easier.
• Avoid using the PreviousStep property and use variables instead.  The PreviousStep propertyrefers dynamically at runtime to the step that last ran and that retrieves information from the step.  The source of the information can change if you rearrange or delete steps, thus resulting in erroneous or missing data.  Set a variable using a the PostExpression property of the step and retrieve data from the variable as needed. 

Code Modules

TestStand does not control instrumentation and automation hardware directly, but instead calls code modules to perform these tasks.  TestStand relies heavily on the design and performance of code modules to operate properly.  Code modules are small pieces of code written in a number of languages, including LabVIEW, C++, C#, and Microsoft Visual Basic.  A test system developer or coworker might create many of the code modules, while a third party might create other code modules.  Additionally, many steps included with TestStand are actually code modules written in C and compiled into DLLs.  Changing a code module used in a test sequence fundamentally affects how TestStand performs actions, but this can be advantageous because TestStand is a fixed and compiled application that cannot be changed, and only the code modules TestStand executes can be modified.  Because a code module must only receive data from TestStand, execute a specific action, and return data to TestStand, you can limit testing caused by a code module change. 

The following are some best practices for using code modules:

• Plan ahead when you select parameter inputs and outputs for the code dodule.  Do not rearrange or rename inputs unless no alternative exists.  For example, when you select a connector pane for a LabVIEW code module, select a connector pane with extra inputs and avoid changing the connector pane.  Any of these actions might cause the step to not execute in TestStand or might cause the step to execute but return results improperly or out of order. 
• Design or implement code module testers that can test all possible inputs and record the output values for each set of inputs.  This can help prove that a single code module executes correctly as it did before the change, but you must still prove that new software works with TestStand, and that it executes and returns correct results.  If the code module passes module testing and can still be called from TestStand, no change or effect to the TestStand sequence exists, and you can limit testing.
• Avoid using the Sequence Context to create variables or to set the value of variables dynamically.  Instead, pass values from code modules to TestStand using the Module Adapter inputs and outputs to make setting variables more obvious to see and easy to track.This also allows you to use the Test Expression button to verify the datatype or expression and ensure that the data type or expression evaluates correctlyto help avoid setting the wrong variable with the wrong data.

Data Types and Process Models

Data types and process models other characteristic components of TestStand that you can change to allow advanced programmers to fine tune performance and behavior.  Many users employ these components in the background without realizing they have done so, and without modifying the components intentionally.  For example, aata types, such as strings, numbers, and arrays, define the characteristics of the variables and constants throughout the system.  A process model is a sequence within a sequence that is generally unseen and that governs how your sequence starts, executes, stops, and reports data, and manages batch or parallel testing.  You can modify data types and process models, but leaving them in their default state guarantees smooth V&V processes.  You might think of these components as being similar to the registry within Microsoft Windows, in which developers must thoroughly understand changes to avoid unintended changes to the system.  Although some changes to the process model might seem relatively simple, the complexity of the changes can affect multiple components in the system, thus making V&V processes difficult to plan. 

The following are some best practices for using data types and process models:

• Treat changes to the process model as seriously as you do when you make a registry change or large operation system change.  Consider creating a backup of the sequence file and process model files, and perhaps a backup image of the entire hard drive if you might undo the change.  . 
• When you create a new data type or process model, copy an existing version and modify it in minimal increments, changing some features and testing it at each phase to ensure that the change occurs as you expected without many unexpected consequences.

TestStand Settings

TestStand includes several options and preferences that you can access from the menus in the TestStand Sequence Editor or in a TestStand User Interface.  Some settings can be easily changed without any noticeable effects in operation, and cause no noticeable change in managed or verified source code files or sequence files, but changes to other settings might significantly affect the behavior of the system.  One such setting is the database schema that handles how data is stored in a database, and how TestStand modifies measurements in preparation for storage.  Another setting, which is as simple enabling a checkbox, can disable all reporting and result in serious implications to any test system.  You must be aware of what the TestStand settings accomplish, how you intend to use the settings, and how the settings affect your test system.

The following are some best practices for using the options and preferences in TestStand:

• Review all available settings early and decide how to set them before the system is verified.  If possible, include the settings you want to use in an early draft of the specification, so that the settings are less likely to change later.  For example, decide what kind of reports you require and select the appropriate settings.  Try to imagine any changes that you might have to make to those settings over time.  If not conceived early, a change to a setting would require revalidation.  Pay attention to database and report settings, especially as they relate to hard drive space or database size.  A system that tests 1000 units per day and generates a report for each unit can become difficult to browse or might fill up the hard drive on the computer.  If a customer must retain all reports but does not use them on a daily basis, keep the size of the reports as small as possible.  If the customer uses reports  only in case of failure on a test run, but never uses the reports again, you can instruct TestStand to reuse the filename and overwrite old reports. 
• You can keep the size of reports as small as possible by turning off reporting for as many steps as possible.  If reporting is activated for every step, test with just five key measurements might include over one thousand entries in the database and reports.  Many steps, such as DoWhile, End, For, and Break, do not provide much useful information in databases or reports.  Steps inside loops write multiple records for each loop.  In fact, many consumers of the reports and database, such as operators and quality engineers, will likely complain that the reports are very difficult if they only want to find what failed on a single test run. 
• Use search directories to eliminate the possibility of finding code in an unexpected place.
• Use relative search paths to locate code if you decide to move or rearrange folders and files. 

Factors External to TestStand

TestStand developers must realize that numerous factors can affect a test system, including a company's network configuration, the computer name, or the correct date and time.  Some of the most obvious but often overlooked settings are those for instruments and the computer itself.  Instrument dettings might include NI-DAQmx settings made in NI Measurement & Automation Explorer (MAX), or GPIB and COM settings for a device.  In Windows, a setting might include a network drive mapping, or a screen saver setting.  Changes to such settings can interrupt or modify the operation of a test system and might be partially undetectable, such as an instrument failing to take valid data, or a motor failing to move.  Such settings can be numerous and validation tests difficult to design.

The following are some best practices for handling external factors:

• If you can set any options dynamically, consider using the Setup step group in a sequence to ensure the are set correctly. 
• If possible, query each setting to ensure that the setting was accepted by the instrument or program.  If the setting cannot be queried, find where the setting is stored and read it from a text, INI, or XML file.  The system can verify and even record the state of items outside of TestStand and keep them under control.
• Find files that contain the settings and manage them using source code control (SCC).

The preceding sections provided a brief introduction to the TestStand components that you can change, resulting in reverification or revalidation of a test system.   Use discipline and best practices to plan ahead and save unnecessary delay and cost. 

General Best Practices for Validation and Verification

This section describes more general best practices that do not apply to any specific TestStand component.  Although not completely exhaustive, the ideas below can help educate and remind you of items you might want to manage while developing any test system subject to V&V processes. 

Documenting and Gathering Requirements

V&V processes center on well-defined specifications.  Validation can also be subject to some objective issues, such as loosely defined needs of the marketplace or the end user.  The most important first step for any test system is to research and document a good working specification and V&V requirements.  Testing can be difficult if specifications are not specific or leave room for interpretation or ambiguities.  Testing can be halted if the auditor or observer finds a setting undocumented or implemented incorrectly.  A well-written specification implemented with care and attention can help ensure a painless V&V process.

Verification requires one or more design documents or drawings to govern what the system must accomplish.   The documents and drawings might cover a component, assembly, or entire system.  The specification and test methodology for verification must be a throughly detailed document with as much information as necessary to create a correct test system.

Thoroughly record any changes to specifications, whether devised by a customer, by engineers, or as the result of learning and discovery.  Employ a change order process to record the change and the reason, and to make the change official.  Verification only passes if you match instructions, settings, and test limits to the correct document.  You might also need to record less obvious changes, such as Quality Manager agrees to reduce measurement resolution to reduce test time, which will increase throughput for Manufacturing Manager.  You might want to debate, test, agree, and release such a change under a change process to ensure all the changes are correctly implemented.

Designing a validation test might seem like more of an art than a science, and although wisdom and experience might seem like the only tools for validation design, remember that gathering requirements can be revealing and useful.  Techniques might include reviewing past performance of other test fixtures or products, interviewing operators and their supervisors, and studying past measurement data.  One company that commonly outsources to test system integrators performs a detailed review of each project to find ways to make the next project better, and places those ideas in a checklist for the next project.  .

For example, if you manufacture a product as simple as a cooling fan for electronic enclosures,compliance requirements might include electrical emissions testing and describe the way the emissions must be tested.  For the same product, an audible fan noise might not exist in the verification spec or test procedure.  Assume that a study of product warranty data reveals that annoyed customers historically return noisy fans that are too loud in home or office devices, and the noisy fans came in batches.  Perhaps the test system could perform random sampling of audio information to save test time while still inspecting the noise level of the batch.  If you treat requirements gathering with meticulous notes, fact-checking, and curiosity, you can help design a system that can more easily pass validation.

Take Advantage of TestStand Modularity

When a system is modular, the system consists of several modules or components.  In comparison to manufacturing, many parts or subcomponents have their own V&V processes so that the product can pass through validation with reduced effort and can incorporate changes and improvements.  For example, in automobiles if an airbag module changes, the module can undergo some electrical tests to ensure that the next version of the airbag is functionally equivalent to the previously validated airbag version.  Some non-electrical requirements, such as speed of deployment, size, shape, and other factors, can all be verified outside the automobile.  Because the airbag has no effect, for example, on fuel modules and exhaust systems, no reason exists to reverify or revalidate the subsystems to approve the vehicle design as a whole.  A revalidation plan might have the fuel system, braking systems, and others marked as no change but the airbag, the steering system, and bumper sensors might be marked for a simplified V&V process.  You can apply such modularity to test systems. 

In the case of TestStand, you can plan code modules to be modular in order to streamline V&V processes.  For example, if you have successfully validated version 2.5 of a sequence and version 1.8 of code moduels, consider the effort involved to change a measurement limit.  If the limit is hard coded in LabVIEW, you must modify and recompile the code, which requires you to revalidate the new code, version 1.9.  If you can make the changes outside of the source code, such as loading limits from a text file, then the validations of versions 1.8 and 2.5 of your code and sequence file are still current.  You might still need to check that the new text file is correct, and that it is compatible with the code and sequence. 

Limit the Interaction between Two Subsystems

One strategy to limit the amount of revalidation is to include distinct lines of independence in your system.  For example, if you have a medical device with firmware loaded to it, followed by a number of functional tests, and a customer-specific file is loaded into memory and tested before shipping, you can create three separate test systems that perform these tasks independently of each other.  Changes to one test station do not affect the others.  In the same way you can create a single test system that performs the work of all three, each of which can be written into its own sequence file and the sequence files can be called one after the other from a master sequence file.  IN this way you can easily show which files are modified so you need to only reverify the files that change.

Some dependence still exists in the independent systems, whether they are separate stations or not.  The functional test is dependent on the previous firmware test working properly.  Although they are interdependent, you want to limit the effects of a change.  In this example, the next step can continue as long as the previous step worked properly.  You can use a step in the functional test to verify that the loaded firmware is compatible.  Because only one output value affects the next step, validation can be minimized to ensure that the requirement is met after any changes.

For example, if test 3 is an Audio Test that includes a volume test at Low, Medium, and High volume and test 4 performs audio quality testing that must be done at Medium volume, you might want to power off and reset the audio volume between tests and design test 4 to set itself to medium volume.  If test 4 is now completely independent on the design and settings of test 3, a change to test 3 can result in a revalidation limited to only that single test.

Manage Files Using Source Code Control Tools

Computer-based test systems consist of numerous files.  Source code, sequences, configuration files, and others change as a test system is developed or modified. 

Code Modules are stored in a format specific to their platform.  For example, LabVIEW generates Virtual Instrument files called VIs (.vi and control and type Definition files (.ctl), and organizes those files in LabVIEW project files (.lvproj).  An application written using applications such as C or Visual Basic includes a number of source files and compiled DLLs. 

TestStand sequence files (.seq) are created within TestStand and you must manage them in the same way you manage software code modules.  You can also store sequence files in binary, text, or XML formats. You can organize sequence files along with your code modules. 

Generally, the system designer creates, saves, and manages the files mentioned in this.  One best practice is to consolidate the files in a single location, or use a management tool, such as a LabVIEW project or TestStand workspace, to reference them in various locations.  You must track and archive all files in support of a given specification.

Files other than those deliberately modified and collected might include .ini files, XML files, or binary files that TestStand or another third-party application stores.  For example, TestStand and LabVIEW both use .ini files to store some configurations and settings made within options and menus.  MAX uses a file designed to be hidden from tampering but that contains information about your instruments configuration.  You can export most settings from MAX to a backup file (.mce) for users to manage.  Most applications, drivers, or other tools generate files, and changes to any of these files can affect V&V processes.

The industry-accepted method for monitoring, controlling, and storing such files is Source Code Control (SCC) programs such Subversion (SVN), Perforce, and Microsoft Visual Source Safe.  Many of these programs are designed to conform to the Microsoft SCC interface and you can use them from within TestStand or LabVIEW.  In some cases, you cannot modify a file without taking temporary ownership of it and documenting your changes in order to save them.  These programs can often tell you which files changed as well as analyze the old and new files to highlight the changes to help simplify verification.  For example, Figures 1 and 2 shows an example in Subversion, in which  only one file, MySettings.xml, has changed, and in that file only the Sheath Flow value was modified.  Using these programs helps simplify validation because the programs can demonstrate that all the other settings have not changed. 

Figure 1. Subversion Showing Only One Modified File in the Working Directory

Figure 2. Comparison in Subversion Showing Only One Modified Value in the File

Verify Files at Run Time

After you complete and verify system configuration , you must archive the system in such a way that you can recall and restore, verify, and distribute the system as the proper specification.  For example, if the MySettings.xml file from the previous example passes V&V along with a certain TestStand sequence file, you can commit both files to source code control and submit the files to an independent document control entity.  You might want to self-audit to verify that the modules are running on the system.  Allow TestStand to verify everything possible during testing, and even record confirmation of dependant files alongside the test results in a database or report.  For example, you might accomplish this using a step that can read the checksum of a file and ensure that it never changes, as shown in Figure 3.  Using file utilities and storing the results along with other test data creates a complete record that verified files are being used correctly. 

Figure 3.  Third-Party Add-on for TestStand to Verify and Record the Checksum at Run Time

Hardware Configuration Management

The requirements for hardware configuration management are no different than for software.  You must properly select, install, program, and configure the instruments not only for the system but also for each individual test.  For example, a digital multimeter (DMM) or oscilloscope has several options to configure for proper communication and signal acquisition that must be verified and validated at the completion of a test system and for changes to hardware in the future.

Use software to help validate hardware at run time.  By reading and storing settings or other factors at run time you can have confidence that items that must be validated along with the software are set and operating as intended.  For example, your TestStand steps might query the calibration dates of an instrument to ensure the calibration is current, and can verify the model number and serial number of the instrument attached to a COM port to  ensure that the instrument has not been replaced.  Designing your test sequence and even purchasing instruments with these considerations in mind can help simplify V&V processes.

If you anticipate that your hardware must change, you must consider the change in a V&V process.  If an instrument fails and another instrument of the same make and model is inserted, think about what you must accomplish to verify if it operates correctly, and design a test to ensure that the change was  successful.  If the instrument settings are critical from one instrument to the next, create a sequence of steps that automatically sets up the instrument when the sequence detects a new instrument.  If another make or model is required, but the application is hard-coded to use that instrument, then you cannot avoid revalidation.  Using Interchangeable Virtual Instrument (IVI) drivers and interfaces for instrument setup can help simplify the transition between two instruments of the same make or model, or between two instruments of dissimilar make or model.  

Understand and Manage Software Upgrades

A common question concerns whether you can upgrade LabVIEW, TestStand, or any other program to take advantage of new features as they become available.  Making such a software upgrade is always a trigger for a revalidation and reverification.  Treat a potential upgrade as a return on investment (ROI) exercise.  For example, to gain a streamlined development interface, you might want to upgrade during development but not after the system is deployed.  However, as is the case with recent TestStand upgrades, improved execution speed can result in shorter test time, greater throughput, and greater revenue.  In both cases, the cost of revalidation is the deciding factor, but the cost can also provide a positive ROI and is therefore worth the expense and effort.

Another easily overlooked upgrade is Microsoft Update, which can help to protect a system from attacks and help repair bugs discovered in Windows.  By default, Microsoft Update occurs automatically on most computers.  Other companies, such as Sun, Apple, and Adobe, also use web-based automatic updates.  Unequivocally, you must disable any automatic changes and upgrades on any system that is subject to V&V processes.  The changes that automatic updates make are not predictable and can have unknown effects on operation and settings.  Some updates automatically reboot your computer after installing. 

Your Internet Technology (IT, MIS, or other) department might have a general policy that they control computers within the company, including using virus scanning software, setting security policies like screen savers, and installing patches and upgrades as needed.  A manufacturing department must work with the IT Department to help manage TestStand systems by leaving them absolutely untouched.  These computers are not typically used to read email or browse the Internet and are therefore less susceptible to worms or viruses.  You must decide what items specifically affect your computers, but your needs might contrast with IT policy, such as removing virus scanners, turning off screen savers, and exempting from company-wide upgrades or patches. 

Conclusion

Significant challenges cane exist with V&V processes for any test system.  The best practices described in this document are just a sample of the methodology that simplifies V&V processes.  You must tailor any of these practices to your company, because some might with your particular  environment.  Enforcing some or all of these disciplines can improve your product quality, yield, rework efforts, and many other tangible and intangible considerations in order to reduce costs, turn greater profit, and please customers.

About the Author

Joe Spinozzi – Cyth Systems LLC

Joe Spinozzi is the co-founder and Director of Operations at Cyth Systems. Equipped with 12 years of experience helping dozens of companies develop products and test systems, he now leads a team of engineers to plan, architect, and complete projects across many industries.  Operating from San Diego, California, Cyth Systems is responsible for test systems large and small currently operating in the United States, Europe, and Mexico.

Reader Comments | Submit a comment »