Best Practices for NI TestStand Report Generation and Customization

A common requirement in automated test systems is to generate a complete report that contains all result information to answer the questions “What was tested?” and “What results were achieved by each unit under test (UUT)?”. Another common requirement is to use a unique report format as part of a well-designed reporting strategy.

This document includes different methods for customizing reporting in NI TestStand by explaining the default reporting in TestStand, results collection, and report generation. This document also includes different examples for best practices. This document assumes you have a basic knowledge of the TestStand API.

Default Reporting in TestStand

When you use a process model with the default ReportOptions to execute a sequence, TestStand creates a report that includes system information and test results when the execution completes. To generate the report, TestStand first collects result data during execution. Based on these results, the process model generates the report. The following sections explain the report generation process and show how to modify the process to customize the content and style of the report.

Report Options

Use the options in the Report Options dialog box to change the content and style of the report in a basic manner. In some cases, changing these settings is sufficient to customize the report. The Report Options dialog box includes a Contents tab and a Report File Pathname tab, as shown in Figure 1.

Figure 1: Report Options Dialog Box – Content Tab

Content Tab

Enable the Disable Report Generation option to prevent the process model from creating a report.

Disable the Include Step Results option to leave all step results out of the report. You can use the Result Filtering Expression, Include Test Limits, and Include Measurements options to exercise more control of the report content. The Result Filtering Expression option is not available for XML reports. You can change the visual appearance of measurement values in the report if the values reside in an array.

Enable the Include Execution Times option to include the time each step module takes to execute and the time the entire execution takes.

Use the Default Numeric Format option to set the display format. The default display format for numeric values is a floating-point representation with 13 decimal places.

The report format you select determines the available possibilities for modifying the style of the report. Refer to the Report Generation section of this document for the possible report formats.

For HTML or XML reports, you can define different color schemes. If the only change you want to make is to change colors in the report, use the Report Colors option.

Stylesheet are enabled for XML and ATML reports.

Refer to the Report Generation section of this document for more information about report formats, On-the-Fly Reporting, and the report body.

Report File Pathname Tab

The Report File Pathname tab contains configuration options for storing and naming report files. Figure 2 shows the default settings. By default, TestStand generates a new report file for each execution and lists all UUTs tested in the execution in the same report file. TestStand generates the filename using the time and date.

Figure 2: Report Options Dialog Box – Report File Pathname Tab

TestStand can dynamically create report filenames and pathnames so you can generate unique report files and locations for each UUT. The bottom of the Report File Pathname tab shows a preview of the report file name and structure.

Result Collection

Before TestStand creates a report, it collects step results and additional information to include in the report. Because the order of execution of steps can differ from the order of the steps in the sequence, TestStand collects results dynamically during execution. Each sequence includes a ResultList local variable to store the results. TestStand generates the ResultList during the sequence execution and stores it in volatile memory (RAM).

The ResultList local variable is initially an empty array of containers. After each step executes, TestStand adds a new container to the end of the array to store the results of the executed step. Each step stores its results in properties defined by the type definition of the step. Some properties, like Step.Result or Step.TS, are common to all predefined step types. TestStand inserts into the ResultList array only certain information from the Step.TS container, such as the step name and timing information. TestStand includes all properties from the Step.Result container in the Result container of the ResultList array.

The two most important elements in the Result container are Result.Error and Result.Status. Result.Error is a container for the Code, Msg, and Occurred variables. Code is a numeric element that represents the number of the error. Msg contains the information to display if an error occurs. If you set Occurred to True, TestStand returns a run-time error. Result.Status is a string element that contains one of the following values: Passed, Failed, Done, Skipped, Error, or Terminated.

Additionally, step types can include more subproperties. For example, the Numeric Limit step includes a Limits (high and low) container and a Comp string element for the comparison type.

TestStand combines all this information into the Result container it inserts at the end of the ResultList, as shown in Figure 3.

Figure 3: Collecting Results of the Execution in the ResultList

TestStand passes the ResultList of a subsequence to the calling sequence when the called sequence finishes execution. The subsequence ResultList becomes part of the Sequence Call step, as shown in Figure 4.

Figure 4: ResultList with Expanded Step Result of a Sequence Call Step

Modifying the ResultList

Because the ResultList is a local variable, you can directly access it to alter its content as you can with any other variable, but National Instruments does not recommend this technique for the following reasons:

• The report does not necessarily include all the information the ResultList contains. Refer to the property flags shown in Figure 10 for more information.
• Because the ResultList is an array of result containers, it can be difficult to determine the correct array element to modify.

Any modifications you make to the ResultList can have a direct effect on the content of the report. If you include results in the ResultList, the report might include that information, depending on other settings you make in TestStand. If you do not include results in the ResultList, the report will never include that information. For information to appear in the report, you must include results in the ResultList.

Note: When you enable Database Logging, TestStand copies the ResultList into a new Logging container after the MainSequence executes. If you delete results from or do not add results to the ResultList during execution of the MainSequence and its subsequences, TestStand does not add this information to the Database Logging container either.

The following sections explain the recommended ways for adding information to the ResultList, preventing result collection, and filtering the ResultList.

Adding Information to the ResultList

You can add information to the ResultList by using the Additional Results option in TestStand 4.1 and later, by using the Result.ReportText string, by adding an entry to the Step.Result container, or by using the Execution.AddExtraResult method of the TestStand API.

Using Additional Results in TestStand 4.1 and Later

Use the Additional Results panel of Properties tab of the Step Settings pane to include additional results in the report for a step, as shown in Figure 5.

Figure 5: Additional Results

You can add any number of additional results to a step. The results appear in the report in the same order as they are configured in the step (top to bottom). TestStand populates the list with the parameters of the module used in the step. Use the checkbox by the parameter to include or exclude the parameter from the report. By default, TestStand does not include the parameters in the report.

Figure 6 shows the predefined settings for logging additional results. If you want to log other information, you have to modify the settings.

Figure 6: Selection of Default Information to Log Using Additional Results

Use the following Additional Results settings:

• Name – An expression that specifies the name of the additional result as shown in Figure 7. TestStand automatically adds the ‘:’ after the Name. Figure 8 shows how the name appears in a report.

Figure 7: Example for Custom Additional Result

Figure 8: Additional Result in Report

• Value to Log – An expression that specifies the value of the additional result.
• Type – Specifies the type of the additional result. You can log any kind of data of a loaded type. The default value is <Any Type> so TestStand adapts to the type of Value to Log automatically. Selecting a custom data type enables a collection of entries, which is useful when using containers. In the report, TestStand indents each element in the container.
• Condition – You can define an expression for TestStand to evaluate as a condition before logging the additional result. TestStand transfers the additional result to the ResultList only if the expression evaluates to True.
• Include in Report – If not set, the additional result does not appear in the report. The Condition does not toggle this option. Include in Report modifies the IncludeInReport flag of the ResultList subproperty.
• Is Measurement – Modifies the IsMeasurement flag of the ResultList entry.
• Is Limit – Modifies the IsLimit flag of the ResultList entry.

Refer to Figure 10 for more information about property flags of the results.

Using Result.ReportText String

Result.ReportText is an additional string element in the Result container to store custom information for the step. During the generation of the report, TestStand checks to see if a non-empty string is passed and includes a new entry that contains the content of Result.ReportText. Figure 9 shows an example of an HTML report that contains information from Result.ReportText.

Figure 9: Report That Contains Information in Result.ReportText of the Statement Step

Use the color option in the Report Options dialog box to customize the display colors in HTML and XML reports.

You can also use expressions, such as a post expression, to set Result.ReportText in every step. TestStand includes the ReportText in the report unmodified. You can also use HTML tags to format the ReportText string for HTML or XML reports.

Adding an Entry to Step.Result Container

Because Step.Result is a container for different kinds of data, you can add new subproperties by using the TestStand API or by customizing a step type. TestStand adds all components of the Step.Result container to the ResultList as soon as the step finishes execution.

However, TestStand does not automatically add all the information from the ResultList to the report. You must set property flags correctly for TestStand to include the new data in the report. However, setting these flags does not necessarily include the values in the report. Refer to the Report Body Details section for more information on why the report might not include these values even though you set the flags.

Use the following flags, as shown in Figure 10, to include data in the report::

• IncludeInReport – You must set this flag to include data in the report. .
• IsMeasurement – Set this flag to filter the value using the Include Measurements option in the Report Options dialog box. IncludeInReport must be set for this flag to take affect.
• IsLimit – Set this flag to filter the value using the Include Test Limits option in the Report Options dialog box. IncludeInReport must be set for this flag to take affect.

Figure 10: Flags of Properties with Remark for Reporting Flags

Using the Execution.AddExtraResult Method of the TestStand API

Use the Execution.AddExtraResult method to add certain step properties to the ResultList when you call this method. The default SequentialModel.seq file includes a good example of how to use this technique. The default result collection process does not collect limits and comparison types of limit steps because they are not part of the Result container. However, you can enable options in the Report Options dialog box to include this data.

The process model includes a Setup Result Settings subsequence that contains two calls of the Execution.AddExtraResult method – one for the limits and one for the comparison type. Both steps have preconditions for the setting in the Report Options dialog box just as shown in Figure 11. After you call the Execution.AddExtraResult method, TestStand adds all Step.Limits and Step.Comp values to the ResultList and uses this data during generation of the report.

Figure 11: Setup Result Subsequence of the Process Model

Preventing Result Collection

You cannot alter the default result collection process, but you can disable it in the following ways:

• Step Level – If you disable the Record Results option on the Run Options panel of the Step Settings Properties tab, TestStand does not add the result container of the step to the ResultList after the step executes. If you disable the Record Results option for a Sequence Call step, TestStand does not insert a new container for the step and does not include the calling step or the subsequence in the report.
• Sequence Level – If you select the Disable Result Recording for All Steps option on the General tab of the Sequence Properties dialog box, TestStand does not add the results of any steps in the sequence to the ResultList. If you select the Disable Result Recording for All Steps option for a Sequence Call step, TestStand includes the sequence in the report but does not include any information for the subsequence because TestStand discards the ResultList of the sequence after the sequence executes.
• Station Level –If you select the Disable Result Recording for All Sequences option on the Execution tab of the Station Options dialog box, TestStand does add any results to ResultList and therefore does not generate a report for this station because an empty ResultList disable report generation.

Filtering the ResultList

You can delete results from the ResultList before report generation so the report generates more quickly and includes only certain step results by using a SequenceFilePostResultListEntry callback sequence in the client sequence file. TestStand calls this engine callback every time it adds a result container of a step to the ResultList.

The SequenceFilePostResultListEntry callback includes the following parameters: 

• Step – A reference to the step that was just executed and indicates that the step is not active anymore and the result contained in this container was already added to the ResultList of the sequence.
• Result – A reference to the result container that was previously added to the ResultList as a new array element. Use this parameter to change the content in Result.
• DiscardResult – A Boolean value that specifies to delete the result container from the ResultList.

For example, you might want to create a report that contains only information about steps that return a Failed or Error status so you can create an error report or because you want to reduce memory usage during sequence execution. You can accomplish this task by adding a SequenceFilePostResultListEntry callback to the sequence file to delete all results from the ResultList that do not match the Failed or Error restrictions.

In the callback, create a new statement and name it Remove Result. Define a precondition as ‘Parameters.Result.Status != "Failed"  && Parameters.Result.Status != "Error"’. Enter ‘Parameters.DiscardResult = true’ as the expression of the step

Return to the MainSequence in the sequence file and add several new steps. Set the Run Mode of one step to Force to Fail and execute the sequence. Notice that the report contains information only about the failed step(s). Return to the sequence file and set the Run Mode of the step back to Normal, execute the sequence again, and compare the report.

Note: Using this filtering technique can help to reduce the memory the ResultList needs; but this technique also generates a large overhead because TestStand has to execute the callback after each step in the sequence file executes.

Report Generation

The process model uses the ResultList during the report generation process to create one part of the report and gathers information elsewhere for other parts of the report. If you do not use a process model during execution, TestStand does not create a report.

Report Components

A report consists of a header, a body, and a footer.

Header

The header contains general information about the test and the test system itself, such as the station ID, the date, the time and the operator. The process model acquires most of this information from the Get Station Info subsequence before the MainSequence executes.

Figure 12: Header of an XML Report

Body

The body contains the results of the tests for the specific unit under test. The results in the ResultList, the property flags, and filters you apply during report generation define the content of the report.

Figure 13: Body of an XML Report

Footer

The footer contains only the following sentence: “End UUT report”. ATML reports do not contain a footer.

Figure 14: Footer of an XML Report

Report Generation Callbacks

Report generation involves many sequences. For example, in the Single Pass entry point of the SequentialModel.seq, almost half of the steps in the Main step group are for reporting, as outlined in green in Figure 15. Some of these sequences are available as callbacks in client sequence files. The Test UUTs entry point also uses the steps shown in Figure 15.

Figure 15: Reporting Steps in Main Step Group of Sequential Model Single Pass Entry Point.

The first task for report generation is to retrieve the settings for the report by using the Get Report Options sequence, which contains a call to the ReportOptions callback sequence that you can override.

The steps in the second green box in Figure 15 prepare the reporting. For example the Clear Report step resets the content of the report-object. The Setup Result Settings sequence document adds limits and comparison types to the ResultList by calling the Execution.AddExtraResult method.

The steps in the third green box in Figure 15 actually generate and save the report. The TestReport callback gathers the entire content of the report into one string, and the Write UUT Report step saves the string to disk.

You can modify the ReportOptions and TestReport callbacks to customize reports. Override ReportOptions when you want to have a static set of settings for the report that do not depend on station settings, such as when you want a specific report format and a defined path to the report files. Override the TestReport callback to include a custom approach to create reports and replace the entire report. Do not override the TestReport callback if you want to change only some parts of the report.

You can modify the ModifyReportHeader, ModifyReportEntry, and ModifyReportFooter callbacks, which the TestReport callback calls, to modify these related elements in the report. However, the ModifyReportHeader, ModifyReportEntry, and ModifyReportFooter callbacks execute only when you enable the Sequence option of the Select a Report Generator for Producing the Report Body option on the Contents tab of the Report Options dialog box.

Refer to the Additional Examples section of this document for more examples of when to override callbacks.

Report Body Details

To fully understand the report components you can customize, you must first understand the way TestStand generates the body of the report. In the Report Options dialog box, you can select to use a sequence or a DLL to create the body of the report. The PropertyObject.GetXML method creates the body of XML reports.

The DLL includes the same functionality as the sequence. However, because the DLL is written in LabWindows^TM\CVI^TM (a National Instruments ANSI C development environment), it is faster during execution compared to the sequence. Because the sequence can be easier to understand and modify, this document focuses on using the sequence to generate the report body.

The body of the report contains the results of the steps. TestStand analyzes the ResultList of the execution, gathers the information to include in the report, and generates the body of the report. Figure 16 shows the hierarchy of the sequences involved in the process of creating the report body.

Figure 16: Hierarchy of Sequences Used to Generate Report Body

The TestReport sequence collects the strings for the header, footer and body of the report and concatenates the string to the full report.

The AddReportBody sequence collects the strings for the results of the sequences and concatenates the strings to the body of the report.

The AddSequenceResultToReport sequence collects the strings for the results of all the steps in the given sequence. TestStand concatenates additional formatting strings, such as indenting, before and after the results.

The PutResultsInReport sequence collects the strings of each step in the ResultList and adds some formatting strings if needed.

The PutOneResultInReport sequence browses the result of the given step in the ResultList and formats the results into a string. The string contains the filtered results and formatting information for the given report type. TestStand filters the results based on property flag values in the ResultList. For example, the string does not include properties that enable the IsLimit flag unless you enable the Include Test Limits report option. If the step is a Sequence Call step, TestStand makes a recursive call to the AddSequenceResultToReport sequence. After building the string, the sequence calls the ModifyReportEntry sequence.

The PutOneResultInReport sequence calls the ModifyReportEntry sequence when TestStand completes the report string for the single step. You can override this callback sequence, which receives the entire string for the report of the step.

On-The-Fly Reporting

As explained in the previous section, the default report generation process creates one string that contains all the necessary data for the report, including formatting information, before saving that string to disk, which can lead to memory errors if the ResultList is substantial.

You can use the On-The-Fly Reporting option on the Contents tab of the Report Options dialog box to control the size of the ResultList and avoid memory issues. With On-The-Fly Reporting, TestStand collects the string that contains the results of steps, periodically writes that string into a temporary file, and deletes the result from the ResultList if you enable the Conserve Memory and Display Latest Results option on the Contents tab of the Report Options dialog box. TestStand deletes the temporary file and saves the final test report to a file at the end of a UUT loop execution. However, using On-The-Fly Reporting creates an overhead during execution of the sequence because TestStand interacts with the temporary file, which can lead to a slow down between executions of single steps.

Changing the Style of Reports

Before you change the way the report displays information, make sure you are using the best report format for the data. The following table outlines the advantages and disadvantages of each report format.

Report Format	Advantages	Disadvantages
ASCII (American Standard Code for Information Interchange)	Many tools available to read the report Creates the smallest files for reports	Cannot include images Time consuming during parsing (automated report analysis)
HTML (HyperText Markup Language)	Can include images and functionality to create a great visual impact	Need web browser for proper display Time consuming during parsing Formatting information and data bound together
XML (Extensible Markup Language)	Can include images and functionality, such as expandable trees Standardized for parsing Formatting information not bound to data (TestStand includes three different stylesheets)	No standard for test and automation industry Need parser for display as defined in stylesheet (will work in Microsoft Internet Explorer 5.0 and later; other browsers might list only content of XML-file)
ATML (Automated Test Markup Language)	Standardized report format for test, default stylesheet included Optimized for parsing	Need parser for display as defined in stylesheet

You can use report files for automated report analysis, especially XML and ATML report files, but using databases for automated analysis is more efficient.

XML Overview

Unlike HTML, XML files do not require any formatting information to be part of the document. Therefore, not every HTML browser is capable of correctly displaying XML files. Microsoft Internet Explorer 5.0 was the first browser capable of displaying XML files, including the formatting information that is normally stored in an additional stylesheet file. The stylesheet can be defined by Cascading StyleSheets (CSS) or by the Extensible Stylesheet Language (XSL). TestStand uses XSL files for formatting information for XML reports. TestStand delivers three different XSL files for XML reports.

Complete the following steps to see the effect stylesheets have on XML file display.

1. Use a sequence with several steps to create an XML report using the report.xsl stylesheet.
2. Use Internet Explorer to open and view the report file. Depending on the security settings, Internet Explorer might return warnings about scripts running in the background.
3. Use WordPad to open the report file and view the code that contains the information about the results. The first line of the file should read "<?xml version="1.0" encoding="iso-8859-1" ?><?xml-stylesheet type="text/xsl" href="D:\Program Files\National Instruments\TestStand 4.1\Components\Models\
   TestStandModels\StyleSheets\report.xsl"?>"
4. Change report.xsl to xyz.xsl and refresh the document in Internet Explorer. You should see an error, as shown in Figure 17.
5. Switch back to WordPad and change xyz.xsl to expand.xsl and refresh the document in Internet Explorer.

Figure 17: Error for Missing Stylesheet

As you can see, the display of the report changes depending on the stylesheet you use. Stylesheets can also include additional functionality, as shown in the expand.xsl stylesheet. Notice that no other changes were made to the XML file, so the content of the report itself stays the same. You can define an appropriate stylesheet to customize the style of XML reports.

Conclusion

Customizing reports in TestStand is a common task. Therefore, TestStand creates reports in a modular way so you can modify or replace modules to customize the report content, functionality, and style. Refer to the flowchart in Figure 18 as you make decisions about how to implement custom reports.  Click on the [+] Enlarge Image link to view image in higher resolution.

[+] Enlarge Image

Figure 18: Flowchart for Selecting the Correct Technique to Customize Reports

Additional Examples

Adding an Image to the Header or Footer

One of the most common report modifications is to include an image in the header of a report. The following example explains how to accomplish this for the HTML report format. Because this change affects only the header or footer, you can change those components directly.

If the default header is acceptable for the report, you can include the ModifyReportHeader callback in your client sequence file and insert the following statement: Parameters.
ReportHeader += “<img src=\”C:\\Windows\\Coffee Bean.bmp\”>” If you leave out the ‘+’, the image replaces the entire header of the report.

If you use the default Sequential Model to execute the MainSequence in this file, the report contain the coffee bean image between the header and the body of the report.

If you want the image to appear before the header, use the following statement instead: Parameters.ReportHeader = “<img src=C:\\Windows\\Coffee Bean.bmp>” + Parameters.ReportHeader.

Using the callback to place the image within the header is a more complex task because TestStand analyzes the Parameters.ReportHeader to split the header into strings and insert the image between the strings. In this case, it might be easier to change the AddReportHeader or AddReportFooter sequence in the reportgen_<reporttype>.seq file. Do not make changes to the installed TestStand files. Copy the files you want to modify into the <User> directory in TestStand 4.0 and earlier or into the <TestStand Public> directory in TestStand 4.1 and later and make changes to the copies of the files.

Adding an Image to the Report of Each Step Based on the Step Result

Adding an image to the step result in a report is similar to adding an image to the header or footer. You can work with the callback (ModifyReportEntry in this case) or modify the reportgen_<reporttype>.seq file. This example shows how to work with the ModifyReportEntry callback.

Insert the ModifyReportEntry callback in your sequence file. The process model calls this callback each time the report generation process finishes and creates the string for a single step. ReportEntry is a string property object that contains the string TestStand writes in the report. For HTML reports , the ReportEntry TestStand passes to the callback already contains all the step result information and the HTML tags in a table. You can add a new row to the table and create a cell that contains the image you want.

Modify the ModifyReportEntry callback to contain a single statement with the following expression: Parameters.ReportEntry += "<tr><td><img src=checkmark.bmp></td></tr>" If you don’t include any preconditions to this step, TestStand inserts the checkmark image for every step, so extend the statement with the following precondition: Parameters.Result.Status == "Passed".

Note: The checkmark.bmp must be in the same directory as the report file.

Removing Indenting from Subsequences

Removing the indenting from subsequences in a report depends on the report format and stylesheet, if available, you use.

ASCII

The AddSequenceResultToReport sequence in the reportgen_txt.seq file adds the indenting by computing the number of blanks needed in relation to the indent level. Therefore, you need to make the following two changes to remove all indenting:

• Set the Compute Indentation String step to Skip.
• Change the value of the Add Sequence Results step Level parameter from Parameters.Level to 0.

HTML

The AddSequenceResultToReport sequence in the reportgen_html.seq file adds the indenting by using <Blockquote>. Set the Add Starting BLOCKQUOTE and Add Ending BLOCKQUOTE steps to Force Skip to remove indenting in the report.

XML (Horizontal Stylesheet)

The horizontal stylesheet defines the following global variable: var gIndentTables = true; // indent tables or not. The default stylesheet sets this value to True. Change the variable to False and save the change to remove indenting in the report. Set the variable back to True to enable indenting again.

About the Author

Norbert Brand - National Instruments

Norbert Brand is an Applications Engineer Specialist for National Instruments in Germany with half a decade of experience in the automated test industry.  Norbert works in the National Instruments support team and helps customers by creating proofs of concept and providing architectural guidance for National Instruments software and hardware products including NI TestStand.

Reader Comments | Submit a comment »