Report Generation Explained

TestStand provides the ability to automatically log data into a report file. This document helps to understand how TestStand implements report generation. This document does not discuss result collection from step execution or report customization.

Report Generation Overview

Sequence files using the default process model execute in one of two ways: Test UUTs (cyclical execution) or Single Pass. At the end of each execution the process model generates a report file detailing the results of the execution. Single Pass execution generates a report when the execution completes. For Test UUTs, results are collected and appended after each UUT is tested but the report is displayed only when the user stops the execution.

Report generation is a task performed by the TestStand process model. When a user executes a sequence, the process model begins by executing the specified entry point. After the process model executes the test sequence, it calls TestReport callback sequence. TestReport callback sequence formats the data into a report string.

Figure 1-1: TestStand default process model file (SequentialModel.seq showing Test UUTs sequence (the execution entry point). This sequence contains a sequence call step that calls TestReport sequence. TestReport callback sequence is the one that generates a test report.


Result data from a test sequence is collected and stored in allocated variables. During a sequence execution, Locals.ResultList array variable stores step result from the executed sequence. This array variable exists in every sequence that executes in TestStand. The result data is stored into Locals.ResultList for two reasons:

A. To gather all data and store it in an array so that this array can be passed to the process model and other sequences
B. To make step result from one step available to another step that executes later in the sequence file.

Figure 1-2: Results from each step within a sequence file are stored in a Locals.ResultList array variable. Each element of the array stores step results from MainSequence.
The Report Options dialog window is located under the Configure menu of TestStand Sequence Editor. From this window various report attributes that determine items to be logged in the report can be selected. Generating a report after a sequence executes is an option that can be toggled on or off. For example, someone may want to disable report generation for debugging purposes, however, once the sequence is debugged they may want to enable report generation. Upon installation, TestStand enables report generation such that test reports are generated in HTML format.

[+] Enlarge Image

Figure 2-1: Report Options dialog window.

Other attributes that can be set through the Report Options dialog window are: Execution Times, Step Results, Output Values, Test Limits and Append Report if File Already Exists.

• Include Step Results option determines whether to include results from each step into the report. Disabling this step will not include any information from any step of the sequence because this option sets a Boolean (in the default process model this Boolean value can be obtained from Parameters.ReportOptions.IncludeStepResults) that is checked before executing “Get Report Body” step from the process model. When this option is unchecked the Boolean is set to False and report body generation is skipped.
• Include Execution Times option allows TestStand to record how long (in seconds) it took to execute a step as well as the total execution time for the entire test sequence.
• Include Test Limits option includes in the report the limits for the step type.
• Include Measurements adds to the report all result properties that indicate that they contain measurements. When you include measurements, you can configure the following options:
  Include Arrays: Specifies whether the report displays measurement values that are numeric arrays. You can select from the following options:
  Do Not Include Arrays: Numeric array values do not appear in the report.
  Insert Table: Numeric array values appear in tables.
  Insert Graph: Numeric array values appear in graphs. This option is only available for HTML and XML reports.
• Append if File Already Exists option determines whether to create a new file for each UUT or to append the data to an existing report file.
• On-The-Fly Reporting: Specifies whether TestStand generates reports after testing a UUT or generates reports concurrently with execution. If you enable this option, the process model uses Post Result callbacks to append new formatted results to the report. When disabled, the process model creates the report after testing each UUT.

Other options that can be set from Report Options dialog window include TestStand’s ability to generate a report in one of three ways: ASCII text format (.txt), HTML format (Web Page) or XML Format.

• Report Format: Specifies the output format of the report file. Use this ring control to select an HTML Document (.html), an ASCII-Text File (.txt), or an XML Document (.xml).
• Report Colors: Specifies the colors of the report. This option is only available for the HTML or XML Document formats.
• Stylesheet: Specifies the path to an XSL stylesheet file that TestStand stores in the processing instruction of an XML report. You can specify an absolute or relative pathname for the file. Report generation determines the absolute path to a stylesheet file using the TestStand search directory paths. This option is only available for the XML Document format.
• Store Absolute Path: Instructs TestStand to use the absolute path for the stylesheet file when storing the path in the processing instructions of an XML report. Relative pathnames in XML reports are relative with respect to the file path of the XML report.

[+] Enlarge Image

Figure 2-2: Report Options dialog window has an option to change report format between HTML, TXT and XML.
The Report Options dialog window also provide two methods for generating a report. These methods, regardless of report format (whether HTML or ASCII format), determine which report generator file to use for generating the report body. The two methods are:

1. Using sequence files reportgen_html.seq or reportgen_txt.seq (both files can be referred as reportgen_xxx.seq)
2. Using DLL (modelsupport.dll)

[+] Enlarge Image

Figure 2-3: Report Options dialog window allows users to change between a sequence file or a DLL to generate the report body.

When the first method is selected, the process model determines at run-time which of the two sequence files to use to generate the report. Both of these files are located in <TestStand directory>\Components\NI\Models\TestStandModel directory. When the DLL option is selected TestStand executes “Call Get Report Body (C Implementation)” step from the TestReport sequence of the process model and skips “Get Report Body” step from the same sequence. However, the header and the footer of the report is generated using one of the reportgen_xxx.seq sequence files. The difference between using the two methods is that the DLL report body generator is more efficient when generating a very long report (or generating report for an execution that has been running for a long period of time). The drawback for using DLL method is that this method does not call ModifyReportEntry callback because the callback is made only when generating the report body from reportgen_xxx.seq sequence file. ModifyReportEntry callback can be used to add extra results to the report.

Report Composition: Header, Body, Footer

The report generated after the execution consists of three parts: Header, Body and Footer. This composition is defined in the TestReport callback of the process model. This callback makes further sequence calls using “Get Report Header”, “Get Report Body” and “Get Report Footer” steps. It should be noted that header, body and footer are executed and prepared independent of each other. Their contents are stored in separate variables (Locals.ReportHeader, Locals.ReportBody and Locals.ReportFooter) until the “Concatenate Report Header, Body, and Footer” step of TestReport sequence is executed. This step puts together the three groups of the report.



Figure 3-1: TestReport sequence of the default process model composes and assembles the test report. “Get Report Body”, “Get Report Header” and “Get Report Footer” are calls made to sequences in reportgen_xxx.seq file.

“Get Report Header” step executes the AddReportHeader sequence from reportgen_xxx.seq file. “Get Report Footer” step executes the AddReportFooter sequence from reportgen_xxx.seq file. Contents of the header and footer are simple. By looking at AddReportHeader and AddReportFooter sequences, one observes that these sequences are made up of several steps without any calls to other sequences. This makes it easy to modify header and footer sections of the report.

Figure 3-2: “Get Report Header” sequence call makes a call to AddReportHeader sequence from reportgen_xxx.seq file. The decision as to which reportgen_xxx.seq file to call is made programmatically during process model execution.

“Get Report Body” step calls AddReportBody sequence from reportgen_xxx.seq file. This sequence is different from AddReportHeader or AddReportFooter because the body of the report has to consider the filtering expression specified in Report Options. There are also nested sequence calls in the AddReportBody sequence. Figure 3-3 shows how data – results from the test sequence – is passed from the process model to reportgen_xxx.seq sequence file.

[+] Enlarge Image

Figure 3-3: The image shows sequence progression starting from TestReport sequence of the process model all the way to PutOneResultInReport of reportgen_xxx.seq file. This progression shows report body generation

Each sequence in reportgen_xxx.seq sequence file (shown in Figure 3-3) has a few steps along with a call to the next sequence. For example, PutResultsInReport sequence has several steps plus one call to the PutOneResultInReport sequence. The last sequence in this chain (PutOneResultInReport) is executed recursively in a loop because this sequence is the one that contains steps that actually put data into the report variable (string). The reason for looping on this sequence is because each call to the sequence takes results from one step and formats them into the report. For example, if the sequence MainSequence had ten steps, then PutOneResultInReport will loop ten times, each time formatting the results from a single step into the report.

In the PutOneResultInReport sequence, not all steps write to the report. Some steps perform calculations such as computing background color. However, steps that contain “Add…” in the step name, for example, “Add Numeric Measurement” step, are the ones that actually write to the report variable. Each of the “Add…” steps add data into the report by first performing a check to see whether a particular property exists. This is done by using PropertyExists(“”) function in the precondition expression for the step. PutOneResultInReport sequence contains steps that use all available properties for all default step types. For example, this sequence contains “Add High Limit” and “Add Low Limit” steps. But only the Numeric Limit Test step contains these two properties. So, for example, when it logs results for an Action step, the precondition expression will cause “Add High Limit” and “Add Low Limit” steps to be skipped.

There is an added capability in TestStand to use callbacks in order to modify any part of the report (header, body and footer). Looking at the TestReport sequence of the process model, there are two particular steps named “Call ModifyReportHeader Callback” and “Call ModifyReportFooter Callback”. Notice that each of these steps execute after making calls to sequences that generate report header and report footer. For example, “Call ModifyReportHeader Callback” step is called after executing the “Get Report Header” step.

Furthermore, reportgen_xxx.seq file contains a similar callback step (“Call ModifyReportEntry Callback”) in PutOneResultInReport sequence. This step makes a call to ModifyReportEntry sequence. The purpose of this callback is to append additional customized messages to each step result.

These callback steps can be overwritten. Notice that these callbacks make calls to their respective sequences - ModifyReportHeader, ModifyReportFooter and ModifyReportEntry - that exist within the process model. By including a sequence of the same name in the current sequence file, these callbacks will use these new sequences rather than the ones in the process model. This method is called “overriding callbacks”.

Figure 3-4: Callback sequence can be overwritten simply by naming a sequence in the sequence file by the same name as the callback sequence. This image shows how ModifyReportEntry callback sequence is overwritten.

Note: Sequence file callbacks can be included in the sequence using two methods.

1. A sequence by the same name is manually added in the sequence file.
2. A sequence can be added by browsing to Edit>>Sequence File Callbacks… Highlight the appropriate callback from the list and click the “Add” button. This will include the particular callback in the current sequence file.

Reader Comments | Submit a comment »