Synchronizing Multiple CompactRIO Chassis

Test engineers who develop distributed acquisition and control systems using the CompactRIO platform can use digital input and output modules to implement synchronization between among CompactRIO systems. CompactRIO FPGA technology and the LabVIEW FPGA module provide flexible tools for implementing application-specific triggering, timing, and synchronization solutions that outperform solutions based on standard communication buses such as Ethernet.

Introduction

CompactRIO provides a rugged and compact platform for developing a wide range of applications in data acquisition and data logging, machine control, closed loop control and more. The flexibility of choosing from a 4-slot and 8-slot chassis and selecting the individual modules enables a developer to select from a virtually endless combination of channels and configurations for their application. However for some applications the number of slots of a single chassis and the associated number of I/O channels may not be adequate to satisfy the needs of the application. In this case multiple CompactRIO chassis may be needed. Another scenario that may require multiple chassis is the distribution of an application over a large physical area where we want to bring the data acquisition and control platform closer to each of multiple locations. In this case different CompactRIO systems will perform part of the overall application and will require some amount of coordination and synchronization with a host system and possibly with each other.

Interchassis Synchronization

Synchronization between multiple CompactRIO systems can exist in a number different forms based on the needs of the application. Some may only require supervisory control and communication such as a start and stop commands from a host system, while others may require hardware level timing and synchronization between individual CompactRIO acquisition systems.

In a data monitoring and control application with a slow update rate (such as a waste water treatment plant), multiple chassis acquire data at different points in a large plant and send it back to a host application using TCP/IP for display and data logging. Based on the input data the host updates each chassis with output values to control devices such as valves and pumps. Control and update rates around 1 Hz are typical for such systems. Tight synchronization between individual Compact RIO systems is not required and timing coordination is handled by the host system using TCP/IP command communication that starts and stops processes and changes the set points controlled by each CompactRIO system. The LabVIEW Real-Time command-based architecture is an ideal solution for this type of application.

Figure 1: Host level synchronization of multiple CompactRIO systems

The next level of synchronization between CompactRIO systems is a common hardware trigger that synchronizes events between systems. In this type of application we can use a common trigger to start the data acquisition on multiple systems and rely on the onboard clock in each system to control the acquisition timing. This approach is suitable for applications where the onboard clock is accurate enough to keep the overall process synchronized within tolerable boundaries during the length of the process. Over longer periods of time, we may need to resynchronize periodically by using repeated triggers.

A trigger between systems can be implemented using digital input/output modules in each system. The master system will send a digital pulse on one of the digital outputs. Each of the slave systems will be looking for this pulse on one of their digital inputs and start their acquisition process when they see the pulse. The coding for this implementation will be similar to the scan clock example described below.

Figure 2: Hardware-level trigger synchronization of multiple CompactRIO systems

A more advanced method of synchronization is to share a clock signal between multiple CompactRIO systems and use the external clock to run specific processes on each CompactRIO system that need to be synchronized. The most common such application is to share a scan clock and run data acquisition and generation processes at the same rate fully synchronized on multiple systems. The following example describes the setup and implementation of this type of application in detail.

Figure 3: Hardware-level clock synchronization of multiple CompactRIO systems

See Also:
Command-based Architecture for LabVIEW Real-Time

Master and Slave Example

To implement a fully hardware synchronized solution between multiple CompactRIO chassis we need a digital I/O module in each chassis. For simple unidirectional synchronization, we need to generate a digital signal from the master device and read the digital signal on each of the slave devices. We can use bidirectional digital modules (cRIO-9401) in each device or use a digital output module on the master (cRIO-9474, cRIO-9472) and digital input modules on the slaves (cRIO-9411, cRIO-942x). Each module has a maximum update/read rate so we need to ensure that the modules are fast enough to handle our synchronization requirements.

For applications that require CompactRIO chassis to be separated by a considerable distance in which noise may be a factor affecting the synchronization signals, differential digital signals should be used in conjunction with the proper shielding or twisted-pair wiring. The cRIO-9411 module should be used to read the differential digital signal in the slave chassis.

Figure 4: Compact RIO Setup for synchronizing using a shared scan clock

To generate a scan clock from the master device we output a digital pulse train with a defined pulse width and pulse period from one of the digital lines on a CompactRIO module. This pulse train is started when all the slave devices and the master device have been armed and triggered to read their clock inputs and are ready to process the first scan clock pulse.

[+] Enlarge Image
Figure 5: LabVIEW FPGA diagram to generate the external scan clock on the master device

Each slave device will monitor its clock input to detect edges on the clock signal. Depending on the application it may look for rising, falling, or both types of edges. When an edge is detected it will run through the process that is controlled by the external clock once and then wait for the next edge on the clock signal.

Figure 6: LabVIEW FPGA diagram to detect the external scan clock on a slave device

To synchronize the master and slave devices more accurately, the master device in fact is both a clock master and a clock slave. It is responsible for generating the external scan clock. However, its acquisition process is also clocked by the external clock, in the same manner as all of the slave device are controlled. The following code segment is used both in a slave and master FPGA diagram to acquire data based on the external scan clock and buffer it for transfer to the LabVIEW Real-Time host application. 16-bit data channels are compacted into 32-bit integers for increased data transfer performance.

[+] Enlarge Image
Figure 7: LabVIEW FPGA diagram to acquire four analog input channels based on an external scan clock

The remainder of the LabVIEW FPGA applications on the master and slave includes code to retrieve the calibration constants for the analog input channels, a Wait process that allows the host application to arm the acquisition to start looking for the first scan clock pulse, and a loop to transfer the acquired data from the FIFO to the host application. All of the code is included in the example referenced from this application note.

Clock Synchronization

Using the LabVIEW FPGA VIs for the master and slave we can build corresponding LabVIEW Real-Time host VIs to run on each of the two CompactRIO processors. the two applications are very similar in design and only vary slightly as to the parameters used to configure and control their FPGA VI. The master application includes a couple of extra controls to configure the scan rate and pulse width of the scan clock, and one additional command to the FPGA to start the scan clock.

In both host VIs we initialize and download the FPGA personality. After setting up the acquisition parameters, we start running the VI on the FPGA and then wait for the FPGA to retrieve the analog input channel calibration values. In this portion of the host the master specifies the scan rate and pulse width which are not set on the slaves.

Figure 8: LabVIEW Real-Time code to initialize the FPGA acquisition VI

After the FPGA VI has read the calibration values, it is ready for acquisition. The acquisition process is armed from the host VI. This places the FPGA VI in a state where it is monitoring the scan clock line and looking for the first pulse to perform the first conversion on the analog input. The arming process is initiated using a button on the front panel of the RT host VI. The acquisition processes are explicitly armed in the application to make sure that all chassis are ready at the same time. This allows us to start the RT host VIs at different times, perform other operations, and then arm the acquisition on all the FPGAs at about the same time.

When all of the CompactRIO chassis are ready and armed for acquisition, the RT host VI on the master starts the generation of the scan clock. The scan clock signal is physically passed to all of the slave chassis initiating the individual conversions on the analog inputs. The FPGA on each chassis monitors the number of scans that have been performed. When the configured number of scans have been acquired the FPGA indicates to its host application that the data can be read from the FPGA buffer. This setup necessitates that the data for the complete acquisition can be stored in the user memory on the FPGA. If more scans are desired than can fit in the FPGA buffer, data needs to be transferred to the host while the acquisition is still ongoing.

Figure 9: LabVIEW Real-Time code to arm and start the acquisition in the FPGA VI

After the acquisition is complete, the data is retrieved from the FPGA buffer one integer at a time. Each integer that is retrieved is acknowledged so that the FPGA places the next value from the buffer in the data indicator. The collected data is converted back I16 integers representing the samples for the analog input channels and then converted to a two-dimensional array representing the channels and scans from our acquisition process. The host then reads the calibration values from the FPGA and uses them to scale the binary data to voltage data and displays the voltage data on a graph.

Figure 10: LabVIEW Real-Time code to read data from the FPGA VI and scale it for display

By measuring the same signal (such as a sine wave) on both the master and the slave chassis, we can test the synchronization of our application and see any skew in the system caused by propagation delay of the scan clock signal from the master to the slaves, or other possible timing differences between systems. The following figure shows a simple 2 kHz acquisition from two synchronized systems of the same signal applied to both.

Figure 11: LabVIEW Real-Time Master and Slave Host VIs

See Also:
Synchronizing Multiple CompactRIO Chassis

Host Integration

In most applications the LabVIEW Real-Time VI shown in this example will be integrated with a host application running on a Windows machine or other system used for visualization, data logging and other supervisory processes. Because the CompactRIO system has not direct method to display data on a front panel we need the additional host to retrieve data from the CompactRIO system for display. In an application where multiple CompactRIO system are synchronized the host system will retrieve data from multiple CompactRIO application and display or log the synchronized data.

The LabVIEW Real-Time command-based architecture, described in the application note linked below, provides a good starting point for developing your host application to integrate with the different CompactRIO chassis used for acquisition.

See Also:
Command-based Architecture for LabVIEW Real-Time

Considerations

The example illustrated above is a basic implementation of an external scan clock between two or more CompactRIO chassis. Additional slave systems using the same code as shown above can simply be added to the system. For proper synchronization all of the chassis in the system need to be initialized and started in the correct order so that the master and slave acquisition process are all ready for the first clock pulse. This is fairly simple in a one-time single shot acquisition shown here. Additional code is needed to extend the application to acquire continuously or to retrigger the acquisition.

For a continuous acquisition the FPGA VI needs to be changed so that data is transferred from the FPGA to the RT host while the acquisition is running. This means the RT host application needs to start reading data immediately after the start of the acquisition rather than wait for the completion of the acquisition. The FPGA VI should keep track of the acquired and buffered data so that it triggers the host to read data when a set amount of data (data block) has been acquired.

For an application where a synchronized single shot acquisition should be retriggered from the RT host application, the FPGA VI needs to loop back after the completion of one acquisition to be ready for the next acquisition. After completing one pass through the acquisition and transferring the data to the host, the VI should return to the while loop where it waits for the Arm command from the host application.

Download Example

Follow this link to download a LabVIEW Example of this concept

Synchronizing Multiple cRIO Chassis - Example

3 ratings | 5.00 out of 5

Print | PDF

Reader Comments | Submit a comment »

Add a link
Great article, very helpful. I read this a while back and didn't realize there was another way to communicate deterministically between cRIOs. You should include a link to your article entitled, "Using Time-Triggered Networks to Communicate Deterministically Over Ethernet with the LabVIEW 8 Real-Time Module" Link: http://zone.ni.com/devzone/cda/tut/p/id/4720
- rex1030@gmail.com - Oct 29, 2007

Exactly what we needed
We are looking at putting together a multi-chassis DAQ system. A major concern was synchronization. This is an excellent, detailed architecture that will drop easily into our high-level design.
- Daniel Marlow, General Dynamics Land Systems. marlowd@gdls.com - Nov 7, 2006