Event-Driven Circular Hard Disk Data Buffering

For engineers who build event-driven, high channel-count data logging systems, hard disk-based circular buffers allow the system to log a comprehensive data set. Unlike conventional RAM buffering, circular hard disk buffers support very long data records that can extend for minutes before and after the event.

Background and Scope

High performance data acquisition systems can generate very large amounts of data, often exceeding hundreds of gigabytes over several hours. If the system is unattended or if on-line analysis is not feasible, the data must be archived during acquisition for off-line analysis. In order to reduce the amount of data to a manageable level, the system can be designed to store only the data surrounding certain events. These events may include external triggers, measurement threshold triggers, and user commands. In machine condition monitoring applications, for example, events are often related to undesirable physical conditions such as vibration and elevated temperature or pressure. The archived data can be used to better understand these phenomena and avoid their occurrence in the future.

Event-driven data archiving can be achieved by creating a circular data buffer that can be copied to a file when an event occurs. This application note discusses a circular data buffer implementation that can handle the large volume of data produced by a high channel-count, high speed data acquisition system. The traditional LabVIEW circular buffer implementation uses shift registers or queues to store the data. The challenge with this approach is that the typical computer memory capacity is not adequate for high performance applications that may fill a large RAM-based buffer within several seconds. In many cases it is desirable to look at data several minutes prior to and after an event, requiring a buffer capacity approaching or exceeding a gigabyte.

This paper assumes that you have a solid understanding of LabVIEW programming methods and understand the basic concepts related to queues and file I/O. At the end of this paper, you will find a link to the example program that illustrates the methods discussed in this paper.

Hardware Considerations

The approach discussed in this document uses hard disk space to store data relative to events, so the system must have enough disk space to handle the circular buffer and any archived event data . As will be discussed later, the circular buffer is implemented using two buffers that are each the size of the event capture window. The system will need enough free disk space to handle these two buffers and the archived event data (equivalent to one buffer per event). In the example program linked to this document, all channels are stored together in blocks of data in each of the two buffers that make up the circular file buffering scheme. The amount of data that needs to be accessed and transferred when an event occurs depends on the amount of time or the number of samples the user is requesting prior to, and after the event. The free disk space required for each event archive in the example program can be calculated using the following equation:

Disk space per event archive in bytes = (bytes/sample) x (samples/sec./channel) x (length of event related data in sec.) x (number of channels of data).

Note that this is a rough estimate as there may possibly be several bytes within the file that contain header information, but the header is relatively small compared to the size of the data. The way the example program has been defined, each of the two buffers making up the circular file buffer need to also have a maximum of an event's worth of data in them once the buffer is filled. Each buffer, therefore, will contain the same amount of data as a single event archive file. For example, a 16-channel system acquiring 2 minute windows of event data (1 minute before the event and 1 minute after) at 25 KSamples/second using a 16-bit Data Acquisition (DAQ) device would result in the following calculation:

Disk space per event archive in bytes = (2 bytes/sample) x (25000 samples/sec./channel) x (120 sec.) x (16 channels) = 96MB.

This means we will need 192MB for both of the buffers that make up the circular file buffer plus 96MB for each event we plan to archive. If we plan to archive 100 events over the course of our test, then we would need a total of 192MB + (100 x 96MB) = 9.729 GB. So we would need at least 10 GB of free hard disk space to store all of the events and manage the circular file buffer. Now that we've discussed the hardware issues, let's turn to the implementation of the event-driven circular hard disk buffer.

Circular Buffer Basics

In a computer we generally think of memory on disk as being contiguous and linear. The circular buffer described in this document is actually two linear buffers that are used to implement a circular buffer. Figure 1 illustrates the concept behind the implementation described in the following sections.


The process begins with writing data to buffer 1 in Figure 1(a) . During this initial state, there is no data available to be read from either of the buffers. In 1(b), we see the system writing to buffer 2, The data in buffer 1 is available for reading, and can be copied to another location during this phase. Figure 1(c) illustrates the acquisition process overwriting the old data in buffer 1. During this phase, the data in buffer 2 is available to be copied. This process continues indefinitely, writing to each of the buffers sequentially and copying data that is currently available as necessary. With this type of buffering scheme, you can see how it is possible to copy data in a continuous manner.

The LabVIEW implementation of this circular buffering scheme is fairly simple. At the configuration stage of the application, we open a reference to two buffer files (buffer 1 and buffer 2). The files are kept open during the entire time that our application runs and are closed only when we exit the application. The main data acquisition loop acquires data from the source and writes the data file associated with buffer 1, then the file associated with buffer 2, then back to the buffer 1 file, and so on.

Events and Circular Buffering

Now that we understand the circular buffer mechanism we can move on to utilizing the buffer in reference to events that occur within the system. As mentioned previously, events can be generated by a variety sources, including analysis of the acquired data to look for a particular physical condition or phenomenon, an external trigger or user command. In many cases, the main requirement is to archive a user-specified amount of data (capture window) prior to and after the event so that it can be further analyzed off-line to better understand the physical conditions at the time of the event. In our implementation of the circular buffer, the buffer size is equivalent to the size of the capture window so that no more than two buffers are required, as you will see below. With this implementation, we end up with four unique cases that our code needs to handle. We'll label these cases A, B, C and D.


In Case A, the event occurs shortly after we start the program and have started writing to buffer 1 for the very first time. We can see in Figure 2(a) that when the event occurs, the amount of data requested prior to the event actually extends back before we began acquiring data and exists solely on buffer 1. In this case, we limit the data copied to begin at the start of acquisition. As previously discussed, the copying process takes place after we have finished writing to buffer 1 as illustrated in part (b) in Figure 2.

Case B is illustrated in Figure 3. We see in part (a) that the event occurs after the system has been running for awhile. When the event occurs, we happen to be writing near the beginning of buffer 2, so the capture window overlaps back into buffer 1. Since buffer 1 is available to read, we copy the buffer 1 portion of the capture window to the archive file. In part(b), we now have transitioned from writing to buffer 2, back to writing to buffer 1. The data in buffer 2 is available to read, so we now copy the rest of the event data to the archive file.


Case C is shown in Figure 4. We can see in part (a) that the event occurs towards the end of one of the buffers (in this case buffer 1). None of the data in the event capture window is available to be read until we toggle to write buffer 2 as shown in part (b). The data at the end of buffer 1 is now ready to be copied to the archive file. In part (c), we have again toggled back to writing to buffer 1 and the final section of the event capture window can be copied from buffer 2 into the archive file.


In Case D we see that the event occurs exactly within one buffer. Again, in Figure 5 part (a) we see that the event occurs in such a way that all of the data is contained in the buffer to which the system is currently writing (in this case, buffer 1). In part (b), we toggle to buffer 2 and buffer 1 is now available to copy data to the archive file.


When we implement this case in LabVIEW code, it turns out that Case D is a special occurrence of Case C.

Now that we have established each of the cases we will encounter when processing events in our system, we can start to implement the process in software.

Implementation of the Event Driven Circular Buffer

At the highest level, the example program (circBuffExample.vi), can be described using the flow chart illustrated in Figure 6.


The "Acquire data and process events" portion of the flow chart is where most of the action occurs within the program itself so let's discuss that portion next. We can further break this part of the code down into three main loops as shown in Figure 7. The top loop handles acquiring the data from some other process and writing the data to the circular file buffer, the middle loop manages the archive data relative to events and the bottom loop monitors for event occurrences. The subsequent sections of this paper will break down each of the three loops shown in the diagram in Figure 7.



As we discussed previously, the code in the top loop in the diagram in Figure 7 writes to the circular buffer files, toggling between them as each buffer fills up. During this process, we need to let the middle loop in our diagram know when we toggle from one file buffer to the other so that it can handle events and archive the data appropriately. Figure 8 shows a high level LabVIEW diagram that implements these concepts. As you can see, the Case structure in this loop executes the "True" case whenever we reach the end of one of the buffer files. This case puts a notification on the Command Queue that we've switched file buffers. The actual implementation of this code can be seen in the upper part of the diagram in circBuffExample.vi.


The middle loop in Figure 7 handles processing of the events by performing the actual archive process. This loop receives the buffer toggle commands from the Acquisition & Buffer Management Loop and event commands from the Event Monitor Loop. This loop is the most complex of the three loops, so we'll spend some time working our way through the various operations handled by this particular piece of code. Figure 9 shows a high level diagram of the operation of this loop. The implementation of this loop in LabVIEW utilizes a queue to pipe commands from the two other loops to tell this loop what to do next.

Figure 10 is the flow chart representation for the decision making process as to which Case the event falls into (A, B, C, or D). This decision is based on where the event occurs relative to the position of the write pointer for the current buffer. Once the evaluation has been made as to which case the event applies, the logic for that particular case is executed. The subsequent section further breaks each of these cases down. Please note that the LabVIEW example linked to this document has notations on the diagram indicating which parts of the code apply to each of the four cases. We will use flow diagrams here to help clarify the logic behind this code. You may find it useful to open up the diagram to circBuffExample.vi and related subVIs to make a visual connection between the code and the flow charts.


Case A:
A high level flow chart for Case A is illustrated in Figure 11. When a Case A data event occurs, we can see that from the flow chart in Figure 10 that we are writing to the very first buffer (buffer 0) and the data theoretically extends before the start of this current write buffer. Since this is the very first buffer we are writing to, this cannot be the case (we weren't writing any data previous to buffer 0). So we first need to modify the size of the event capture window so that it does not extend before the start of buffer 0. After making this calculation, we put the event into the Event Delay Queue for later processing. Once we receive a "toggle buffer command", data is now available to write, so we open a new archive file and write the necessary blocks of data from the buffer to the archive. Since all of the data we need occurs strictly on this first buffer, the process is completed.



Case B:
The flow chart for Case B can be seen in Figure 12. For Case B, we see that the event capture window overlaps into the previous buffer (the current read buffer), so that some of the data is available for immediate transfer. We open a new archive file and transfer the available blocks of data from the read buffer. The rest of the required data resides on the current write buffer, so we must delay the rest of the transfer by inserting the event with the number of blocks left to complete the event capture into the Event Delay Queue. Once we receive the indication that the buffering routine has toggled write buffers, we are now able to transfer the remaining blocks from the current read buffer and the process is complete.



Case C:
The flow chart for Case C can be seen in Figure 13. For Case C, the start of the event capture window occurs on the current write buffer. Consequently, the first step is to put the necessary information into the Event Delay Queue and then wait for a signal from the buffer logic that we are toggling write buffers. Once the toggle occurs, we can now open a new archive file and transfer the available blocks of data into the archive file. The process is still incomplete since part of the end of the event capture window requires data from the current write buffer. This requires us to put the event information back into the Event Delay Queue for a second time. Once the buffers toggle, we can now append the rest of the event data to the archive file and the process for Case C is complete.



Case D:
Figure 14 contains the flow chart for Case D. It turns out that Case D is just a special implementation of Case C. In Case D, as you may recall, all of the data occurs on the current write buffer. When we get this event we place it in the Event Delay Queue and once we toggle the write buffer, we open a new archive file and transfer all of the necessary data from the buffer to the archive. The process for Case D is then complete.

Now that we understand the mechanism behind managing the circular file buffer and writing archive files relative to events, we need to discuss briefly how the event itself is sourced to the Event Management & Archiving Loop. The third and final loop in the diagram in Figure 7 is the Event Monitor Loop. Figure 15 shows the basic implementation of event monitoring and sourcing in the example program included with this document.

This particular implementation is very basic. When the user depresses a Boolean control on the front panel of the example, a "data event" command is put in the Command Queue. Recall that this queue is monitored by the Event Management & Archiving Loop. When an event is delivered, the archiving process for that particular event is initiated and all the channels acquired are stored in the archive file relative to the event time. Much more can be done inside the Event Monitor Loop that makes the entire application much more powerful. Some possibilities for enhancement include piping some of the data being acquired in the Acquisition and Buffer Management Loop into the Event Monitor Loop and perform some real-time processing. We can then trigger events off of the processing results (e.g. crossing an amplitude threshold, frequency threshold, etc.). We could also monitor external devices via digital lines or some other communication and source an event based on this monitoring. The archiving process could be further enhanced by delivering additional information to the Event Management & Archiving Loop that would indicate which specific channels to store within the archive file (instead of storing all channels each time).

Archive File Headers

Previously, we talked briefly about the fact that we are storing header information with each of the archive files. The header we've chosen to use for this application uses the XML data format which is basically a string that has been formatted similarly to HTML. In the example, we take a LabVIEW cluster that contains information pertaining to the data stored, including the number of bytes per element of data, the number of channels of data and the number of elements per block per channel of data and convert this to an XML string to be stored with each archive file (see Figure 16). This information allows us to correctly format the data when we read it back out of the archive file at some later point. The only other thing we need to know is the data type of the data in the archive file. It should be noted that it is not necessary to convert the cluster to XML if we are always reading the data back within a LabVIEW application. We could instead just write the cluster directly without converting it and reading it straight back out later. Using an XML header, however, allows applications built in other languages to read and decode the XML header information.In high-channel count, high-speed data logging applications, engineers will many times archive data relative to some phenomenon or event that occurs within the system to help reduce the amount of data stored. While conventional RAM buffering can be used to record a few seconds of data surrounding these events, longer data records (on the order of minutes) are often necessary for accurately evaluating system performance when an adverse physical event occurs. The event driven circular buffer file technique discussed in this paper allows engineers to build systems that store a much larger window of data relative to asynchronous events compared to traditional RAM-based techniques.

Example

See Also:
Event-Driven Circular Hard Disk Data Buffering Example

Reader Comments | Submit a comment »