Streaming Images to Disk with NI Vision

Types of Acquisitions

Machine vision and scientific imaging applications typically employ one of several common acquisition architectures. Acquisitions can be either continuous or finite in duration. Continuous acquisitions are used in applications such as monitoring, inspection, sorting, and process control. Image sizes and frame rate vary widely from one application to another, however there is one defining attribute of a continuous application. These systems must be able to process each image, on average, in less than one frame period of the acquisition. Multiple image buffers are used to allow for some fluctuation in the processing time, but no amount of memory will suffice if the processing cannot keep up. Finite acquisitions are often used for event recording, where images are acquired for a predetermined amount of time (or frames) and processing can occur off-line. The definition of finite can range from second to minutes or even days and is determined by the needs of the system. For acquisitions that can fit entirely within memory, there is rarely a concern about speed. Modern PCs can read and write data to RAM much faster than even the fastest image acquisition cards, such as the NI PCIe-1429. However, acquisitions stored in RAM are limited by the amount of system memory. Most systems running a 32-bit operating system are limited to a total of 4GB of RAM. In practice, less than 3GB will be available to the user, and this will be shared with other applications. Recording duration can be extended by writing some or all of the image data to disk. NI Vision provides efficient and easy to use tools to write acquired images to disk.

Manage the Data Pipe

Designing an effective image recording system requires a basic knowledge PC architecture and an understanding of the path an image will follow from camera to disk. Various components in a computer will affect the overall performance of the system and it is important to predict and eliminate bottlenecks that can restrict data flow. Figure 1. shows a simplified view of the path image data follows from the camera into system memory. High speed vision systems require a camera connection with ample bandwidth. A Camera Link interface provides up to 680 MB/s throughput (full configuration), simple cabling, compatibility with a wide variety of cameras, and is a clear choice for many high speed applications.

After acquiring an image from the camera, an image acquisition device will usually send the image data to system memory. The device's connection to the memory controller is the next potential bottleneck to consider. Conventional framegrabbers based on 32/33 PCI technology are limited to about 100 MB/s. Many cameras exceed this data rate and the PCI bus quickly becomes the limitation in high speed application. PCI-X and PCI Express provide higher throughput alternatives. PCI-X is an established connection technology, typically found in high end workstations and servers and supports a theoretical throughput up to 1GB/s. Like PCI however, this bandwidth is shared with other devices on the bus. PCI Express is a rapidly emerging technology, which has the potential to scale to even higher data rates than PCI-X. PCI Express guarantees that bandwidth for one slot is not shared with others. The NI PCIe-1429 uses a x4 PCIe link (1GB/s) to acquire up to the maximum Camera Link bandwidth of 680MB/s.

The data path from the memory controller to RAM is usually does not pose a restriction. Modern DDR RAM is more than fast enough for high speed cards like the NI PCIe-1429 and RAM technology is steadily advancing.



Once images are acquired into system memory, the data can be written to disk. Figure 2. shows the path of data from system memory to a typical SATA disk array. Just like other peripheral devices, disk controllers are available with a variety of interfaces and the same rules apply as with the image acquisition device. A 32/33 PCI interface is simply not sufficient for high speed streaming applications. PCI-X disk controllers are widely available and offer up to 1GB/s throughput to the controller. PCI Express disk controllers have recently become available with x8 PCIe connections, which in theory support up to 2GB/s. Transferring data from the disk controller to the hard drive is a two step process. The data is first sent to the hard drive cache, physically located on the drive, and from there is written by a magnetic head to the rotating platters. The speed of the first transfer to cache is governed by the connection technology. SATA1 and SATA2 connections support rates up to 150 MB/s and 300 MB/s respectively. It's important to note that this bandwidth is dedicated to each drive and not shared. Writing data from the hard drive cache to the physical media is usually a slower process. Hard drive technology is constantly advancing, however currently high end SATA drives can sustain writing rates of 60 to 70 MB/s (outer tracks). Taking this into account the SATA link (SATA1 or 2) does not limit data transfer. Compared the interface from RAM to the controller, these data rates appear slow. Consider that one disk controller may connect to 8, 16, or even 24 drives, and the usefulness of a high speed PCI Express connection becomes apparent.


Figure 3. shows the path of data from system memory to a typical SCSI disk array. All components are similar to the SATA scenario, except the link between the controller and the hard drives. The fastest commonly available SCSI interface, Ultra320, provides a theoretical throughput of 320MB/s. This bandwidth is shared with all devices on the bus (up to 16 including the controller) and not dedicated to each device. The second step of writing data to the physical media is often even faster with SCSI drives than SATA drives. Modern high speed SCSI drives can sustain writing rates of 90+MB/s. SCSI drives typically come with enterprise class reliability and warranties, with a correspondingly higher price.

At first glance, the higher per drive writing rate for SCSI systems appears to be a clear advantage, however this speed is can only be utilized with a few drives on each shared Ultra320 channel. Multiple channel SCSI controllers can spread drives across each channel, such that a four channel controller could theoretically provide 1280 MB/s throughput with an adequate complement of drives. Furthermore, multiple controllers can be operated in parallel to achieve even higher rates. This applies to SATA controllers too.

RAID Simplified

A vision system with multiple hard drives needs some means to harness the collective throughput of the drives with one simple interface. Writing different images to each drive manually in a round robin fashion (D:\image01.bmp, E:\image02.bmp, F:\image03.bmp, etc) would work, but this is unnecessarily cumbersome. RAID allows the system to treat multiple physical drives as one logical drive. The user can write all data to one logical drive, and the disk controller apportions out the data to each drive.

RAID stands for Redundant Array of Independent Disks (or Inexpensive Disks, depending on who you ask). RAID systems are described by their topology (or level) which is denoted by a number (0, 1, 5, 10, etc). RAID 0 systems split up user data evenly and stripe it across each drive in the array. In theory the total throughput will be the sum of the writing rate of each drive. All other RAID levels trade some performance for data redundancy and are less interesting for high speed streaming applications. Use RAID 0 to achieve the best throughput.

Test Setup

The following tests evaluate the performance of a modest SATA drive array. Benchmark were performed on 2 separate machines with the hard drives and controller being the only common component. Tests were run with one drive enabled as a single drive and with all four drives configured as a RAID 0 array. In all cases, the disks were dedicated to image writing and a separate system drive was used for the OS and applications. Substantially similar results were recorded on both systems, indicating that the disk controller and/or hard drives were the limiting factor. All tests were started with the disk(s) empty and run until 40% of the total capacity was written. All images were written to uncompressed AVI format: 8 bit images as 1 byte per pixel and RGB images as 4 bytes per pixel.

System 1: SuperMicro X6DAE-G2 (Intel 7525), 2 x Xeon 2.4 GHz 1GB DDR333 RAM NI PCIe-1429 Broadcom BC4852 disk controller 4 x Western Digital WD360 disk drives Windows XP Pro

System 2: Tyan K8WE (nForce Pro 2200), 2 x Opteron 2.0 GHz 1GB DDR400 RAM NI PCIe-1429 Broadcom BC4852 disk controller 4 x Western Digital WD360 disk drives Windows XP Pro

Results

The following tables and graph show the best rate that was achieved reliably for each image size, image type, and hard drive configuration. Table 1 shows the results achieved with one hard drive enabled. In some runs, the drive sustained 70 MB/s for a short period of time, however the results in Figure 4 represent the highest rate writing 40% of the disk capacity without dropping a frame.



Table 2 shows the writing rates achieved with all four hard drives in a RAID 0 configuration. In most cases a 3x improvement was seen compared to the single disk tests. At very small image sizes, other limiting factors affect writing performance. The maximum frame rate of 2500 fps (100 x 100 x 8 bit) did not improve from the single drive case. In some runs, the drives sustained 210 MB/s for a short period of time, however the results in Figure 4 represent the highest rate writing 40% of the disk capacity without dropping a frame.





Figure 4 shows a graphical representation of frame rate vs. image size and shows two interesting features. For smaller image sizes, the writing rate appears to be limited the overhead of each file IO operation. At larger image sizes, the performance followed a constant data rate trend, as demonstrated by a straight line on the log-log plot. The delineation between small and large images depends on the image type and drive configuration.
These tests demonstrated how NI Vision tools can stream image data to disk at 180 MB/s. These results were achieved with standard, off-the-shelf technology at a relatively low cost. The cost of four drives and the disk controller was less than $900. More importantly, this solution is highly scalable. The disk controller used in these tests will accommodate 8 drives and up to 4 controllers can be combined in parallel to create an array of 32 drives. Published benchmarks show such a system streaming data to disk at more than 1GB/s. It is not inconceivable that NI Vision, combined with the NI PCIe-1429, could stream images to disk continuously at the maximum Camera Link rate of 680 MB/s.

This data should serve as a guideline for developing a system to meet your requirements. The following tips will help get the best performance from a system.

• Dedicate a drive to be used only for temporary image writing. Use a separate drive for the OS and any applications.
• Start each run with a empty drive if possible. Hard drives achieve the fastest writing rates when empty and the performance diminishes as the drive platters fill from the outside inward. Many drives show 2 to 1 reduction in throughput from empty to full. When specifying a system, locate a data transfer diagram for your drives and use this to estimate the writing rate at the capacity you intend to use.
• Disable the system page file for the drive.
• Disable virus scanning for the drive you are writing to. Virus scanners can have tremendously detrimental effect on file IO. In some of the four drive tests, the maximum writing rate was reduced from 180 MB/s to 30 MB/s with virus scanning enabled.

Reader Comments | Submit a comment »

great article - the code sample would be nice
Excellent document. A code sample would be nice. Should be trivial, but that is the point. Thanks, Bill
- Bill Kuhn, William P. Kuhn, Ph.D., LLC. bill.kuhn@wpkuhn.com - Feb 9, 2006