Avoiding Shared Resources and Priority Inversions for Deterministic Applications

The first thing you should consider when implementing a system with LabVIEW Real-Time is whether you need determinism. The embedded qualities of LabVIEW Real-Time, including reliability, ability to off-load processing, and headless black box design, are attractive for many different types of applications, even those that do not require determinism. Two common types of applications which require determinism are real-time control and real-time event response, both of which have specified jitter tolerance. If your application only needs the embedded characteristics that LabVIEW Real-Time affords, no special considerations are necessary when programming. On the other hand, if your application has real-time needs, you should employ certain programming techniques when trying to achieve high levels of determinism. The purpose of this document is to define a priority inversion and distinguish between two major types of priority inversions and how they can affect a real-time application.

In LabVIEW Real-Time, there are many resources that two or more threads may need to share. These shared resources include global variables, non-reentrant subVIs, the LabVIEW Memory Manager, queues, semaphores, single-threaded DLLs, etc. When a lower priority thread needs a shared resource such as a global variable, it protects the resource from external access while in use. The lower priority thread acquires a mutex and accesses the global. If a time-critical priority thread wakes up, it kicks the lower priority thread off of the processor even while the lower priority thread is using the resource. However, if the time-critical thread requests to use the same protected resource (the global), it is forced to wait because of the protection or mutex around the shared resource. This conflict is an inversion of priorities or “priority inversion”; the lower priority thread has suddenly become more important than the time-critical thread, because it must finish its work and relinquish the shared resource before the time-critical thread can proceed.

If a higher priority thread is forced to wait on a lower priority thread because of a protected resource, the RTOS under LabVIEW Real-Time uses a method called priority inheritance to resolve the priority inversion as quickly as possible. Priority inheritance allows the lower priority thread to temporarily “inherit” the time-critical priority setting, long enough to finish using the shared resource and to remove the protection. Once the protection is removed, the lower priority thread resumes its original lower priority setting and is taken off of the processor. Now the time-critical priority thread is free to proceed and use the resource (i.e. access the global).



Priority inversions, which are generally caused by shared resources, induce jitter. In general, you should avoid or at least minimize jitter because it reduces the level of determinism in your application. Thus do not try to use shared resources in your time-critical thread. The amount of jitter induced by a shared resource depends on the type of shared resource involved. For instance, when accessing a global variable, a thread can finish a read/write operation on a global within a consistent length of time or with very little variance in time. In other words, reading/writing to a global variable is bound in time, and consequently, the jitter induced by sharing global variables between threads is bound in time. On the other hand, when a thread allocates memory, it solicits the LabVIEW Memory Manager (another type of shared resource). Depending on the size of memory requested, the LabVIEW Memory Manager may be locked or “mutexed” for a few microseconds up to several seconds. As was the case with global variables, a low priority thread can acquire a mutex on the LabVIEW Memory Manager and cause a priority inversion if a higher priority preempts the low priority thread and tries to allocate memory. Unlike global variables, the LabVIEW Memory Manager induces jitter that is very broad in magnitude because parallel operations could be trying to allocate blocks of memory in a wide variety of sizes. The larger the block of memory allocated, the longer the priority inheritance takes to resolve the priority inversion. Thus jitter induced by mutexing the LabVIEW Memory Manager is unbounded in time and can be much more detrimental to a real-time application than a time bound priority inversion involving a global variable. Avoid allocating memory within time critical processes. Instead, preallocate all of the arrays before performing time critical operations. Refer to Preallocating Arrays for Deterministic Loops, document linked below, for more information.

We mentioned that single threaded DLL's are also shared resources and can also lead to priority inversion problems. One of these DLL's requires special attention - the NI-DAQ driver DLL. Since the DAQ driver is a shared resource, only perform DAQ operations serially (not in parallel). Most importantly, limit calls to NI-DAQ to the time critical thread only. This prevents lower priority VIs from causing priority VIs from causing priority inversions by using the DAQ driver when the time critical thread needs it.

One of the most important things you can do to avoid high jitter in time critical processes is to remove all non-deterministic operations from your time-critical thread. For instance, if you need to log data to the hard drive or communicate over a network, move those tasks into a separate, lower priority VI. To communicate data between the two VI, use an interthread communication method that does not cause jitter, such as using the Real-Time Queue (LabVIEW Real-Time 5.1.2) or Real-Time FIFO (LabVIEW Real-Time 6i or later). Refer to the NI Developer Zone documents linked below for more information.

As we have seen, programming a deterministic real-time application requires that you avoid priority inversions, which induce jitter. Consequently, you must pay close attention to common shared resources, including global variables and the LabVIEW Memory Manager, and minimize their presence in time-critical code. Finally, we have identified the difference between a time bound priority inversion (mutexing a global variable) and an unbounded priority inversion (mutexing the LabVIEW Memory Manager). While both should be avoided in a time-critical thread, we learned that unbounded priority inversions pose a larger threat to determinism than time bound priority inversions.

Related Links:
Preallocating Arrays for Deterministic Loops
Real-Time FIFO Example
Real-Time FIFO for Deterministic Data Transfer Between VIs.
Understanding priorities in LabVIEW Real-Time applications

Reader Comments | Submit a comment »