Using cRIO and SBRIO in a Hardware Centric Approach to Improve Medical Device Reliability

This article examines the medical device life cycle and shares how NI Alliance partner HEI Incorporated worked to reduce design time, while meeting stringent requirements such as Food and Drug Administration (FDA) certification. Anyone with a mobile phone has experienced a dropped call. Although inconvenient, there generally are no catastrophic consequences. In contrast, consider devices whose failure can result in serious injury. For such devices, agencies have been put into place to oversee the development process and ensure integrity and safety of the final system. These agencies place regulations upon the development process and offer guidelines for best practices for all phases of development, from concept through manufacturing.

THE MEDICAL DEVICE DESIGN PROCESS

Figure 1. This diagram shows the medical device design life cycle as defined by the FDA.

Figure 1 shows the FDA definition of the medical device life cycle. The stated purpose of these guidelines is to ensure safety to the general public. Generally, these regulations must be applied to any component, part, or accessory of a medical device; any software used in the production of a device; and any software used in the implementation of the device manufacturer’s quality system; such as software that records and maintains the device history record.

HEI Inc, is a contract medical device design company that supports the entire life cycle of its customer’s product, from early business development support, through design and development, new product introduction and prototyping, initial production ramp, volume manufacturing and end of life manufacturing.  The medical device life cycle can be divided into three main phases. The first is the early product life cycle. This is the least regimented of the phases, and is primarily focused on the research and development of theories and ideas. This phase can last from a few weeks to many years depending on the complexity of the system being studied.

Figure 2. Graphical System Design has helped HEI develop functional prototypes of the end device in record time.

A fundamental part of the early development phase is data collection and analysis. Many tools are used by researches and product design specification teams to help streamline this process. The National Instruments(NI) LabVIEW product is an example tool HEI uses at this stage. In addition, using NI’s commercial off-the-shelf modular instruments gives HEI designers a big ‘sand-box’ of I/O connectivity to create and design next generation products without all the limitations of a fixed design or I/O signal set.

Once understanding the problem, HEI can design a solution. For device development and prototyping HEI can reuse the math and signal processing capabilities along with LabVIEW's intuitive graphical programming to develop new algorithms. Then, using commercial-off-the-shelf hardware, HEI verifies the algorithm’s performance against real-world data. In many cases HEI uses NI’s FPGA prototyping platforms as a solution for functional yet experimental prototypes of the final device. HEI uses the NI LabVIEW Real-Time and FPGA modules, along with the NI CompactRIO based upon Xilinx FPGAs to quickly iterate between the algorithm design and the device prototype stage. Using off-the-shelf hardware for prototyping reduces the time-consuming step of hardware development and integration, making it easier to focus upon delivering a bulletproof software design.

Figure 3. Software designs are typically validated and verified during the middle of the product life cycle.

The second phase of the medical device life cycle can be called the middle product life cycle (see Figure 3), which addresses the productization, verification, validation, and manufacturing of the designed device. The focus in this phase begins with well-defined specification documents that have clear and measurable requirements. Once these specifications are defined, it is time to develop a clear mapping between requirement documentation and actual implementation code.
In today’s complex medical device market, customers require solutions to get their new products to market faster. HEI uses a clear product development process that is based upon its use of quality tools to accelerate the traditional development process.

To understand what makes HEI’s development process effective, it is important to first understand how the traditional “waterfall” method of product development process works. The waterfall method looks like the following:

Figure 4. Waterfall development attempts to complete each stage before moving on to the next step of the process.

Waterfall development attempts to complete each stage before moving on to the next step of the process. As such, it is highly dependent on having complete and accurate specifications at the beginning of the project. The nature of medical device product development is that product requirements evolve with the development of the product. This requires a process that takes this evolution into account. The scrum/sprint development process at HEI is one answer to this problem.

 

Figure 5. In the scrum/sprint process, only high level system architecture and specifications are required to start the project. The project is divided into “sprints” which are 4-6 weeks in duration.

In the scrum/sprint process, only high level system architecture and specifications are required to start the project. The project is divided into “sprints” which are 4-6 weeks in duration. Within the sprint, all tasks are identified, and placed on a burn down list. Tasks are allocated to the team in a planning session at the beginning of each sprint. Team meets daily for a brief stand-up meeting, called a “scrum”. In the scrum, each team member answers the following three questions: “what did I complete yesterday”, “what will I complete today” and “what obstacles are in my way”. The project manager, or “scrum master”, manages the burndown list to track progress on a daily basis.  A schematic of the process looks like this:

Figure 6. In the scrum/sprint process, only high level system architecture and specifications are required to start the project. The project is divided into “sprints” which are 4-6 weeks in duration.

HEI’s use of the scrum/sprint development process has reduced development times by 30%, allowing new products to be implemented months ahead of time.   Of course, medical device product development is nothing if it is not compliant to FDA and other regulatory requirements.  The scrum/sprint product development process has been audited by the FDA and is compliant to all elements of the Quality System Regulation (QSR). The following table summarizes the two development approaches:

Table 1. Software designs are typically validated and verified during the middle of the product life cycle.

 

Figure 7. The late product life cycle takes the product to market, and market feedback helps determine the features for the next generation of the device. This completes the cycle and returns you to the concept phase.

The third and final phase of the life cycle is called the late product life cycle (see Figure 7). Very little engineering work is necessary in this phase, but customer feedback and market successes do help drive concept development of the next-generation product when the cycle starts anew.  Because HEI uses the scrum/sprint product development process, while leveraging off-the-shelf FPGA based hardware and high-level FPGA SW design tools that scale from research to manufacturing, they are able to quickly iterate on future product derivatives. In many cases, one product’s core architecture can be used in multiple product focused industries. For example, a medical pump can be used for IV flow and medication flow. The same general architected of pump controller can be used for transfusion control.

 

WHY HARDWARE TRUMPS SOFTWARE IN MEDICAL DEVICES

As many are aware, medical device recalls reached an all time high. The experts at the FDA have narrowed key causes toward two primary issues – defects in manufactured raw materials and poorly developed software. Raw materials can be more closely monitored, but the software issue is difficult to address.  As the number of lines of code in devices increase the problem will only worsen. Consumer safety officers at the FDA say the burden of safety shifts to the medical device designer.  HEI feels there is a potential solution to this problem, but it’s not in more testing, code reviews, and process. Instead, they try to write less software and push more of the logic into hardware elements like FPGAs. Here, we’ll look at some of the common causes of software failure and how HEI is addressing these with FPGAs.

 

Deadlock

Most modern devices need to be able to handle multiple tasks at the same time, yet most modern embedded processors are still limited to one processing core. This limits the processor to executing one instruction at a time, and parallel processes are made to share the main CPU. In addition, other shared resources like communication drivers, hard disk, and UI elements present opportunities for deadlock; the condition when two or more processes are each waiting for each other to release a resource.

Deadlock can be very difficult to reproduce and debug, since the situation often relies on multiple processes and usually requires a specific and synchronized sequence of events to occur.  Unit testing alone will not catch most deadlock issues; they are usually uncovered by code reviews, adept system testers, or luck.

With FPGAs, “processes” that are independent have their own physical circuitry on the FPGA, and therefore, there are no shared resources. On each clock tick, combinatorial logic latches in each circuit and values are stored in separate registers. No deadlocking can occur because neither process relies on the other’s resources. This allows one to put much more faith in the results of simulation and unit testing, since other unknowns like resource contention are no longer an issue.

 

Incompatible Middleware

When developing embedded software, one almost never implements every line of code from scratch. Instead, various tools are available to make the firmware designer more productive; these range from simple drivers to network stacks to operating systems and even code generation tools. Though these systems are generally well tested individually, no software is bug free. With so many possible combinations of tools and libraries, the likelihood of a designer of using components together in a novel way is relatively high.

For this reason, the FDA mandates that for all off-the-shelf software used in medical devices we must validate that the software stack works for our specific use case. What does that mean? Is says that if you are using a signal processing library that contains a fixed-point Fast Fourier Transform, and you are detecting the presence of a certain frequency component, you do not need to validate that the FFT returns the correct answer for all possible inputsR rather, you need to validate that it returns what is expected for all valid inputs according to specifications. For example, if HEI is detecting only human audible tones, there is no reason to test that the function returns correct values for inputs over 20 kHz.
Unfortunately, as HEI learned before, software components that seem independent are not necessarily so. Therefore, if HEI is using a software stack with an SPI driver within an RTOS, they need to validate all of these components together to really have confidence in the FFT. If the FFT passes a valid output to the SPI driver, but the SPI driver crashes, then obviously there is a problem. If HEI then decides to modify the SPI driver, they need to validate the entire software stack again. This can become very cumbersome, and one faulty component can delay the entire system development significantly. For this exact reason, HEI reuses as much known good and proven middleware RTOS drivers. Examples include the middleware drivers that are a part of NI’s Single Board RIO platforms which include Real-time OS based high-performance processors plus full featured FPGAs.

In the case of the FPGA, there is still the concept of external IP- commonly called IP cores, and HEI use of this IP needs to be validated just like software IP. However, once HEI has validated all of the use cases, they can have confidence that it will behave as expected when integrated with other components. Let’s look at the FFT example again. If HEI used an FPGA, they can acquire or generate an FFT IP core and validate its numerical correctness for each use case– this is the same as with the software. However, the risk of intermittent failure decreases drastically because the processor middleware has been removed. There is no longer an RTOS, and the SPI driver is its own IP core whose operation does not directly affect the FFT. Furthermore, if HEI modifies the SPI driver implementation, there is not the need to re-validate the unaffected areas of the system.

 

Buffer Overflow

Buffer overflow occurs when a program tries to store data past the end of the memory that is allocated for that storage, and it ends up overwriting some adjacent data that it shouldn’t. This can be a really nasty bug to diagnose, since the memory that was overwritten could be accessed at any time in the future, and it may or may not cause obvious errors. One of the more common buffer overflows in embedded design are a result of high speed communication of some sort, perhaps from a network, disk, or A/D converter. When communications are interrupted for too long buffers can overflow, so we need to account for this to avoid crashes.

This can be helped by an FPGA in two ways. In one example, the FPGA can be used to manage a circular or double buffered transfer, and it can offload that burden from a real-time processor. In this case, the FPGA serves as a co-processor that reduces the interrupt load on the main processor. This is a common configuration, especially among high-speed A/D converters. In a second example, the FPGA can be used as a safety layer of protection where all of the patient facing I/O is routed through the FPGA before it gets to the processor. In this case, one can add additional safety logic to the FPGA so that outputs can be placed in a known and safe state in the event of a software crash on the processor. The FPGA serves as a watchdog, and its logic ensures that the patient risk is lowered in the event of a software failure. With the architectural decision of placing an FPGA in the primary signal chain, these two methods can be combined to guard against buffer overflow and to better handle the situation if and when it does occur.

Both software and hardware are necessary in almost all electronic medical devices, and the balance between the reliability of hardware and the flexibility of software must be struck for every system HEI builds. However, when developing safety critical systems like medical devices, the complexity and flexibility delivered by software can have an adverse effect on the safety of the device.

Software will always be a part of electronic medical devices, and as devices become more sophisticated, the software will naturally become more complex. With FPGAs HEI can reduce the impact of this complexity by implementing more features in hardware, and therefore eliminating some of the most common errors in embedded software design.

 

IT IS ALL ABOUT TIME-TO-MARKET

It is for these reasons above at HEI, they complement the design process with tools that are well known in the medical design community. NI LabVIEW is a high-level programming tool used by researchers to efficiently define a solution to a problem. At HEI they apply LabVIEW to the design flow process. They then leverage off-the-shelf proven hardware building blocks such as the NI Single-Board RIO hardware which shares both a real-time processor and Xilinx FPGA.

Building on top of this hardware are NI’s underlying middleware drivers that have been proven in many applications. This allows HEI to focus upon moving as much SW out of the processor and into the FPGA. It also allows HEI to concentrate on the customization circuitry that is application specific. In turn, the more they move to the FPGA, the faster the design process is reduced in the future, as well as validation in the current product.

The decisions of designing a product for medical applications using a traditional custom design approach or a modular COTS approach is a challenge. Designers will always give you excuses to why it needs to be custom designed. In the end, income producing delivery time-to-market, and the ability to get customer feedback should always be #1. Most medical devices are Bill-of-Material cost conscious, when in fact they should be cost-to-deployment cost conscious. Validation, testing, and certification easily can double or triple a product’s true cost to customer. In the end, it all gets factored into the solution. Do the math, if you can produce a product and get it into the hands of customer’s faster, even though the BOM cost is higher, it takes a lot of product volume to find that cross over point. Many design teams also fail to see that since the product is in the market one are gathering data for the next, better product.

 

- Greg Crouch

Greg Crouch is the embedded systems business manager for the industrial and embedded product lines at National Instruments. Greg graduated cum laude with a bachelor of science in electrical engineering from Texas A&M University and has been a part of the National Instruments family for more than twenty years. Greg has served roles in R&D and sales and marketing, and is active in STEM program initiatives for K-12.

Reader Comments | Submit a comment »

I am researching the development of medical devices, but from a terminology point-of-view. As a terminologist who worked for software companies in the past, I am familiar with most software development concepts. I appreciated learning about the specifics in the development of medical devices. The article is well-structured and easy to understand.
- bikterminology@gmail.com - Jan 22, 2011