Creating Vision-Guided Motion Systems

Vision-guided motion control systems are useful in a variety of applications ranging from automated cell colony analysis to simple pick and place applications. Designing such a system can be challenging but will open up new opportunities to maximize your productivity. Some of the main challenges to overcome when creating such a system is in eliminating image distortion, correlating the motion units with the vision units, and ensuring that the coordinate systems for both the motion control and the machine vision are in line.

Overview of Creating Vision-Guided Motion Systems

For the past several years, motion control systems that use machine vision as a key component have been increasing in popularity. More and more engineers and scientists are realizing that combining current machine vision and motion control technology provides many advantages for solving difficult applications. Advances in the software and hardware technology have also helped fuel this trend by making combined motion control and machine vision systems easier to create and lower cost. When designing these types of systems, you should be aware of the current technological advances, methods, and tools designed to help you implement your system.

There are several things to consider when creating a vision-guided motion control system. One of the first things to consider is how your system will be set up. Consider a vision-guided motion system used to place a cover on a cell phone where the phone may not be in the same location and orientation each time. To make it easy, assume the cell phone is on an X-Y-Theta stage to correct for both position as well as orientation. Vision could be used to locate the cover and measure how far the motion would need to move to correctly position and orient it. There are several issues you would need to consider when creating this type of system such as how the vision is going to communicate with the motion to accurately move the part into position. Enabling communication between the motion and vision components requires a type of calibration. As shown in figure 1, there are several steps you should consider when calibrating a vision-guided motion system. First, you should overcome any distortions in the image system that could cause distances to be incorrectly translated to the motion control system. Next, you need to correlate distances measured in the image (usually in pixels) to distances measured on the stage or motor such as steps or counts. Finally, you need to correlate the vision coordinate system with the motion control system to correct for any misalignment between the two coordinate systems. This translation from distances in an image to distances in a motion control system depends on a variety of parameters including the distance the camera is from the object being moved and the lens type. Using a different zoom on the camera changes the size of the objects from the camera’s perspective and consequently any measurements made at the object will be different from what they were before moving the camera. Figure 1 shows a diagram of a simple vision and motion system.

Correcting for Common Forms of Distortion

When calibrating a vision-guided motion control system where the motion control moves in a 2D plane, you would ideally like to make sure your camera is perfectly perpendicular to the planar surface or stage that is moving relative to the camera. If the camera is not perpendicular, you will get perspective distortion and the part of the surface closest to the camera will appear larger than the part of the surface furthest from the camera even if the two parts are the same size. Some software packages, such as the NI Vision software, can correct for perspective distortion during the calibration process using a special calibration feature.

Lens distortion is another source of error when calibrating the system. Lens distortion comes from imperfections in the lens near the edges. This can cause straight lines to appear curved near the edge of the image. Like perspective distortion, lens distortion can also be accounted for during the calibration process using special features available with some image processing packages such as the NI Vision software.

Correlating Image Units to Motion Control Units

Calibrating a vision guided motion control system can be done in a variety of ways. One of the easiest methods is to calibrate the system experimentally using data you gather from the machine vision and motion control components. Using this method, you move the motion control system to various spots and record the translation distances using the machine vision system. With the data you gather, you can determine an equation to translate pixels units from the camera into units the stage can use such as counts on an encoder. Going back to the cell phone pick and place example, assume you move your stage 1 cm in the X direction and using your microscope you observe that the stage has moved 100 pixels in the X direction. You can then create a calibration constant that would be .01 cm/pixel telling you that each pixel you measure is equal to .01 cm on the stage.
After eliminating or accounting for the effects of distortion in your images, another precaution you should consider when designing a system is to ensure that the coordinate system of the camera is lined up with the coordinate system of the motion control component. Depending on the application and the amount of accuracy you would like to achieve, this may be critical to achieving maximum performance. A camera that is not lined up perfectly with the motion control coordinate system will cause errors in the positions it commands. For example, if you use a stage and the camera is not lined up with the stage you could move the stage just in the X direction and the camera would record movement of the stage both the X and the Y direction. Coordinate system misalignment can be corrected using coordinate transformations. Coordinate transformations can take the coordinates from one coordinate system such as that of the stage and change them into another coordinate system such as that of the camera. Robotics applications require these coordinate transformations to determine what the joint values need to be to get the end effector to a desired position. In the image below, the black coordinate system is for the camera while the green coordinate system, rotated at some angle, is for the stage.

One of the first things you need to determine is how large the angle is between the two coordinate systems. We will call this angle the offset angle. You determine the angle by simply moving the stage in a known vector, preferably something fairly simple and then measuring the beginning and ending position using the machine vision system. The difference between the commanded move on the stage and the measured move using the image from the camera will help you determine what the angle of rotation is. For example, assume you move the stage 2000 units on the X axis only (0,0 to position 2000,0). Because you have only moved in the X axis, any movement on the Y axis in the image would help you determine the offset angle. If the image from the camera shows that the stage moved 173 pixels in the X axis and 100 pixels in the Y axis, you can calculate the angle of rotation by taking the inverse tangent of the Y value over the X value. You can then calculate the offset angle by taking the inverse tangent of 100/173 which is equal to 30 degrees.

After determining the offset angle, you next need to determine the conversion between vision units and motion control units. You do this by comparing the length of the move as measured in the image to the length of the move as measured on the motion control. To determine the length of the move in the image, use the hypotenuse from the triangle created when putting the X vector with the Y vector. Thus, using the Pythagorean theorem, you find that the length, Z, is Z = sqrt(X^2+Y^2).

Length of move measured in image = sqrt(100^2+173^2) = 200 pixels

Now you know that for the example above, the ratio of vision units to motion control units is 200:2000 or 1:10. One caveat you should understand when calibrating is that the more samples you take when calibrating, the more accurate your calibration will be in the end. Another caveat to be aware of is that the motion control system may have slight imperfections which may cause it to move slightly different in different parts over the length of travel. To increase repeatability, you should find the conversion ratio by taking multiple samples from different points. Also, using this ratio, you may notice that in order to observe one pixel of movement in the image, you would have to move the stage at least 10 units. It is common to find that in these types of systems, the resolution of the stage is often much greater than the resolution of the camera meaning that small changes in the image may lead to large changes in the motion. If you are constantly checking the stage position to see where it is at, you may find that it will jump around the target position. If this is the case, you may want to also consider implementing a deadband area such that when the stage gets to a position that you feel is close enough to the target position, the motion control no longer continues to make adjustments.

Coordinate Transformations

The next step we need to do is use the information about the systems to determine how to translate the location of a point from the camera’s coordinate system to the stage’s coordinate system. To better illustrate how to figure out the transformation, assume want to move the stage from position 0,0 to a new location X1 and Y1 (represented in blue) in the camera coordinate system where X1 and Y1 are in the positive quadrant as shown in the image below. For a coordinate transformation, the information you need to know is the X and Y coordinate of the point in the stage coordinate system represented by the dotted red lines in the image. The information you already know is the X and Y coordinates in the camera coordinate system and using these you can easily find the angle of offset in the camera coordinate system. Finding the angle of the vector in the stage coordinate system is as simple as subtracting the known angle of offset from the angle of the vector in the camera coordinate system. You can form a right triangle with the Vector, H, as the hypotenuse and the unknown Y2 and X2 values as the two sides. With a right triangle, you can find the length of two of the sides if you know the length of the other side and one angle. Because we found the size of the angle in the stage coordinate system, we can use the law of sines and the law of cosines to find the other two sides. This comes out to be:
sin(theta3)=Y2/H becomes H*sin(angle in stage frame)= Y2
cos(theta3)=X2/H becomes H*cos(angle in stage frame)= X2


Using this simple set of equations, you can determine the coordinate transformation from known coordinates in an image to unknown coordinates for the stage. Some of the caveats you must understand when implementing this method are that the when the vector falls in other quadrants the equations will be different. However, determining the equations for the other quadrants is very similar to the method shown above.

For certain applications, you can perform motion-vision calibration using closed loop control. The idea behind this approach is to use the difference between the ideal location and observed location of a feature in the image as the error signal that drives a control loop. For, example, the feature could be the location of a notch on a part you are positioning where you would like to align the notch to some ideal location in the image. As the control loop adjusts the part or the camera in successive iterations, the error gets smaller until there is no difference between the ideal location and the observed location of the feature. Using this method, you can easily align the camera and stage coordinate systems. You can take this approach a step further by using estimation algorithms that enable you to control the components while taking into account the imperfections in the system, such as lighting variations and slow frame rates of image acquisition.

Conclusion
Vision-guided motion control systems are useful in a variety of applications ranging from automated cell colony analysis to simple pick and place applications. Designing such a system can be challenging but will open up new opportunities to maximize your productivity. Calibration is of coarse only part of creating a total system but it is a very important part. Overcoming distortion in the vision system, correlating vision units to motion units, and doing coordinate transformations are important things to consider to improve accuracy, repeatability, and overall value in your vision-guided motion system.

Reader Comments | Submit a comment »

some comments...
Hi, I've just read this article, because I'm about to make a very similar application. I was looking for information about coordinate system conversion. What I find strange is that in your graphs, the center of the 2 coordinate systems is the same : (0,0)vision and (0,0)motion. There is always an offset, isn't there? To correlate the 2 coordinate systems, I have a proposal : If you move with the robot to two points and you find the locations of those two points in the vision coordinate system, then you could split up the conversion to a gain and offset for X and a gain and offset for Y. Then you can quickly convert to both systems, and the conversion from pixels to mm is allready included. What do you think?
- Thomas Vandenberghe, Thomas Vandenberghe. thomas.vandenberghe@ninix-tech.com - Dec 7, 2005

Great but. . .
This is a great article, but where are the VIs? A zip file with VIs that have the calculations included would be a great addition to this article.
- Robert Young, DISTek Integration, Inc.. boby@distek.com - Jun 16, 2005

My Compliments, and Comment
Hi, folks! Regarding your "Creating Vision-Guided Motion Systems" article: Good Job! I found it to be very informative (as are most NI articles. I often file them away for future reference because they contain a lot of good information I might not find elsewhere). I happened to be investigating rotational transformations recently, so your article was perfect timing. I also came across another web page that you might find relevant: http://www.akiti.ca/RotateTrans.html This page includes two rotational transformations, definitions of the x-y-z transformations, and it also provides numerical results, so you can see a specific example worked out. Anyhow, I thought you might be interested. Regards, Duncan
- Duncan McLachlan, Confidential. duncan.mclachlan@mail.com - May 25, 2004