A recent announcement from a company that specializes in sensors used for gesture detection sparked thoughts about using this type of control in mechatronic devices. The company mentioned using gesture controls in TV sets and set-top boxes. That type of control works for interactive games, too.
It got me thinking: Why not extend gesture controls to mechatronic devices? A Google search located many academic papers about this type of control, and some ambitious hackers have created gesture controllers for robots that use a Wii game interface or accelerometers. My thoughts tend more to real-world applications, such as teaching robots to mimic human operations (like performing tasks for disabled people based on hand, head, or eye motions) and safety applications that could shut down equipment.
iPoint 3D components from the Fraunhofer Heinrich Hertz Institute (HHI) let people use their fingers to manipulate 3D images.
Mechatronics engineers already have programming languages such as ROBOTC (based on C) and RAIL (based on Pascal) that control robot actions and sensors. But these languages use the same fundamental line-by-line code I learned in the mid-1960s! The mechatronics capabilities of equipment and robots have expanded, but programmers still control them with old-fashioned languages. National Instruments' LabVIEW software provides a higher-level graphical programming approach that better abstracts engineers from languages. So we have taken a step in the right direction.
Most mechatronics engineers would rather tackle new tasks than go through the same coding process to get actuators to move and motors to run. Now vision systems can capture human motions in three dimensions. Some sort of translator could convert this 3D information into similar robotic motions. Likewise, people who create mechatronic protheses could use captured human motions to configure the response of actuators based on sensor inputs in an artificial limb. This sort of thing might seem like science fiction, but I wager we'll see gesture controls in more and more products.
In the area of safety, for example, many machines require operators to place each hand on a control switch before the controller starts any action. Instead of having operators move their hands to special switches, why not simply let them hold up their hands or fingers in front of a gesture sensor? This type of control could improve productivity, reduce the effects of repetitive motions, and improve safety.
What else might clever engineers learn to control using gestures? Discuss in the comment section below.
Jon, your idea is interesting, but I think it needs a lot more investigation before being implemented. As for programming the robots, even when you have these high level languages, as in LabView, the code you write is internally stored in a form more like the "older" styles of code you mention. In the end, these things have to be translated into something a computer can understand. That means a formal language that can be compiled, optimized and translated into machine code.In the end, these codes also have to be precise and provable. I see a lot of higher level languages used, but still the lower level ones are needed to do things the developers of the high level codes did not anticipate.
I agree naperlou - while higher level languages are convenient and typically much easier to use (loved CEC's Testpoint back in the day), nothing beats the control you can get at the lower levels. This coming from someone who still uses assembly when programming PICs- love moving those bits around!
Nancy Golden, Using assembly language with gesture controls may improve response time because of the software being one level above machine code. Therefore, bit processing is closer to the target microcontroller than a high level language like "C" code. Just thinking out loud folks!
Hi, mrdon. I bet bit-bashing code in assembly language provides only a small time advantage over code written in C and compiled. Current compilers provide optimizations and can get very "close" to an MCU's hardware for control. I would much rather read control code in C than try to figure out bit configurations and settings in another programmer's assembly-language code. (Been there, done that!)
OK, mrdon. Some MCU manufacturers offer wizards to help programmers configure the bits and bytes through a graphical interface. The Texas Instruments "GRACE" software, for example, provides this capability for some of the MSP430 processors. A programmer selects the necessary peripheral devices and then selects the desired settings. GRACE creates C-language source code to do the job.
I think the idea of extending vision-based gesture recognition to industrial robots makes a lot of sense. We've written about use of the Kinect vision sensor, a major new input device for vision-based gesture recognition, applied to robotics, as well as this robotic gesture-recognition software based on a 3D bend-and-twist fiber optics sensor used in the film industry for motion capture: http://www.designnews.com/author.asp?section_id=1386&doc_id=245683 Interestingly, those researchers said their next rev would be Kinect-based. A different approach we wrote about would help industrial robots predict humans' next moves in assembly based on a decision-tree algorithm: http://www.designnews.com/author.asp?section_id=1386&doc_id=246646 A member of the Embedded Vision Alliance (whose representatives lectured at our recent Digi-Key CEC on embedded vision) has also written this article on vision-based gesture recognition focusing on Kinect, which also discusses software: http://www.digikey.com/us/en/techzone/microcontroller/resources/articles/vision-based-gesture-recognition.html
The Microsoft Kinect's range of uses that have been reported represent just the tip of the proverbial iceburg. I'm certain we will see it being used for many more applications, including those within the areas of robotics.
Respectfully, you may have a misconception about LabVIEW that is (unfortunately) somewhat common. Many people assume that LabVIEW provides an interpreted programming approach because the code is largely represented graphically instead of through text.
The truth is that "G" (i.e. graphical code) in LabVIEW has been a compiled language for several decades now. At the end of code compilation, LabVIEW code is converted into machine code in the same manner as other programming languages. LabVIEW now uses a very powerful, open-source compiler known as LLVM that is also used by industry leaders like Apple, Adobe and Sun Microsystems.
In addition to being able to compile to machine code, LabVIEW code can also be compiled to run on silicon in the form of FPGAs. The reliablity and performance achieved through FPGAs have made popular as part of the control systems in many robotics and mechatronics systems.
You can read more about the LabVIEW compiler here:
Hi, Naperlou, et al. LabVIEW creates compiled code. It does not create some sort of intermediate code that requires an interpreter. Instead, you get native code that runs on your target processor. And as far as I know, you can mix in C-language code, too, if you need to do something at that level. Also, LabVIEW will compile and run applications in FPGAs.
Gesture controls for video games and entertainment systems are fine; if a gesture is incorrectly there is nothing worse happens than needing to do a reset. Gesture controls on something as fast and powerful as an industrial robot could easily knock ones"block" off, quite literally. Equipment and machines that are capable of being unsafe are probably not a good choice for control inputs that are subject to interpretation. That is something that needs to be kept in mind when choosing an input system, but may not have come to mind when considering a departure from the more standard methods.
William K, I think gesture controls could be used in industrial applications by providing a simple movement/motion protocol. The protocol could have a similar format like sign language but less complicated in physical movement. For example, to stop an industrial process one would gesture a hand pressing an e-Stop (emergency) pushbutton. Also, for critical industrial processes, redundant switches/sensors can be packaged on the machine as well.
Hi, William. People would train robots to respond only to specific gestures. Thus, someone couldn't use an industrial robot to follow his or her every move and start tearing apart an assembly line. Gesture control would require security to prevent unauthorized use or "training."
I can see that an industrial robot that would run the program based on a gesture for a start command could be safer, and possibly even better. BUT as an engineer responsible for safe operations I would NEVER consider leaving out the big red EMERGENCY STOP button. Adding a gesture input in addition would be OK, but to put something as important as an emergency stop function on the wrong end of a lot of software would be very irresponsible. JUst remember those cars with thatbSTUPID start/Stop button that does not work in an emergency situation. There is a very valid reason that E-Stop is not done in software.
The Machinist Calc Pro computes speeds and feed rates for milling, turning, and drilling: cutting speed, spindle speed, feed rate (inches/minute), cutting feed, etc.
During a recent meeting with engineering-school faculty and alumni, Contributing Technical Editor Jon Titus talked about whether colleges should educate generalists or specialists. What do you think?
From Dell / Intel® New Paradigms in Design Work Scott Hamilton, vertical market strategist for Dell Precision workstations, 5/2/2013 5
Early in my career, I worked as a draftsman and remember the days of drawing on vellum with numbered pencils and Mylar with plastic lead. This was a fun experience in the sense that I ...
I've been using workstations for more than 10 years and love finding ways to get more performance from my system. With demanding professional applications that require more power each ...
A lasting memory from my first job as an engineer in an auto assembly plant is standing on hard concrete at six in the morning, vending-machine coffee clutched in hand, listening to ...
A quick look into the merger of two powerhouse 3D printing OEMs and the new leader in rapid prototyping solutions, Stratasys. The industrial revolution is now led by 3D printing and engineers are given the opportunity to fully maximize their design capabilities, reduce their time-to-market and functionally test prototypes cheaper, faster and easier. Bruce Bradshaw, Director of Marketing in North America, will explore the large product offering and variety of materials that will help CAD designers articulate their product design with actual, physical prototypes. This broadcast will dive deep into technical information including application specific stories from real world customers and their experiences with 3D printing. 3D Printing is
To save this item to your list of favorite Design News content so you can find it later in your Profile page, click the "Save It" button next to the item.
If you found this interesting or useful, please use the links to the services below to share it with other readers. You will need a free account with each service to share an item via that service.
Tweet This
2 
