he gestural interface used by Tom Cruise in the movie Minority Report was based on work by MIT Media Lab’s Hiroshi Ishii, who has already commercialized similar large-scale gestural interface systems. However, such systems comprise many expensive cameras or require the user to wear tracking devices on their fingers. To develop a similar yet cost effective gestural interface system that is within reach of many more people other researchers at MIT have instead been working to develop screens with embedded optical sensors to track the movement of the user’s fingers that could quickly make touch screens seem outdated.Touch screens, like those found in iPhones, use capacitive sensing, where the touch of a finger disrupts the electrical connection between sensors which determine the location of the touch.
Gestural interfaces use embedded optical sensors to track the movement of the user’s fingers so they don’t have to come into contact with the display.Many have pegged gestural interfaces as the next big thing in computer interfaces, including Microsoft whose Project Natal uses a peripheral embedded with a small camera to capture gestural information.
However, according to the MIT researchers, such systems are limited because the cameras are offset from the center of the screen and therefore don’t perform well at short distances, which limits their ability to provide a seamless transition from gestural to touch screen interactions. This can be overcome if the cameras are set far enough behind the screen, as they are in Microsoft’s SecondLight, but this makes the displays bulky and requires expensive hardware to render the screen alternately transparent and opaque.“The goal with this is to be able to incorporate the gestural display into a thin LCD device” – like a mobile phone – “and to be able to do it without wearing gloves or anything like that,” says researcher Matthew Hirsch, a PhD candidate at the Media Lab says.Hirsch, along with MIT Media Lab professors Ramesh Raskar and Henry Holtzman and visiting researcher Douglas Lanman, have instead been working on a project that uses embedded sensors to turn displays into giant lensless cameras that can recognize hand gestures.
For their system to work, the Media Lab researchers used an LCD display with an array of optical sensors right behind it. The liquid crystals are like a lens, displaying a black-and-white pattern that alternates to let light through to the sensors. Because the pattern alternates so rapidly with whatever the LCD is otherwise displaying – be it a game, a list of files, or whatever – the viewer never notices it. Lanman explains that they could have used an array of liquid crystals to simulate a sheet of pinholes by having the central pixel in each 19-by-19 block being white (transparent) while the others are all black. The problem with this is that it would only allow a small amount of light to reach the sensors, and it would therefore require exposure times that are too long to be practical. So in order to maximize the amount of light that reaches the sensors the team developed a block of pixels 19 x 19 that is subdivided into a regular pattern of black and white rectangles of different sizes. This configuration allows much more light to pass through because there are just as many white squares as black.
The 19 x 19 blocks are all adjacent to each other and the pattern of black-and-white squares allows the system to computationally disentangle the overlapping images, capturing the same depth of information that a pinhole array would, but capturing it much more quickly.The Media Lab team has had to mock up its own LCD with built-in optical sensors to test its approach because these screens are not yet available commercially. In experiments in the lab, the researchers showed that they could manipulate on-screen objects using hand gestures and move seamlessly between gestural control and ordinary touch screen interactions.MIT news reports that Paul Debevec, director of the Graphics Laboratory at the University of Southern California’s Institute for Creative Technologies, whose doctoral thesis led to the innovative visual effects in the movie The Matrix says: “I like this one [gestural interface] because it’s really integrated into the display. Everyone needs to have a display anyway. And it is much better than just figuring out just where the fingertips are or a kind of motion-capture situation. It’s really a full three-dimensional image of the person’s hand that’s in front of the display.”
The name of the game today in technology is ‘interaction’. This is the touchscreen generation where interfacing with technology is a reality, not to the extent that one would see in a science fiction movie, although all of those concepts are ideologies that are already into play. Researchers and developers are working on models that will allow us, the end user, to be able to live in a science fiction like lifestyle where interacting with thetechnology that surrounds us could be conducted with a simple wave of the hand. We’ve seen simple devices like Wacom’s Pen Tablet design that’s been used by artists and designers alike to create works of art or shape our various environments. There have been tablet PCs that have allowed those on the go to interact directly with a computer without having to break their pace and then there’s the touch screen mobile phone that allows us to tap into the infinite vastness of the internet or do a simple calculation of 2 + 2 without the use of a physical key. Touch technology has come a long way from the tablet PC and the iPhone and that’s saying quite a lot considering the latter is still reigning high on any touch screen device list. Haptics, the system of touch feedback, gives touch technology a little more of a reaction in terms of user interactivity and now even that has taken on new meaning. With devices like the Pen de Touch, a prototype pen-shaped device that enables users to interact with virtual environments as though they were real. It will give users a sense of boundaries by providing feedback through the stylus in your hand if you exceed the border of a designated area. The day is soon approaching wherein we’ll be able to pull Tom Cruise’s, Minority Report style moves with your home computers making technology even more intimately interactive. We’ve already seen Microsoft’s Surface Interactive tables that incorporate even multi-touch functionality right on a table top. Consider it to be a rather large touchscreen mobile without coming in the same size of course. Motorola is working with researchers to develop Anywhere Touch technology that would allow users to create a touch screen interface on any surface they desire and even have it multi-touch enabled. They call it ‘tactilizing” any surface. The software technology leverages acoustics to analyze sound waves departing from the point of a touch to precisely transform any product into a touch device. The claim is that it’s cost effective as well as intuitive enough for it to be made affordable. A similar device is the virtual keypad that uses a micro projector to display a virtual keyboard on any flat surface allowing users to tap the surface like it were a standard keyboard for results to appear on the screen. The device connects to a PC or even a mobile handset via Bluetooth. Researchers down at MIT have showcased an even more advanced form of technology interactivity which is exactly what the movie Minority Report featured. In fact MIT’s Media Lab’s, Hiroshi Ishii, was the one actually responsible for the basis of the concept system shown in the same movie. Thetouchscreen generation is slowly moving into the world of gesture control. MIT researchers have devised a concept that uses optical sensors placed behind an ordinary LCD panel to track hand movements in front of the panel. These sensors then interact with the computer allowing users to manipulate data on the screen by simply moving their hands around. We’ll soon be able to slide, push, zoom into or out of data and pull it all, very literally out of thin air.
