of the Massachusetts institute of technology, Microsoft and Adobe’s researchers have developed a kind of algorithm, this algorithm can be through the analysis of the vibration of the video object reconstruction audio information. In a set of experiments, they can through the sound insulation glass, in 15 feet away from the vibration of the potato chip bags decipher the understandable language.
in other experiments, they from a film aluminum foil, the surface of the glass of water, or even a potted plant leaf extract the useful video audio signal. Researchers will learn meeting in this year’s super computer graphics Siggraph presented their findings.
one of the electrical engineering and computer science at the Massachusetts institute of technology graduates Abe Davis (first author) said: “when an object hit voice, can cause the vibration of the object. The vibration produced a very subtle visual signals to the naked eye cannot observe. People haven’t realized the existence of information.”
on the Siggraph papers signed and Fredo Durand and Bill Freeman – both are at the Massachusetts institute of technology, a professor of computer science and engineering; Neal Wadhwa, graduates of the Freeman group; Dr Michael Rubinstein — from Microsoft researcher, with graduated from Freeman; And Gautham Mysore, Adobe’s researcher.
reconstruction audio from video to video frequency of sample number of video frames per second (capture) than audio frequency. In some of their experiments, the researchers used a second high-speed cameras to capture video frames between 2000 and 6000. This is far higher than that of some smart phones 60 frames per second frequency, but far less than some commercial high-speed camera up to 100000 frames per second.
however, in other experiments, they are using the common digital camera. Since most cameras sensors on the design of the existence of coincidence, even if the video is record standard 60 frames per second, researchers can still detect the high frequency vibration information. Although audio reconstruction and not as accurate as high-speed camera, but it still is a good way to determine the room in the speaker’s gender, number, and even can be gained according to the acoustic characteristics of the speaker voice enough accurate information – the speaker’s identity.
researchers technology can be well used in the law enforcement of evidence, but Davis was quite excited, and call it “new imaging”. He said: “we are read from the object in the sound, this gives us a lot of information around an object’s voice, but also gives us a lot of information of the object itself. Because of the way different acoustic reflections.” In the ongoing study, the researchers have been trying to from the object itself in a short period of time for sound reflection situation to identify the structure and material of the object.
the Siggraph papers mentioned in the experiment, the researchers also measured the mechanical properties of the objects in motion, to determine the measured their motion range is about one over ten of a micron. Although this is only equivalent to five over one thousand of the close-up images of a pixel, but from a single pixel color values change over time, the range of movement is likely to be less than one pixel.
, for example, we assume that an image of the two regions have clear boundary: on one side of the border are all blue, but on the other side is all red. But at the border, camera sensor can also receive the red light and blue light, so that blend to produce a purple light. If the removal of consecutive video frames, even less than the width of a pixel, blue light area is still in the red light area, which leads to the purple light light blue. The color of the migration contains invasion degree of information.
however, some boundary is better than a single pixel in the image is blurred. So the researchers borrowed a early work used in the algorithm, this algorithm enlarge the tiny changes of the video, and made no movement is detected earlier revealed: hospital neonatal wards the baby’s breathing or wrist pulse beating.
the technology through the section battery image filter, can be passed to measure volatility changes in consecutive video frames, such as the border along the horizontal, vertical, diagonal, and even a few different angles the different direction of changing color values.
researchers have developed an algorithm, this algorithm can be sound waves hit the object, combining the output of the filter to deduce the movements of objects as a whole.
the different edge of an object could move in different directions, so the algorithm first will record all measurements, so they don’t cancel each other out. At the same time, it also favor the clear boundary between different color value measurement.
the researchers to analyze the traditional video created derivative algorithm based on the algorithm. Even if is the commercial equipment, digital camera sensor also contain tens of thousands of photoelectric detector array. In fact, design the sensor hardware is not high, so it can read a line at a time of photoelectric detector measurement data. Generally speaking, this is not a problem, but for fast moving objects, it may lead to a strange illusion. Helicopter, for example, it may be shall be defined in a line between read and reads the next line movement.
for Davis and his colleagues, this error can be seen as a kind of function. Although traditional slightly twisted open hole on the edge of the objects in the video can’t don’t check, but it contains the object information of high frequency vibration. This information is enough to produce a fuzzy but is likely to be useful audio signals.
at the university of California, Berkeley, an associate professor of electrical engineering and computer science Alexei Efros said: “this is a new, refreshing, so far, no other team can accomplish. We are scientists, sometimes we also see the James Bond movie. We think, ‘this is the theater in Hollywood, is impossible to do that, this is ridiculous. ‘but, suddenly, you’ve got it. This is completely different from the horror film in Hollywood, you will see that as a result of his chips bags vibration surveillance video of a murderer had to confess.”
Efros thinks, characterization of material performance can make the technology very well. But he added: “I’m sure this will be a nobody anticipated application. I think that is a sign of good science when you do something because you think it is cool, then someone can’t imagine things entirely apply it to you. Have the idea is really good!”