Summary
Modern robots are equipped with advanced vision hardware, often 3D (using a stereo camera or a Kinect-like device). However, multimodal perception is still in development, especially in the area of haptics (that is, touching things). Not many robots have haptic sensors, and even when they do, they aren't standardized and we don't always know how to interpret the data.
In my project, we designed and built the Proton Pack, a portable, self-contained, human-operated visuo-haptic sensing device. It integrates visual sensors with an interchangeable haptic end-effector, perceiving a surface simultaneously.
Here are the major parts of the project:
- Calibration: in order to relate all the sensor readings to each other, we went through several calibration processes to determine the relative camera positions and the mechanical properties of each end-effector.
- Surface classification: to make sure that the data collected with the Proton Pack is relevant to haptic perception, we did some proof-of-concept classification among small sets of surfaces. This was also used to validate hardware improvements that we made early on.
- Data collection: we used the Proton Pack to collect a large dataset of end-effector/surface interactions. Both the Proton Pack and several sets of material samples traveled across the Atlantic for this project.
- Machine learning: we attempted to train a computer to "feel with its eyes" (that is, predict haptic properties from visual images) using the dataset we collected.
Results
Code
