Remote perception of depth and shape of objects and surfaces
View Patent ↗A computer implemented method of mobile visual assistance to allow users to rapidly distinguish distances to and shapes of multiple objects. 3D images of the environment are obtained using a plurality of user worn video sensors, such as head mounted video cameras. The users are often equipped with a glove like device comprising pointing direction sensors and various haptic transducers. As the user moves though the environment, computer processors use the video sensors to detect various objects in front of the user and report these using secondary or coarse level haptic transducers. The user can use the glove-like device to point at various objects of specific interest, and the system can report further information such as shape or texture data using primary haptic transducers, allowing the user the haptic version of a sharper narrow field of view superimposed on a haptic version of a wider peripheral field of view.
1. A method of visual assistance for a human user in an environment, said method comprising:
using a plurality of user worn video sensors to acquire 3D images of said environment, said environment comprising a plurality of different objects, each different object comprising an object shape and a different 3D object location;
using at least one pointing sensor to determine a virtual pointer and a pointing direction comprising a narrow angle surface encompassing said virtual pointer, and a field of view (pointing region) comprising a wider angle surface surrounding said pointing direction and said narrow angle surface;
a) distance analyzing, using at least one computer processor, computer vision distance algorithms, and said at least one pointing sensor, said 3D images of said environment to identify, along said pointing direction and pointing region, and relative to the location of said user: far pointing direction objects, far pointing region objects, near pointing direction objects, and near pointing region objects;
b) using said at least one computer processor, and a database of haptic signatures, to assign object location haptic signatures to any of said far pointing direction objects, far pointing region objects, near pointing direction objects, and near pointing region objects; and
mapping, using said at least one computer processor and a plurality of spatially separated user worn haptic transducers, said object location haptic signatures onto haptic outputs of said haptic transducers;
wherein at least some of said haptic transducers comprise primary haptic transducers;
further analyzing a shape of at least said near pointing direction objects, using said at least one computer processor and computer vision image recognition algorithms, at least portions of said near pointing direction objects against reference shapes, and assigning corresponding reference shapes to at least portions of said near pointing direction objects;
each reference shape having a corresponding shape haptic signature, said corresponding shape haptic signature selected from said database of haptic signatures;
using said at least one computer processor, said corresponding reference shapes, and said database of haptic signatures to assign at least one object shape haptic signature to at least said portions of said near pointing direction objects; and mapping, using said at least one computer processor and said haptic transducers, said object shape haptic signatures onto haptic outputs of said primary haptic transducers.
2. The method of claim 1 , wherein said haptic transducers comprise primary haptic transducers and secondary haptic transducers, further using said at least one computer processor to map object location haptic signatures from at least said near pointing direction objects onto said primary haptic transducers.
3. The method of claim 1 , wherein said haptic transducers comprise primary haptic transducers and secondary haptic transducers, further using said at least one computer processor to map object location haptic signatures from at least said near pointing region objects onto said secondary haptic transducers.
4. The method of claim 1 , wherein said computer vision distance algorithms comprise stereo image rectification and depth map computation algorithms; and
said computer vision image recognition algorithms comprise any of geometric modelling methods, convolutional neural network methods, and single shot multibox detection SSD frameworks.
5. The method of claim 1 , wherein said video sensors are spatially separated video sensors configured to be worn on a head of said user.
6. The method of claim 1 , wherein said haptic transducers comprise at least two haptic transducers, comprising any of:
f) eccentric rotating mass transducers;
g) linear resonance array transducers;
h) piezo transducers;
said haptic transducers configured to produce detectable haptic sensations on a skin of said user.
7. The method of claim 1 , wherein at least some of said haptic transducers and said at least one pointing sensor comprise at least one glove type device, covering a portion of said user's hand, that is configured to be worn on a hand of said user.
8. The method of claim 7 , wherein at least some of said haptic transducers comprise primary haptic transducers configured to provide haptic input to at least one finger or thumb of said user.
9. The method of claim 8 , wherein at least some of said haptic transducers comprise secondary haptic transducers configured to provide haptic input to regions of said user's arm or hand that exclude said user's fingers and thumb.
10. The method of claim 1 , wherein said at least one pointing sensor comprise any of at least one accelerometers, gyroscopes, magnetometers, or video cameras.
11. The method of claim 1 , further using said at least one computer processor to determine a distance or direction to any of said far pointing direction objects, far pointing region objects, near pointing direction objects, and near pointing region objects; and
further modifying said object location haptic signatures corresponding to at least portions of any of said far pointing direction objects, far pointing region objects, near pointing direction objects, and near pointing region objects according to said distance or direction.
12. A method of visual assistance for a human user in an environment, said method comprising:
using a plurality of user worn video sensors to acquire 3D images of said environment, said environment comprising a plurality of different objects, each different object comprising an object shape and a different 3D object location;
using a glove type device comprising at least one pointing sensor to determine a virtual pointer and a pointing direction comprising a narrow angle surface encompassing said virtual pointer, and a field of view (pointing region) comprising a wider angle surface surrounding said pointing direction and said narrow angle surface;
said glove type device also comprising at least one primary haptic transducer;
a) distance analyzing, using at least one computer processor, computer vision distance algorithms, and said at least one pointing sensor, said 3D images of said environment to identify, along said pointing direction and pointing region, and relative to the location of said user: far pointing direction objects, far pointing region objects, near pointing direction objects, and near pointing region objects;
b) using said at least one computer processor, and a database of haptic signatures, to assign object location haptic signatures to any of said far pointing direction objects, far pointing region objects, near pointing direction objects, and near pointing region objects;
c) further analyzing a shape of at least said near pointing direction objects, using said at least one computer processor and computer vision image recognition algorithms, at least portions of said near pointing direction objects against reference shapes, and assigning corresponding reference shapes to at least portions of said near pointing direction objects;
each reference shape having a corresponding shape haptic signature, said corresponding shape haptic signature selected from said database of haptic signatures;
using said at least one computer processor, said corresponding reference shapes, and said database of haptic signatures to assign at least one object shape haptic signature to at least said portions of said near pointing direction objects; and
mapping, using said at least one computer processor, said object location haptic signatures onto haptic outputs of any of said primary haptic transducers and secondary haptic transducers, and mapping said at least one object shape haptic signature to any of said primary haptic transducers.
13. The method of claim 12 , wherein said computer vision distance algorithms comprise stereo image rectification and depth map computation algorithms; and
said computer vision image recognition algorithms comprise any of geometric modelling methods, convolutional neural network methods, and single shot multibox detection SSD frameworks.
14. The method of claim 12 , wherein said video sensors are spatially separated video sensors configured to be worn on a head of said user.
15. The method of claim 12 , wherein any of said primary haptic transducers and said secondary haptic transducers comprise any of:
f) eccentric rotating mass transducers;
g) linear resonance array transducers;
h) piezo transducers;
said primary haptic transducers and said secondary haptic transducers configured to produce detectable haptic sensations on a skin of said user.
16. The method of claim 12 , wherein said at least one pointing sensor comprise any of at least one accelerometers, gyroscopes, magnetometers, or video cameras.
17. The method of claim 12 , further using said at least one computer processor to determine a distance or direction to any of said far pointing direction objects, far pointing region objects, near pointing direction objects, and near pointing region objects; and
further modifying said object location haptic signatures corresponding to at least portions of any of said far pointing direction objects, far pointing region objects, near pointing direction objects, and near pointing region objects according to said distance or direction.