Gesture tracking system
Various implementations disclosed herein include devices, systems, and methods that identify a gesture based on event camera data and frame-based camera data (e.g., for a CGR environment). In some implementations at an electronic device having a processor, event camera data is obtained corresponding to light (e.g., IR light) reflected from a physical environment and received at an event camera. In some implementations, frame-based camera data is obtained corresponding to light (e.g., visible light) reflected from the physical environment and received at a frame-based camera. In some implementations, a subset of the event camera data is identified based on the frame-based camera data, and a gesture (e.g., of a person in the physical environment) is identified based on the subset of event camera data. In some implementations, a path (e.g., of a hand) by tracking a grouping of blocks of event camera events in the subset of event camera data.
1. A method of gesture recognition, the method comprising:
at an electronic device having a processor;
obtaining event camera data corresponding to light reflected from a physical environment and received at an event camera, the event camera data produced based on changes in light intensity detected at pixel sensors of the event camera;
obtaining frame-based camera data corresponding to light reflected from the physical environment and received at a frame-based camera;
identifying a subset of the event camera data based on identifying that the event camera data of the subset corresponds to a portion of a person based on the frame-based camera data; and
identifying a gesture based on the subset of event camera data.
2. The method of claim 1 , wherein the event camera uses a first light wavelength band and the frame-based camera uses a different second light wavelength range.
3. The method of claim 1 , further comprising illuminating the physical environment with an infrared (IR) light source, wherein the event camera uses reflected IR light and the frame-based camera uses reflected visible light.
4. The method of claim 3 , wherein the IR light source is variable in size, intensity, or steerable.
5. The method of claim 1 , wherein the event camera uses reflected IR light within an intensity range.
6. The method of claim 1 , wherein identifying the subset of the event camera data comprises excluding event camera data corresponding to a background portion of the physical environment.
7. The method of claim 1 , wherein identifying the subset of the event camera data comprises color analysis of the frame-based camera data.
8. The method of claim 1 , wherein identifying the subset of the event camera data comprises temporally and spatially correlating a field of view of the event camera with a field of view of the frame-based camera.
9. The method of claim 1 , wherein the electronic device comprises the event camera or wherein a second electronic device comprises the event camera, the second electronic device separate from the electronic device.
10. The method of claim 1 , wherein the event camera data corresponds to pixel events triggered based on changes in light intensity at pixel sensors exceeding a comparator threshold or wherein the event camera data excludes data corresponding to visible light (VL) intensity changes.
11. The method of claim 1 , wherein the gesture comprises a hand movement.
12. The method of claim 1 , wherein identifying the gesture comprises evaluating the event camera data using a machine learning algorithm.
13. The method of claim 1 , wherein identifying the gesture comprises identifying a path of one or more hands of a person in the physical environment.
14. A system comprising:
a non-transitory computer-readable storage medium; and
one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the system to perform operations comprising:
obtaining event camera data corresponding to light reflected from a physical environment and received at an event camera, the event camera data produced based on changes in light intensity detected at pixel sensors of the event camera;
obtaining frame-based camera data corresponding to light reflected from the physical environment and received at a frame-based camera;
identifying a subset of the event camera data based on identifying that the event camera data of the subset corresponds to a portion of a person based on the frame-based camera data; and
identifying a gesture based on the subset of event camera data.