Multi-SoC hand-tracking platform
A multi-System on Chip (SoC) hand-tracking platform is provided. The multi-SoC hand-tracking platform includes a computer vision SoC and one or more application SoCs. The computer vision SoC hosts a hand-tracking input pipeline. The one or more application SoCs host one or more applications that are consumers of input event data generated by the hand-tracking input pipeline. The applications communicate with some components of the hand-tracking input pipeline using a shared-memory buffer and with some of the components of the hand-tracking input pipeline using Inter-Process Communication (IPC) method calls.
1. A computing system comprising:
a first System on Chip (SoC) comprising:
first one or more processors; and
a first memory storing first instructions that, when executed by the first one or more processors, cause the first SoC to perform operations of a hand-tracking input pipeline of an Augmented Reality (AR) system, the operations comprising:
generating real-world scene environment frame data based on hand movements being made by a user of the AR system using a camera component of the hand-tracking input pipeline;
generating gesture input event data based on the real-world scene environment frame data and language model feedback data received from a component of the AR system; and
communicating the gesture input event data to the component of the AR system; and
a second SoC comprising:
second one or more processors; and
a second memory storing second instructions that, when executed by the second one or more processors, cause the second SoC to perform operations of the component of the AR system, the operations comprising:
generating the language model feedback data based on context data of registered gestures;
communicating the language model feedback data to the hand-tracking input pipeline;
receiving the gesture input event data;
classifying the gesture input event data as undirected or directed events based on component registration data of one or more application components of the AR system; and
routing the gesture input event data to an application component of the one or more application components based on the classifying of the gesture input data and the component registration data.
2. The computing system of claim 1 , wherein the hand-tracking input pipeline and the component of the AR system communicate via an Inter-Process Communication (IPC) protocol.
3. The computing system of claim 1 , wherein the first instructions that, when executed by the first one or more processors, cause the first SoC to perform operations of the hand-tracking input pipeline, further cause the first SoC to perform operations comprising:
generating skeletal model data based on the real-world scene environment frame data.
4. The computing system of claim 3 , wherein the first instructions that, when executed by the first one or more processors, cause the first SoC to perform operations of the hand-tracking input pipeline, further cause the first SoC to perform operations comprising:
generating hand classifier probability data based on the skeletal model data.
5. The computing system of claim 4 , wherein the first instructions that, when executed by the first one or more processors, cause the first SoC to perform operations of the hand-tracking input pipeline, further cause the first SoC to perform operations comprising:
generating the gesture input event data based on the hand classifier probability data.
6. The computing system of claim 1 , wherein the first instructions that, when executed by the first one or more processors, further cause the first SoC to perform operations comprising:
generating coordinate transformation data based on the real-world scene environment frame data.
7. The computing system of claim 1 , wherein the AR system comprises a head-worn device.
8. A computer-implemented method, comprising:
generating, by first one or more processors of a first System on Chip (SoC), real-world scene environment frame data based on hand movements being made by a user of an Augmented Reality (AR) system using a camera component of a hand-tracking input pipeline of the AR system;
generating, by the hand-tracking input pipeline of the AR system using the first one or more processors, gesture input event data based on the real-world scene environment frame data and language model feedback data received from a component of the AR system;
communicating, by the hand-tracking input pipeline of the AR system using the first one or more processors, the gesture input event data to the component of the AR system;
generating, by the component using second one or more processors of a second SoC, the language model feedback data based on context data of registered gestures;
communicating, by the component using second one or more processors, the language model feedback data to the first SoC;
receiving, by the component using second one or more processors of a second SoC, the gesture input event data;
classifying, by the component using second one or more processors of a second SoC, the gesture input event data as undirected or directed events based on component registration data of one or more application components of the AR system; and
routing, by the component using second one or more processors of a second SoC, the gesture input event data to an application component of the one or more application components based on the classifying of the gesture input data and the component registration data.
9. The computer-implemented method of claim 8 , wherein the hand-tracking input pipeline and the component communicate via an Inter-Process Communication (IPC) protocol.
10. The computer-implemented method of claim 8 , further comprising:
generating, by the hand-tracking input pipeline of the AR system using the first one or more processors, skeletal model data based on the real-world scene environment frame data.
11. The computer-implemented method of claim 10 , further comprising:
generating, by the hand-tracking input pipeline of the AR system using the first one or more processors, hand classifier probability data based on the skeletal model data.
12. The computer-implemented method of claim 11 , further comprising:
generating, by the hand-tracking input pipeline of the AR system using the first one or more processors, the gesture input event data based on the hand classifier probability data.
13. The computer-implemented method of claim 8 , further comprising:
generating, by the hand-tracking input pipeline of the AR system using the first one or more processors, coordinate transformation data based on the real-world scene environment frame data.
14. The computer-implemented method of claim 8 , wherein the AR system comprises a head-worn device.
15. A non-transitory computer-readable storage medium, the computer-readable storage medium comprising:
first instructions for a first System on Chip (SoC) that, when executed by the first SoC, cause the first SoC to perform first operations of a hand-tracking input pipeline of an Augmented Reality (AR) system, the first operations including:
generating real-world scene environment frame data based on hand movements being made by a user of the AR system using a camera component of the hand-tracking input pipeline;
generating gesture input event data based on the real-world scene environment frame data and language model feedback data received from a component of the AR system; and
communicating the gesture input event data to the component of the AR system; and
second instructions for a second SoC that, when executed by the second SoC, cause the second SoC to perform second operations of the component of the AR system, the second operations including:
generating the language model feedback data based on context data of registered gestures;
communicating the language model feedback data to the hand-tracking input pipeline;
receiving the gesture input event data;
classifying the gesture input event data as undirected or directed events based on component registration data of one or more application components of the AR system; and
routing the gesture input event data to an application component of the one or more application components based on the classifying of the gesture input data and the component registration data.
16. The non-transitory computer-readable storage medium of claim 15 , wherein the first operations further comprise:
communicating the gesture input event data to the component via an Inter-Process Communication (IPC) protocol.
17. The non-transitory computer-readable storage medium of claim 15 , wherein the first operations further comprise:
generating skeletal model data based on the real-world scene environment frame data.
18. The non-transitory computer-readable storage medium of claim 17 , wherein the first operations further comprise:
generating hand classifier probability data based on the skeletal model data.
19. The non-transitory computer-readable storage medium of claim 18 , wherein the first operations further comprise:
generating the gesture input event data based on the hand classifier probability data.
20. The non-transitory computer-readable storage medium of claim 15 , wherein the first operations further comprise:
generating coordinate transformation data based on the real-world scene environment frame data.
21. The non-transitory computer-readable storage medium of claim 15 , wherein the AR system comprises a head-worn device.