Co-located pose estimation in a shared artificial reality environment
Artificial reality (AR) systems track pose and skeletal positioning for multiple co-located participants, each having a head mounted display (HMD). Participants can join a shared artificial reality event or experience with others in the same location. Each participant's HMD can independently render AR content for the participant based on the participant's pose and pose information obtained from other participants' HMDs. A participating HMD may broadcast tracking estimates for skeletal points of interest (e.g., joints, finger tips, knees, ankle points, etc.) that are within the field-of-view of the HMD's cameras and/or sensors. A participating HMD may receive skeletal position information determined by other HMDs, and aggregate the received tracking information along with internal tracking information to construct an accurate, full estimate of its own pose and skeletal positioning information for its corresponding participant.
1. A method comprising:
obtaining, from an image capture device of a first head mounted display (HMD), first image data representative of a physical environment;
determining, by processing circuitry of the first HMD, a first HMD pose representing a position and orientation of the first HMD;
determining, by the processing circuitry of the first HMD from the first image data, first body position information of the users of one or more second HMDs within the physical environment;
transmitting, by the first HMD, the first HMD pose and the first body position information for use by the one or more second HMDs;
receiving, by the first HMD from each second HMD of the one or more second HMDs, a second HMD pose of the respective second HMD and second body position information determined by the second HMD from second image data obtained by the second HMD, wherein the second body position information includes body position information for a user of the first HMD;
integrating, by the processing circuitry of the first HMD, the first body position information with the second body position information to create first solved body position information of the user of the first HMD; and
rendering, by the processing circuitry of the first HMD for display at the first HMD, artificial reality content in accordance with the first pose and the first solved body position information.
2. The method of claim 1 , further comprising:
transmitting, by the first HMD, the first solved body position information for use by the one or more second HMDs; and
receiving, by the first HMD, second solved body position information for users of the one or more second HMDs;
wherein rendering the artificial reality content comprises rendering the artificial reality content in accordance with the second solved body position information.
3. The method of claim 1 , further comprising:
receiving confidence levels for the first body position information and the second body position information;
wherein integrating the first body position information and the second body position information comprises integrating the first body position information and the second body position information in accordance with the confidence levels.
4. The method of claim 3 , wherein integrating the first body position information and the second body position information in accordance with the confidence levels comprises integrating the first body position information and the second body position information in accordance with weights assigned to the first body position information and the second body position information based, at least in part, on the confidence levels.
5. The method of claim 1 , further comprising filtering out from the second body position information, body position information received by the first HMD from a second HMD of the one or more second HMDs that is more than a threshold distance from the first HMD.
6. The method of claim 1 , further comprising filtering p from the second body position information, body position information received by the first HMD from a second HMD of the one or more second HMDs that is not one of n closest second HMDs to the first HMD, where n is a predetermined or configurable number.
7. The method of claim 1 , further comprising:
wherein determining the first body position information of the users of the one or more second HMDs comprises passing the first image data through a machine learning model trained to recognize body positions.
8. The method of claim 1 , wherein determining the first body position information of the users of the one or more second HMDs comprises determining, from the first image data, the first body position information of the users without markers disposed on the users.
9. An artificial reality system comprising:
an image capture device comprising circuitry configured to capture first image data representative of a physical environment;
a first head mounted display (HMD) comprising processing circuitry configured to output artificial reality content;
a pose tracker comprising processing circuitry configured to:
determine a first HMD pose representing a position and orientation of the first HMD,
determine, from the first image data, first body position information of the users of one or more second HMDs within the physical environment,
transmit the first HMD pose and the first body position information for use by the one or more second HMDs,
receive, from each second HMD of the one or more second HMDs, a second HMD pose of the respective second HMD and second body position information determined by the second HMD from second image data obtained by the second HMD, wherein the second body position information includes body position information for a user of the first HMD, and
integrate the first body position information with the second body position information to create first solved body position information of the user of the first HMD; and
a rendering engine comprising processing circuitry configured to render, for display at the first HMD, artificial reality content in accordance with the first HMD pose and the first solved body position information.
10. The artificial reality system of claim 9 , wherein the pose tracker is further configured to:
transmit the first solved body position information for use by the one or more second HMDs; and
receive second solved body position information for the users of the one or more second HMDs,
wherein the rendering engine is further configured to render, for display at the first HMD, the artificial reality content in accordance with the second solved body position information.
11. The artificial reality system of claim 9 , wherein the pose tracker is further configured to receive confidence levels for the first body position information and the second body position information, and wherein the pose tracker integrates the first body position information and the second body position information in accordance with the confidence levels.
12. The artificial reality system of claim 9 , wherein the first HMD and the one or more second HMDs are co-located HMDs within the physical environment participating in an artificial reality application.
13. The artificial reality system of claim 12 , wherein the artificial reality application executes on each of the co-located HMDs.
14. The artificial reality system of claim 9 , wherein the image capture device is integrated with the first HMD.
15. A non-transitory computer-readable medium comprising instructions that, when executed, cause one or more processors of an artificial reality system to:
obtain, from an image capture device of a first head mounted display (HMD), first image data representative of a physical environment;
determine a first HMD pose representing a position and orientation of the first HMD;
determining, from the first image data, first body position information of the users of one or more second HMDs;
transmit, by the first HMD, the first HMD pose and the first body position information for use by the one or more second HMDs;
receive, by the first HMD from each second HMD of the one or more second HMDs, a second HMD pose and second body position information determined by the second HMD from second image data obtained by the second HMD, wherein the second body position information includes body position information for a user of the first HMD;
integrate, by the first HMD, the first body position information with the second body position information to create first solved body position information of the user of the first HMD; and
render, for display at the first HMD, artificial reality content in accordance with the first pose and the first solved body position information.
16. The non-transitory computer-readable medium of claim 15 , wherein the instructions further comprise instructions to:
transmit, by the first HMD, the first solved body position information for use by the one or more second HMDs; and
receive, by the first HMD, second solved body position information for the one or more second HMDs;
wherein the instructions to render the artificial reality content comprise instructions to render the artificial reality content in accordance with the second solved body position information.
17. The non-transitory computer-readable medium of claim 15 , wherein the instructions further comprise instructions to:
receive confidence levels for the first body position information and the second body position information;
wherein the instructions to integrate the first body position information and the second body position information comprise instructions to integrate the first body position information and the second body position information in accordance with the confidence levels.