Rendering and anchoring instructional data in augmented reality with context awareness
In some examples, an augmented reality (AR) server receives instructional data to be rendered in AR. The AR rendering server extracts multiple instruction steps from the instructional data and determines multiple spatial identifiers associated with the multiple instruction steps respectively. The multiple spatial identifiers correspond to multiple spatial objects in a real-world environment. The AR rendering server then generates AR rendering data for displaying the multiple instruction steps on an AR device at selected locations associated with the multiple spatial objects in the real-world environment. The AR rendering data is then transmitted to the AR device.
1. A method performed by one or more processing devices, comprising:
receiving instructional data to be rendered in augmented reality (AR);
extracting a plurality of instruction steps from the instructional data;
predicting a plurality of spatial identifiers associated with the plurality of instruction steps respectively using a prediction model, wherein the plurality of spatial identifiers correspond to a plurality of spatial objects respectively in a real-world environment;
generating one or more heatmaps associated with the plurality of spatial objects based on previous user behavior data associated with the plurality of spatial objects, each heatmap comprising a first region where a distribution level of the previous user behavior data is greater than a threshold value and a second region where the distribution level of the previous user behavior data is less than the threshold value;
selecting an anchoring location for an instructional step from the second region in each heatmap associated with a corresponding spatial object to obtain a plurality of anchoring locations;
generating AR rendering data for the plurality of instruction steps to be displayed via an AR device at the plurality of anchoring locations associated with the plurality of spatial objects based, at least in part, upon the plurality of spatial identifiers and a spatial profile of the real-world environment; and
transmitting the AR rendering data for the plurality of instruction steps to the AR device.
2. The method of claim 1 , wherein the instructional data is received from a scanning device.
3. The method of claim 1 , wherein the instruction data is received from a computing device.
4. The method of claim 1 , wherein the prediction model comprises a pre-trained Bidirectional Encoder Representations from Transformers (BERT)-based model.
5. The method of claim 1 , wherein the prediction model is trained with a collection of instructions and crowdsourced spatial identifiers until a prediction accuracy is more than 80%.
6. The method of claim 1 , wherein the prediction model is trained with a number of instructions and respective spatial identifiers selected from object keywords appeared in the number of instructions.
7. The method of claim 1 , wherein the previous user behavior data collected from user interactions with the plurality of spatial objects.
8. The method of claim 1 , wherein the previous user behavior data comprises one or more of head pose data or hand gesture data.
9. The method of claim 1 , further comprising:
extracting time information from one or more instruction steps of the plurality of instruction steps; and
rendering one or more timers to be displayed via the AR device associated with the one or more instruction steps.
10. The method of claim 1 , wherein the spatial profile for the real-world environment is retrieved from a cloud storage, wherein the spatial profile comprises geometry data, location data, and identity data for the plurality of spatial objects in the real-world environment.
11. The method of claim 1 , wherein the AR device is configured to detect the plurality of spatial objects by scanning the real-world environment and display the plurality of instruction steps sequentially at the plurality of anchoring locations associated with the plurality of spatial objects.
12. A system, comprising:
a memory component;
a processing device coupled to the memory component, the processing device to perform operations comprising:
receiving instructional data to be rendered in augmented reality (AR);
extracting a plurality of instruction steps from the instructional data;
determining a plurality of spatial identifiers associated with the plurality of instruction steps respectively, wherein the plurality of spatial identifiers correspond to a plurality of spatial objects in a real-world environment;
generating one or more heatmaps associated with the plurality of spatial objects based on previous user behavior data associated with the plurality of spatial objects, each heatmap comprising a first region where a distribution level of the previous user behavior data is greater than a threshold value and a second region where the distribution level of the previous user behavior data is less than the threshold value;
selecting an anchoring location for an instructional step from the second region in each heatmap associated with a corresponding spatial object to obtain a plurality of anchoring locations;
generating AR rendering data for the plurality of instruction steps to be displayed via an AR device at the plurality of anchoring locations associated with the plurality of spatial objects in the real-world environment; and
transmitting the AR rendering data to the AR device.
13. The system of claim 12 , wherein the plurality of spatial identifiers are determined using a prediction model, wherein the prediction model comprises a pre-trained Bidirectional Encoder Representations from Transformers (BERT)-based model.
14. The system of claim 13 , wherein the prediction model is trained with a number of instructions and respective spatial identifiers selected from object keywords appeared in the number of instructions.
15. The system of claim 12 , wherein the AR rendering data is generated based at least in part upon the plurality of spatial identifiers and a spatial profile of the real-world environment, wherein the spatial profile for the real-world environment is retrieved from a cloud storage, wherein the spatial profile comprises geometry data, location data, and identity data for the plurality of spatial objects in the real-world environment.
16. A non-transitory computer-readable medium, storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising:
receiving instructional data to be rendered in augmented reality (AR);
extracting a plurality of instruction steps from the instructional data;
a step for determining a plurality of spatial identifiers associated with the plurality of instruction steps respectively, wherein the plurality of spatial identifiers correspond to a plurality of spatial objects in a real-world environment;
generating one or more heatmaps associated with the plurality of spatial objects based on previous user behavior data associated with the plurality of spatial objects, each heatmap comprising a first region where a distribution level of the previous user behavior data is greater than a threshold value and a second region where the distribution level of the previous user behavior data is less than the threshold value;
selecting an anchoring location for an instructional step from the second region in each heatmap associated with a corresponding spatial object to obtain a plurality of anchoring locations;
a step for generating AR rendering data for the plurality of instruction steps to be displayed via an AR device; and
transmitting the AR rendering data for the plurality of instruction steps to the AR device.
17. The non-transitory computer-readable medium of claim 16 , wherein the instructional data is received from a scanning device.
18. The non-transitory computer-readable medium of claim 16 , wherein the instructional data is received from a computing device.
19. The non-transitory computer-readable medium of claim 16 , wherein the operations further comprise:
extracting time information from one or more instruction steps of the plurality of instruction steps; and
rendering one or more timers to be displayed via the AR device associated with the one or more instruction steps.