Systems and methods for using artificial intelligence with assistive bots in extended reality environments
Assistive bots, such as computer-generated avatars that do not correspond to real world persons, provide automated instructional capabilities through the use of artificial intelligence to real world users seeking information in extended reality (XR) environments, like students learning in XR environments.
1. A method for assisting a user experiencing an extended reality environment comprising:
displaying an assistive avatar in the extended reality environment to the user with a display of an extended reality hardware device being used by the user, wherein the assistive avatar is automatically controlled in real time by software communicating with the extended reality hardware device and the assistive avatar is not under real time control of a human;
receiving an input from the user in the extended reality environment that provides an inquiry to the assistive avatar for additional information about either a virtual object or a real-world object that the user is holding, touching or pointing to in the extended reality environment, wherein the input further includes the user indicating the virtual object or the real-world object that the user is holding, touching or pointing to, and the additional information is not already present at such time and location in the extended reality environment when the user provides the inquiry to the assistive avatar;
retrieving a response to the inquiry; and
providing the response to the inquiry via the assistive avatar as an answer to the inquiry, wherein said retrieving a response includes submitting a request for information based on the inquiry being provided as text data to an artificial intelligence process that applies a large language model to the text data and then transmits the response that includes the additional information about the virtual object or real-world object.
2. The method of claim 1 , wherein the input includes the user touching or pointing at a virtual page that includes text.
3. The method of claim 1 , wherein the input includes the user touching or pointing at a virtual tablet being displayed in the extended reality environment.
4. The method of claim 1 , wherein the input includes a user speaking the inquiry to the assistive avatar.
5. The method of claim 4 , wherein the response that includes the additional information presents one or more of a picture, graphic, three-dimensional object display and video.
6. The method of claim 5 , wherein the input includes the user speaking a pronoun to reference the virtual object or real-world object.
7. The method of claim 4 , wherein the input includes the user touching or pointing at a virtual page that includes text while the user is speaking the inquiry to the assistive avatar.
8. The method of claim 7 , wherein the input includes the user speaking a pronoun to reference the text or virtual page.
9. The method of claim 4 , wherein the input includes the user touching or pointing at a screen of a virtual tablet being displayed in the extended reality environment while the user is speaking the inquiry to the assistive avatar.
10. The method of claim 9 , wherein the input includes the user speaking a pronoun to reference an object or text shown on the screen of the virtual tablet.
11. The method of claim 4 , wherein the input includes the user speaking a pronoun to reference the virtual object or real-world object.
12. The method of claim 11 , wherein the pronoun is the word “this”.
13. The method of claim 1 , wherein the input includes an image captured from the real world of the user from a camera.
14. The method of claim 13 , further comprising receiving the image from a camera of the extended reality hardware device.
15. The method of claim 14 , further comprising receiving the image from an external camera of a smartphone or webcam device.
16. The method of claim 14 , wherein the camera of the extended reality hardware device is worn by the user.
17. The method of claim 16 , wherein the image is received by the extended reality hardware device while in a passthrough mode that allows the user to see objects in the user's real world.
18. The method of claim 17 , further comprising the extended reality hardware device receiving the image while the camera is restricted from capturing surrounding real-world environment other than subject matter of the image permitted to be received into the extended reality environment.
19. The method of claim 1 , further comprising evaluating context of the user's inquiry to optimize the request for information based on the inquiry to the artificial intelligence process that transmits the reply to the response.
20. The method of claim 1 , wherein the response that includes the additional information presents one or more of a picture, graphic, three-dimensional object display and video.