Three dimensional spatial instructions for artificial intelligence assistance authoring

1 . A data processing system comprising:

a processor; and

a machine-readable storage medium storing executable instructions that, when executed, cause the processor alone or in combination with other processors to perform operations of:

receiving a first request from a client device of a user for creating a mixed-reality experience associated with performing a task on a complex multipart object;

obtaining three-dimensional object geometry information for the complex multipart object by:

capturing multiple views of an object part of the complex multipart object in a plurality of keyframes;

performing an interactive segmentation process on selected keyframes of the plurality of keyframes to segment the object part from a remainder of the complex multipart object;

propagating segmentation to a remainder of the plurality of keyframes to generate multiple three-dimensional representations of the object part; and

fusing the multiple three-dimensional representations of the object part into a single three-dimensional representation of the object part;

obtaining a description of the task to be performed including a plurality of subtasks each associated with a user action to be performed on a respective part of the complex multipart object;

constructing a prompt to a vision-language model planner using a prompt construction unit, the prompt instructing the vision-language model planner to generate a task list based on the three-dimensional object geometry information and the description of the task to be performed;

providing the prompt as an input to the vision-language model planner;

obtaining the task list from the vision-language model planner; and

generating content for the mixed-reality experience using the task list in response to a second request to execute the mixed-reality experience.

2 . The data processing system of claim 1 , wherein the vision-language model planner is a large language model (LLM).

3 . The data processing system of claim 1 , wherein constructing the prompt further comprises:

accessing a prompt template that provides instructions to the vision-language model planner; and

updating the prompt template based on the three-dimensional object geometry information and the description of the task to be performed.

4 . The data processing system of claim 1 , wherein the task list comprises a list of subtasks associated with the task, a user action to be performed for each subtask, and an object part on which the user action is to be performed.

5 . A data processing system comprising:

a processor; and

a machine-readable storage medium storing executable instructions that, when executed, cause the processor alone or in combination with other processors to perform operations of:

receiving a first request from a client device of a user for creating a mixed-reality experience associated with performing a task on a complex multipart object;

obtaining three-dimensional object geometry information for the complex multipart object;

obtaining a description of the task to be performed including a plurality of subtasks each associated with a user action to be performed on a respective part of the complex multipart object;

constructing a prompt for a vision-language model planner to cause the vision-language model planner to generate a series of queries to the user to provide a description of each subtask of the plurality of subtasks;

causing the client device of the user to present the series of queries to the user in a mixed reality environment;

receiving audio content from the client device of the user that include a spoken response to the series of queries; and

providing the audio content to a transcription model to generate a textual transcript of the spoken response to the series of queries;

constructing a prompt to the vision-language model planner using a prompt construction unit, the prompt instructing the vision-language model planner to generate a task list based on the three-dimensional object geometry information and the description of the task to be performed, the task list comprising a list of the plurality of subtasks associated with the task, a user action to be performed for each subtask, and an object part on which the user action is to be performed;

providing the prompt as an input to a vision-language model planner;

obtaining the task list from the vision-language model planner; and

generating content for a mixed reality experience using the task list in response to a second request to execute the mixed reality experience.

6 . The data processing system of claim 5 , wherein obtaining the three-dimensional object geometry information for the complex multipart object further comprises:

capturing multiple views of an object part of the complex multipart object in a plurality of keyframes;

performing an interactive segmentation process on selected keyframes of the plurality of keyframes to segment the object part from the rest a remainder of the complex multipart object;

propagating segmentation to a remainder of the plurality of keyframes to generate multiple three-dimensional representations of the object part; and

fusing the multiple three-dimensional representations of the object part into single three-dimensional representation of the object part.

7 . A data processing system comprising:

a processor; and

a machine-readable storage medium storing executable instructions that, when executed, cause the processor alone or in combination with other processors to perform operations of:

receiving a first request from a client device of a user for creating a mixed-reality experience associated with performing a task on a complex multipart object;

obtaining three-dimensional object geometry information for the complex multipart object;

obtaining a description of the task to be performed including a plurality of subtasks each associated with a user action to be performed on a respective part of the complex multipart object;

constructing a prompt to a vision-language model planner using a prompt construction unit, the prompt instructing the vision-language model planner to generate a task list based on the three-dimensional object geometry information and the description of the task to be performed;

providing the prompt as an input to the vision-language model planner;

obtaining the task list from the vision-language model planner;

generating content for the mixed-reality experience using the task list in response to a second request to execute the mixed-reality experience;

constructing a prompt for the vision-language model planner to cause the vision-language model planner to generate a series of queries to the user to demonstrate each subtask of the plurality of subtasks;

causing the client device of the user to present the series of queries to the user in a mixed reality environment;

receiving video content from the client device of the user that include demonstrations of the user performing each subtask of the plurality of subtasks in response to the series of queries; and

analyzing the video content using a video analysis model to identify a user action performed for each subtask of the plurality of subtasks.

8 . The data processing system of claim 7 , wherein analyzing the video content using the video analysis model further comprises:

selecting an action symbol representing the user action performed for each subtask of the plurality of subtasks, the action symbol providing a visual representation of the user action to be performed.

9 . The data processing system of claim 8 , wherein the machine-readable storage medium further includes instructions configured to cause the processor alone or in combination with other processors to perform operations of:

detecting a hand trajectory of a hand of the user for each subtask of the plurality of subtasks; and

determining the user action performed for each subtask of the plurality of subtasks based on the hand trajectory.

10 . The data processing system of claim 9 , wherein the machine-readable storage medium further includes instructions configured to cause the processor alone or in combination with other processors to perform operations of:

determining a pose for the action symbol for each subtask of the plurality of subtasks based on the hand trajectory associated with each subtask.

11 . A method implemented in a data processing system for creating a mixed reality experience, the method comprising:

receiving a first request from a client device of a user for creating a mixed-reality experience associated with performing a task on a complex multipart object;

obtaining three-dimensional object geometry information for the complex multipart object by:

capturing multiple views of an object part of the complex multipart object in a plurality of keyframes;

performing an interactive segmentation process on selected keyframes of the plurality of keyframes to segment the object part from a remainder of the complex multipart object;

propagating segmentation to a remainder of the plurality of keyframes to generate multiple three-dimensional representations of the object part; and

fusing the multiple three-dimensional representations of the object part into a single three-dimensional representation of the object part;

obtaining a description of the task to be performed including a plurality of subtasks each associated with a user action to be performed on a respective part of the complex multipart object;

constructing a prompt to a vision-language model planner using a prompt construction unit, the prompt instructing the vision-language model planner to generate a task list based on the three-dimensional object geometry information and the description of the task to be performed;

providing the prompt as an input to the vision-language model planner;

obtaining the task list from the vision-language model planner; and

generating content for the mixed reality experience using the task list in response to a second request to execute the mixed reality experience.

12 . The method of claim 11 , wherein the vision-language model planner is a large language model (LLM).

13 . The method of claim 11 , wherein constructing the prompt further comprises:

accessing a prompt template that provides instructions to the vision-language model planner; and

updating the prompt template based on the three-dimensional object geometry information and the description of the task to be performed.

14 . The method of claim 11 , wherein the task list comprises a plurality of subtasks associated with the task, a user action to be performed for each subtask, and an object part on which the user action is to be performed.

15 . A data processing system comprising:

a processor; and

a machine-readable storage medium storing executable instructions that, when executed, cause the processor alone or in combination with other processors to perform operations of:

receiving a request from a client device of a user to present a mixed reality experience on the client device, the mixed reality experience comprising an interactive guide for performing a task involving a complex multipart object, the interactive guide comprising a plurality of subtasks to be performed by the user;

obtaining a task list associated with the mixed reality experience from a mixed reality experience datastore;

constructing a prompt to a vision-language model planner using a prompt construction unit, the prompt instructing the vision-language model planner to initialize the mixed reality experience based on the task list and to generate a series of queries to the user to demonstrate each subtask of the plurality of subtasks;

providing the prompt as an input to the vision-language model planner;

generating content for the mixed reality experience based on the task list using the vision-language model planner;

causing the client device of the user to present a visualization of the content including causing the client device of the user to present the series of queries to the user in a mixed reality environment;

receiving video content from the client device of the user that include demonstrations of the user performing each subtask of the plurality of subtasks in response to the series of queries; and

analyzing the video content using a video analysis model to identify a user action performed for each subtask of the plurality of subtasks.

16 . The data processing system of claim 15 , wherein the vision-language model planner is a large language model (LLM).

17 . The data processing system of claim 15 , wherein the task list comprises a list of subtasks associated with the task, a user action to be performed for each subtask, and an object part of the complex multipart object on which the user action is to be performed.

18 . The data processing system of claim 15 , wherein generating content for the mixed reality experience based on the task list using the vision-language model planner further comprises:

interactively guiding the user through a list of subtasks associated with a task to be performed.

19 . The data processing system of claim 15 , wherein the machine-readable storage medium further includes instructions configured to cause the processor alone or in combination with other processors to perform operations of:

presenting action symbols associated with the task to be performed for each of the subtasks of the plurality of subtasks, the action symbols providing a graphical representation of an action to be performed for each subtask.