Granted Patent B2

US 12,293,825 · App. 17/859,608 · Granted May 6, 2025

Inventors: Guido Remi Marcel Gallopyn (Newburyport, MA); Justin Hubbard (Kanagawa, JP); Reid W. Coleman (Rehoboth, MA)

Assignee: Microsoft Technology Licensing, LLC.

G16H40/20

Abstract

Claims (67)

1. A computer-implemented method, executed on a computing device, comprising:

obtaining machine vision information via a machine vision system during an encounter with an encounter participant;

detecting a humanoid shape associated with the encounter participant based on the machine vision information;

based on detecting the humanoid shape, configuring a plurality of audio acquisition devices of an audio recording system to form an audio recording beam directed towards the encounter participant;

obtaining encounter information, via the machine vision system and the audio recording system, during the encounter with the encounter participant;

accessing participant information stored in a resource, the participant information associated with the encounter participant;

processing the encounter information to update the participant information;

processing a free-form input from a user to identify a task;

determining a query to be presented to the user regarding the task based at least in part on one or more of the free-form input, the task, and the participant information; and

outputting the query for presentation to the user.

2. The computer-implemented method of claim 1 wherein the plurality of audio acquisition devices of the audio recording system are configured to form a plurality of audio recording beams directed towards a plurality of encounter participants, and wherein the audio recording system utilizes null-steering precoding to cancel interference between the plurality of encounter participants and noise.

3. The computer-implemented method of claim 1 , further comprising:

generating an order from the free-from input;

converting the order in a non-standardized format into a standardized format;

communicating with one or more order fulfillment systems using the standardized format; and

dispatching the order to the one or more order fulfillment systems.

4. The computer-implemented method of claim 1 wherein processing the encounter information to update the participant information comprises:

updating the participant information in external medical resources before determining the query to be presented to the user regarding the task.

5. The computer-implemented method of claim 1 wherein processing the encounter information to update the participant information comprises marking the encounter information as unverified encounter information.

6. The computer-implemented method of claim 1 wherein determining the query to be presented to the user regarding the task based at least in part on the free-form input, the task, and the participant information comprises:

determining an impediment to performance of the task; and

determining the query to solicit additional information from the user to aid in resolving the impediment; and

wherein supplemental input is received from the user responsive to the query to improve the safety of performing the task.

7. The computer-implemented method of claim 1 wherein the free-form input is processed by a natural language understanding component to identify information expressed in the free-form input indicating an intent to perform the task.

8. A computer program product residing on a non-transitory computer readable medium having a plurality of instructions stored thereon which, when executed by a processor, cause the processor to perform operations comprising:

obtaining machine vision information via a machine vision system during an encounter with an encounter participant;

detecting a humanoid shape associated with the encounter participant based on the machine vision information;

based on detecting the humanoid shape, configuring a plurality of audio acquisition devices of an audio recording system to form an audio recording beam directed towards the encounter participant;

obtaining encounter information, via the machine vision system and the audio recording system, during the encounter with the encounter participant;

accessing participant information stored in a resource, the participant information associated with the encounter participant;

processing the encounter information to update the participant information;

processing a free-form input from a user to identify a task;

determining a query to be presented to the user regarding the task based at least in part on the free-form input, the task, and the participant information; and

outputting the query for presentation to the user.

9. The computer program product of claim 8 wherein the plurality of audio acquisition devices of the audio recording system are configured to form a plurality of audio recording beams directed towards a plurality of encounter participants, and wherein the audio recording system utilizes null-steering precoding to cancel interference between the plurality of encounter participants and noise.

10. The computer program product of claim 8 wherein the operations further comprise:

generating an order from the free-from input;

converting the order in a non-standardized format into a standardized format;

communicating with one or more order fulfillment systems using the standardized format; and

dispatching the order to the one or more order fulfillment systems.

11. The computer program product of claim 8 , wherein processing the encounter information to update the participant information includes updating the participant information in external medical resources before determining the query to be presented to the user regarding the task.

12. The computer program product of claim 8 wherein processing the encounter information to update the participant information comprises marking the encounter information as unverified encounter information.

13. The computer program product of claim 8 wherein determining the query to be presented to the user regarding the task based at least in part on the free-form input, the task, and the participant information comprises:

determining an impediment to performance of the task; and

determining a query to solicit additional information from the user to aid in resolving the impediment; and

wherein supplemental input is received from the user responsive to the query to improve the safety of performing the task.

14. The computer program product of claim 8 wherein the free-form input is processed by a natural language understanding component to identify information expressed in the free-form input indicating an intent to perform the task.

15. A computing system comprising:

a memory; and

a processor configured to process participant information associated with an encounter participant, wherein processing the participant information comprises:

obtaining machine vision information via a machine vision system during an encounter with an encounter participant;

detecting a humanoid shape associated with the encounter participant based on the machine vision information;

based on detecting the humanoid shape, configuring a plurality of audio acquisition devices of an audio recording system to form an audio recording beam directed towards the encounter participant;

obtaining encounter information via the machine vision system and the audio recording system during the encounter with the encounter participant and processing the encounter information in real-time to update the participant information, wherein the processor is further configured to process a free-form input from a user, using natural language understanding, to identify a task, wherein the processor is further configured to determine a query to be presented to the user regarding the task based at least in part on the free-form input, the task, and the participant information; and wherein the processor is further configured to output the query for presentation to the user.

16. The computing system of claim 15 wherein processing the encounter information in real-time to update the participant information comprises marking the encounter information unverified encounter information.

17. The computing system of claim 15 wherein processing the encounter information in real-time to update the participant information comprises updating the participant information in external medical resources before determining the query.

18. The computing system of claim 15 wherein the processor is further configured to:

generate an order from the free-from input;

convert the order in a non-standardized format into a standardized format;

communicate with one or more order fulfillment systems using the standardized format; and

dispatch the order to the one or more order fulfillment systems.

19. The computing system of claim 15 wherein determining the query to be presented to the user regarding the task based at least in part on the free-form input, the task, and the participant information comprises:

determining an impediment to performance of the task; and

determining the query to solicit additional information from the user to aid in resolving the impediment; and

wherein supplemental input is received from the user responsive to the query to improve the safety of performing the task.

20. The computing system of claim 16 wherein determining the query to be presented to the user regarding the task based at least in part on the free-form input, the task, and the participant information comprises:

determining an impediment to performance of the task based upon, at least in part, the unverified encounter information.

Assignments (4)

Continuity (3)

References Cited (91)

Cited By (1)

US 12,572,980