Systems and methods for performing commands in a vehicle using speech and image recognition
Systems and methods are disclosed herein for implementation of a vehicle command operation system that may use multi-modal technology to authenticate an occupant of the vehicle to authorize a command and receive natural language commands for vehicular operations. The system may utilize sensors to receive data indicative of a voice command from an occupant of the vehicle. The system may receive second sensor data to aid in the determination of the corresponding vehicular operation in response to the received command. The system may retrieve authentication data for the occupants of the vehicle. The system authenticates the occupant to authorize a vehicular operation command using a neural network based on at least one of the first sensor data, the second sensor data, and the authentication data. Responsive to the authentication, the system may authorize the operation to be performed in the vehicle based on the vehicular operation command.
1. A method comprising:
receiving audio data generated using one or more microphones of a machine, the audio data representative of user speech from a first occupant of the machine;
processing the audio data to determine that the user speech from the first occupant indicates an identifier of a second occupant of the machine and an operation associated with a component of the machine, the second occupant being different than the first occupant;
receiving image data generated using one or more image sensors of the machine, the image data representative of an image depicting at least a portion of an interior of the machine;
based at least on the user speech from the first occupant indicating the identifier of the second occupant, determining, using at least the image data, that the image depicts the second occupant within a region of the machine that is associated with the component; and
causing the operation to be performed with respect to the component.
2. The method of claim 1 , further comprising:
storing data that associates the component with the region of the machine,
wherein the causing the operation to be performed with respect to the component is based at least on the image data depicting the second occupant within the region and the component being associated with the region.
3. The method of claim 1 , wherein the first occupant is associated with a second region within the machine that is different than the region associated with the second occupant within the machine.
4. The method of claim 1 , wherein the identifier of the second occupant comprises one or more of:
a synonym of the second occupant,
a colloquial phase of the second occupant,
a shorthand name of the second occupant,
a name of the second occupant, or
a related descriptor of the second occupant in a different language than a voice command interface.
5. The method of claim 1 , further comprising storing, based at least on a position of the second occupant with the machine, data that associates the second occupant with at least one of the component or the region of the machine that is associated with the component.
6. The method of claim 1 , further comprising:
storing first data representative of an image depicting the second occupant;
storing second data that associates the first data with the identifier of the second occupant; and
identifying, based at least on the first data and the second data, that a portion of the image data represents the second occupant,
wherein the determining that the image depicts the second occupant within the region of the machine that is associated with the component is based at least on the identifying the portion of the image data represents the second occupant.
7. The method of claim 1 , further comprising:
receiving second audio data generated using the one or more microphones of the machine, the second audio data representative of second user speech indicating at least a second identifier of a third occupant and a second operation associated with a second component;
determining, based at least on the second audio data and second image data, a position of the third occupant within the machine;
determining, based at least on the position of the third occupant within the machine, the second component is associated with the second operation; and
causing the second operation to be performed with respect to the second component.
8. A system comprising:
one or more processing units to:
receive audio data generated using one or more microphones of a machine, the audio data representative of user speech from a first occupant of the machine;
process the audio data to determine that the user speech indicates an identifier of a second occupant of the machine and an operation associated with a component of the machine, the second occupant being different than the first occupant;
receive image data generated using one or more image sensors of the machine, the image data representative of an image depicting at least a portion of an interior of the machine;
based at least on the user speech from the first occupant indicating the identifier of the second occupant, determine, using the image data, that the image represents the second occupant within a region of the machine that is associated with the component; and
cause the operation to be performed using the component.
9. The system of claim 8 , wherein the first occupant is within a second region of the machine that is different than the region of the machine.
10. The system of claim 8 , wherein the identifier of the second occupant comprises one or more of:
a synonym of the second occupant,
a colloquial phase of the second occupant,
a shorthand name of the second occupant,
a name of the second occupant, or
a related descriptor of the first occupant in a different language than a voice command interface.
11. The system of claim 8 , wherein the one or more processing units are further to:
store data that associates the component with the region of the machine,
wherein the determination that the image represents the second occupant located within the region of the machine that is associated with the component is further based at least on the data that associates the component with the region.
12. The system of claim 8 , wherein the one or more processing units are further to:
store data that associates the second occupant with at least one of a position within the machine or the component.
13. The system of claim 8 , wherein the one or more processing units are further determine, based at least on the second occupant being located within the region of the machine that is associated with the component and the user speech identifying the operation, to use the component to perform the operation.
14. The system of claim 8 , wherein the one or more processing units are further to:
store second image data representative of a second image depicting the second occupant;
wherein the determination that the image represents the second occupant located within the region of the machine that is associated with the component is further based at least on the second image data.
15. The system of claim 8 , wherein the one or more processing units are further to:
determine a position of a third occupant within the machine;
receive second audio data generated using the one or more microphones of the machine, the second audio data representative of second user speech from the first occupant, the second user speech indicating at least a second operation associated with a second component; and
based at least on the position of the third occupant, cause the second operation to be performed with respect to the second component.
16. The system of claim 8 , wherein a position of the first occupant is determined based at least on the first occupant entering the machine.
17. The system of claim 8 , wherein the system is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for generating or presenting at least one of virtual reality content or augmented reality content;
a system for performing deep learning operations;
a system implemented using a robot;
a system for performing conversational AI operations;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
18. A processor comprising:
one or more processing units to:
determine, based at least on audio data representative of user speech from a first occupant of a machine, that the user speech indicates an operation and an identifier of a second occupant that is different than the first occupant;
based at least on the user speech from the first occupant indicating the identifier of the second occupant, determine, using data representative of an image, that the image represents the second occupant located within a region of the machine that is associated with a component;
determine, based at least on the second occupant being located within the region, to use the component to perform the operation; and
cause the operation to be performed with respect to the component.
19. The processor of claim 18 , wherein the processor is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for generating or presenting at least one of virtual reality content or augmented reality content;
a system for performing deep learning operations;
a system implemented using a robot;
a system for performing conversational AI operations;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
20. The processor of claim 18 , wherein the one or more processing units are further to:
determine that the image also represents the component,
wherein the determination to use the component to perform the operation is further based at least on the image further representing the component.