Video integration with home assistant

1. A method for handling spoken commands based on characteristics observed by an integrated video service, the method comprising:

receiving a video stream from a streaming video camera;

analyzing a field of view in the received video stream to determine a location for one or more identified or potential users;

beamforming audio from microphones of a home assistant device based on the location of the one or more identified or potential users;

adjusting an audio output based on the location of the one or more identified or potential users;

receiving a spoken command from the one or more identified or potential users; and

outputting a response to the spoken command.

2. The method of claim 1 , wherein the adjusting the audio output comprises:

adjusting an audio output volume based on the location of the one or more identified or potential users.

3. The method of claim 1 , wherein the analyzing of the field of view in the received video stream comprises:

determining a location of the home assistant device; and

determining a distance between the home assistant device and the one or more identified or potential users.

4. The method of claim 1 , further comprising:

analyzing a gaze of the one or more identified or potential users.

5. The method of claim 4 , comprising:

substituting the analyzed gaze of one of the one or more identified or potential users for a hot word or hot phrase.

6. The method of claim 1 , further comprising:

monitoring for movement of the one or more identified or potential users.

7. The method of claim 6 , wherein the monitoring of the movement of the one or more identified or potential users comprises:

monitoring for lip movement of the one or more identified or potential users;

detecting the lip movement by one of the one or more identified or potential users; and

determining that one user of the one or more identified or potential users spoke a command received by the home assistant device.

8. The method of claim 7 , wherein the command is executed based on permissions associated with the user that spoke the command.

9. The method of claim 1 , wherein there are multiple audio output devices and wherein the adjusting the audio output comprises:

selecting an audio output device of the multiple audio output devices based on the location of the one or more identified or potential users.

10. The method of claim 9 , further comprising:

analyzing another field of view in the received video stream to determine another location for the one or more identified or potential users;

beamforming microphones of the home assistant device based on the other location of the one or more identified or potential users; and

adjusting an audio output based on the other location of the one or more identified or potential users.

11. The method of claim 10 , wherein the adjusting the audio output based on the other location comprises:

selecting an audio output device to follow movement of the one or more identified or potential users.

12. A system for handling spoken commands based on characteristics observed by the system, the system comprising:

a streaming video camera;

a cloud-based integrated video service configured to:

receive a video stream from the streaming video camera;

analyze a field of view in the received video stream to determine a location for one or more identified or potential users;

beamform audio from microphones of a home assistant device based on the location of the one or more identified or potential users;

adjust an audio output based on the location of the one or more identified or potential users;

receive a spoken command from the one or more identified or potential users; and

output a response to the spoken command.

13. The system of claim 12 , wherein the adjustment of the audio output configures the cloud-based integrated video service to:

adjust an audio output volume based on the location of the one or more identified or potential users.

14. The system of claim 12 , wherein the cloud-based integrated video service is configured to:

analyze a gaze of the one or more identified or potential users; and

substitute the analyzed gaze of one of the one or more identified or potential users as a substitute for a hot word or hot phrase.

15. The system of claim 12 ,

the cloud-based integrated video service configured to:

monitor for movement of the one or more identified or potential users.

16. The system of claim 15 , wherein the monitoring of the movement of the one or more identified or potential users configures the cloud-based integrated video service to:

monitor for lip movement of the one or more identified or potential users;

detect the lip movement by one of the one or more identified or potential users; and

determine that one user of the one or more identified or potential users spoke a command received by the home assistant device.

17. The system of claim 16 , wherein the command is executed based on permissions associated with the user that spoke the command.

18. The system of claim 12 , wherein there are multiple audio output devices and wherein the adjustment of the audio output configures the cloud-based integrated video service to:

select an audio output device of the multiple audio output devices based on the location of the one or more identified or potential users.

19. The system of claim 18 , wherein the cloud-based integrated video service is configured to:

analyze another field of view in the received video stream to determine another location for the one or more identified or potential users;

beamform microphones of the home assistant device based on the other location of the one or more identified or potential users; and

adjust an audio output based on the other location of the one or more identified or potential users.

20. The system of claim 19 , wherein the adjustment of the audio output based on the other location configures the cloud-based integrated video service to:

select an audio output device to follow movement of the one or more identified or potential users.