Machine learning for home understanding and notification
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for machine learning for home understanding and notification. In one aspect, a method includes obtaining reference videos from a camera within a home, determining from the reference videos that a particular person routinely leaves the home with a particular object at a particular time of day, determining from a sample video from the camera within the home that the particular person appears to be leaving the home without the particular object at the particular time of day, and in response, providing a notification regarding the particular object.
1. A home monitoring system comprising:
one or more processors; and
at least one computer-readable storage medium couple to the one or more processors having stored thereon instructions which, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
obtaining reference videos from a camera within a home;
obtaining connectivity data from devices connected to a wireless network for the home;
determining from the reference videos and from the connectivity data from the devices that a particular person routinely leaves the home with a particular device at a particular time of day;
determining from a sample video from the camera within the home and connectivity data from the particular device that the particular person appears to be leaving the home without the particular device at the particular time of day; and
in response to determining from the sample video from the camera within the home and the connectivity data from the particular device that the particular person appears to be leaving the home without the particular device at the particular time of day, providing a notification regarding the particular device.
2. The system of claim 1 , further comprising:
receiving, from the particular person, user-provided activity data for the particular device.
3. The system of claim 2 , wherein the user-provided activity data comprises one or more of video data, image data, three-dimensional model data for the particular device.
4. The system of claim 1 , wherein determining from the sample video from the camera within the home that the particular person appears to be leaving the home without the particular device at the particular time of day comprises analyzing the sample video utilizing one or more site-specific models for the particular device and the particular person.
5. The system of claim 1 , wherein providing the notification regarding the particular device comprises providing the notification to a user device of a user.
6. The system of claim 5 , further comprising:
receiving from the user and responsive to the notification, user feedback including user-based labeling, wherein the user-based labeling includes one or more shared labels for the particular device.
7. The system of claim 6 , wherein, the one or more shared labels define a relationship between two or more particular devices.
8. The system of claim 1 , further comprising:
receiving, from a user a natural language query referencing the particular device, the natural language query including one or more terms;
determining that devices appearing in reference videos are similar to a site-specific model of the particular device; and
providing, to the user, a subset of reference videos including the particular device.
9. The system of claim 1 , further comprising training a site-specific human model for the particular person using one or more of soft biometrics and facial recognition software analysis of the reference videos.
10. The system of claim 1 , wherein reference videos comprise one or more of imaging data and audio data.
11. A method comprising:
obtaining, by one or more processors, reference videos from a camera within a home;
obtaining, by the one or more processors, connectivity data from devices connected to a wireless network for the home;
determining, by the one or more processors and from the reference videos and from the connectivity data, that a particular person routinely leaves the home with a particular device at a particular time of day;
determining, by the one or more processors and from a sample video from the camera within the home and from connectivity data from the particular device, that the particular person appears to be leaving the home without the particular device at the particular time of day; and
in response to determining from the sample video from the camera within the home and the connectivity data from the particular device that the particular person appears to be leaving the home without the particular device at the particular time of day, providing a notification regarding the particular device.
12. The method of claim 11 , further comprising:
receiving, from the particular person, user-provided activity data for the particular device.
13. The method of claim 12 , wherein the user-provided activity data comprises one or more of video data, image data, three-dimensional model data for the particular device.
14. The method of claim 11 , wherein determining from the sample video from the camera within the home that the particular person appears to be leaving the home without the particular device at the particular time of day comprises analyzing the sample video utilizing one or more site-specific models for the particular device and the particular person.
15. The method of claim 11 , wherein providing the notification regarding the particular device comprises providing the notification to a user device of a user.
16. The method of claim 15 , further comprising:
receiving from the user and responsive to the notification, user feedback including user-based labeling, wherein the user-based labeling includes one or more shared labels for the particular device.
17. The method of claim 16 , wherein, the one or more shared labels define a relationship between two or more particular devices.
18. The method of claim 11 , further comprising:
receiving, from a user a natural language query referencing the particular device, the natural language query including one or more terms;
determining that devices appearing in reference videos are similar to a site-specific model of the particular device; and
providing, to the user, a subset of reference videos including the particular device.
19. The method of claim 11 , further comprising training a site-specific human model for the particular person using one or more of soft biometrics and facial recognition software analysis of the reference videos.
20. A non-transitory computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:
obtaining reference videos from a camera within a home;
obtaining connectivity data from devices connected to a wireless network for the home;
determining from the reference videos and from the connectivity data from the devices that a particular person routinely leaves the home with a particular device at a particular time of day;
determining from a sample video from the camera within the home and connectivity data from the particular device that the particular person appears to be leaving the home without the particular device at the particular time of day; and
in response to determining from the sample video from the camera within the home and the connectivity data from the particular device that the particular person appears to be leaving the home without the particular device at the particular time of day, providing a notification regarding the particular device.
21. The system of claim 1 , wherein the particular device is a mobile phone of the particular person, and
wherein determining, from the sample video from the camera within the home and the connectivity data from the particular device, that the particular person appears to be leaving the home without the particular device comprises determining, by a wireless receiver located near an exit of the home and connected to the wireless network of the home, that the mobile phone is out of range of the wireless receiver while the particular person is determined to be leaving the home.
22. The system of claim 1 , wherein connectivity data comprises a signal strength and/or time-of-flight measurement between the particular device and a plurality of receivers positioned around the home, and
wherein the connectivity data for the particular device can be utilized to triangulate a position of the particular device.
23. The system of claim 1 , wherein connectivity data comprises a connection strength between two different devices.
24. The system of claim 1 , wherein providing the notification regarding the particular device comprises providing an alert through a device of the devices connected to the wireless network of the home that is not the particular device.