Scene-based sensor networks
An approach to sensor data is based on scenes. One aspect concerns a computer-implemented method for specifying and obtaining a variety of sensor data and processed sensor data related to a scene. The method incorporates a Scene-based API that uses SceneModes and SceneData. An application requesting sensor data communicates a SceneMode to a group of one or more sensor devices and/or sensor modules via the Scene-based API. The SceneMode determines the SceneData to be captured or provided by the sensor group, which typically includes different types of sensor data related to the Scene and also further processed or analyzed data. The application receives the SceneData from the sensor group via the Scene-based API, with the SceneData organized into SceneShots which are samples of the Scene.
1. A system for multiple applications to specify and obtain a higher level understanding of scenes based on sensor data captured by a group of one or more sensor devices, the system comprising:
a plurality of applications;
a sensor-side technology stack comprising a group of one or more sensor devices that capture sensor data and processor capability for artificial intelligence and/or machine learning; and
an application programming interface (API) for communication between any of the applications and the sensor-side technology stack;
wherein the applications communicate SceneModes to the sensor-side technology stack via the API; and for each application:
the SceneMode defines SceneData that is related to the scene of interest for that application and that is to be generated by the sensor-side technology stack;
the SceneData is produced by the sensor-side technology stack based on a workflow for a capture, processing and analysis of sensor data, the SceneData including image data and also including data produced by artificial intelligence and/or machine learning applied to the image data; and
the SceneMode does not specify a particular workflow for producing the SceneData, including not specifying sensor-level settings for the capture of the sensor data used to produce the SceneData; and
the sensor-side technology stack determines, based on the SceneMode, the workflow for producing the SceneData, the workflow including sensor-level settings for the sensor devices that capture the sensor data from the scene used to produce the SceneData; and
the SceneData is returned to the application via the API, the returned SceneData organized into SceneShots that are samples of the scene.
2. The system of claim 1 wherein the API, the SceneMode and a data structure for the SceneData are defined in one or more standard(s).
3. The system of claim 2 wherein the standard(s) define multiple predefined SceneModes.
4. The system of claim 3 wherein the predefined SceneModes include at least one of a Home Surveillance, Baby Monitoring, and Large Area Surveillance SceneModes.
5. The system of claim 2 wherein the standard(s) support user-defined SceneModes.
6. The system of claim 1 wherein the SceneMode provides context for the scene of interest.
7. The system of claim 1 wherein the artificial intelligence and/or machine learning is used in the workflow to derive events from the sensor data.
8. The system of claim 1 wherein the processor capability for the artificial intelligence and/or machine learning is located outside the sensor devices.
9. The system of claim 1 wherein the processor capability for the artificial intelligence and/or machine learning is located in the cloud.
10. The system of claim 1 wherein the workflow for the artificial intelligence and/or machine learning does not occur in real-time with respect to capture of the sensor data.
11. The system of claim 1 wherein the artificial intelligence and/or machine learning is used in the workflow for object recognition.
12. The system of claim 1 wherein the artificial intelligence and/or machine learning is used in the workflow for recognition of humans.
13. The system of claim 1 wherein the artificial intelligence and/or machine learning is used in the workflow for face recognition.
14. The system of claim 1 wherein the artificial intelligence and/or machine learning is used in the workflow for emotion recognition.
15. The system of claim 1 wherein the artificial intelligence and/or machine learning is used in the workflow for gesture recognition.
16. The system of claim 1 wherein the SceneMode defines the SceneData to include at least one of object recognition, recognition of humans, face recognition and emotion recognition.
17. A computer-implemented method for an application to specify and obtain a higher level understanding of a scene, the method comprising:
communicating a SceneMode from an application to a sensor-side technology stack via an application programming interface (API), the sensor-side technology stack comprising a group of one or more sensor devices and processor capability for artificial intelligence and/or machine learning, wherein:
the SceneMode defines SceneData that is related to the scene and that is to be returned to the application;
the SceneData is produced by the sensor-side technology stack based on a workflow for a capture, processing and analysis of sensor data, the SceneData including image data and also including data produced by artificial intelligence and/or machine learning applied to the image data; and
the SceneMode does not specify a particular workflow for producing the SceneData, including not specifying sensor-level settings for the capture of the sensor data used to produce the SceneData; and
receiving the SceneData from the sensor-side technology stack via the API, wherein:
the sensor-side technology stack determines, based on the SceneMode, the workflow for producing the SceneData, the workflow including sensor-level settings for sensor devices that capture the sensor data from the scene used to produce the SceneData; and
the SceneData is organized into SceneShots that are samples of the scene.
18. The computer-implemented method of claim 17 wherein the API, the SceneMode and a data structure for the SceneData are defined in one or more standard(s).
19. A non-transitory computer-readable storage medium storing executable computer program instructions for an application to specify and obtain a higher level understanding of a scene, the instructions executable by a computer system and causing the computer system to perform a method comprising:
communicating a SceneMode from an application to a sensor-side technology stack via an application programming interface (API), the sensor-side technology stack comprising a group of one or more sensor devices and processor capability for artificial intelligence and/or machine learning, wherein:
the SceneMode defines SceneData that is related to the scene and that is to be returned to the application;
the SceneData is produced by the sensor-side technology stack based on a workflow for a capture, processing and analysis of sensor data, the SceneData including image data and also including data produced by artificial intelligence and/or machine learning applied to the image data; and
the SceneMode does not specify a particular workflow for producing the SceneData, including not specifying sensor-level settings for the capture of the sensor data used to produce the SceneData; and
receiving the SceneData from the sensor-side technology stack via the API, wherein:
the sensor-side technology stack determines, based on the SceneMode, the workflow for producing the SceneData, the workflow including sensor-level settings for sensor devices that capture the sensor data from the scene used to produce the SceneData; and
the SceneData is organized into SceneShots that are samples of the scene.
20. The non-transitory computer-readable storage medium of claim 19 wherein the API, the SceneMode and a data structure for the SceneData are defined in one or more standard(s).