Curation of custom workflows using multiple cameras, with AI trained from the workflow
A multi-layer technology stack includes a sensor layer including image sensors, a device layer, and a cloud layer, with interfaces between the layers. A method to curate different custom workflows for multiple applications include the following. Requirements for custom sets of data packages for the applications is received. The custom set of data packages include sensor data packages (e.g., SceneData) and contextual metadata packages that contextualize the sensor data packages (e.g., SceneMarks). Based on the received requirements and capabilities of components in the technology stack, the custom workflow for that application is deployed. This includes a selection, configuration and linking of components from the technology stack. The custom workflow is implemented in the components of the technology stack by transmitting workflow control packages directly and/or indirectly via the interfaces to the different layers.
1 . A workflow method for a plurality of non-human technological entities to develop a situational awareness of a space, the entities including cameras that view the monitored space, wherein the workflow method comprises:
the cameras capturing images of the monitored space;
artificial intelligence and/or machine learning (AI/ML) entities detecting events from the captured images;
generating SceneMarks with attributes that are descriptive of the detected events;
a first AI/ML entity performing analysis of the SceneMarks; and adding information to the attributes of at least one of the SceneMarks based on said analysis, and/or detecting an event based on said analysis and generating a new SceneMark for said detected event;
wherein the workflow method contextualizes the captured images from the SceneMarks to provide awareness of situations in the monitored space; and the first AI/ML entity is trained based on Scenemarks generated by execution of the workflow method.
2 . The workflow method of claim 1 wherein the first AI/ML entity is trained to predict a generation of SceneMarks.
3 . The workflow method of claim 2 wherein the first AI/ML entity is trained to predict the generation of SceneMarks, based on received SceneMarks.
4 . The workflow method of claim 1 wherein the first AI/ML entity is trained to predict a generation of attributes for SceneMarks.
5 . The workflow method of claim 1 further comprising:
grouping SceneMarks into Scenes, wherein the first AI/ML entity is trained to group SceneMarks into Scenes.
6 . The workflow method of claim 1 further comprising:
automatically triggering action based on the SceneMarks, wherein the first AI/ML entity is trained to learn which actions are triggered by which sequences of SceneMarks.
7 . The workflow method of claim 1 wherein the first AI/ML entity is trained to determine when a person or object identified in two different SceneMarks are the same person or object.
8 . The workflow method of claim 7 wherein the first AI/ML entity determines that the person or object in the two different SceneMarks are the same person or object, based on attributes of the two different SceneMarks.
9 . The workflow method of claim 7 wherein the first AI/ML entity determines that the person or object in the two different SceneMarks are the same person or object, based on timestamps of the two different SceneMarks and a known proximity of cameras capturing the images that generated the two different SceneMarks.
10 . The workflow method of claim 1 wherein the first AI/ML entity is a generative pre-trained transformer AI entity.
11 . The workflow method of claim 1 wherein the first AI/ML entity is trained based on labeled SceneMarks.
12 . The workflow method of claim 11 wherein the labeled SceneMarks include labels that group SceneMarks into Scenes.
13 . The workflow method of claim 11 wherein the labeled SceneMarks include labels that are descriptive of the situation in the monitored space.
14 . The workflow method of claim 1 wherein the first AI/ML entity is trained on SceneMark tokens.
15 . The workflow method of claim 1 wherein the situation occurs across multiple cameras viewing the space, and providing awareness of the situation is based on SceneMarks generated from images captured by the multiple cameras and is also based on a known proximity of the cameras.
16 . The workflow method of claim 1 further comprising:
dynamically adjusting the entities in the workflow and/or the analysis performed by the entities, based on the SceneMarks.
17 . The workflow method of claim 1 wherein the entities include other non-camera sensors, and the workflow method further comprises:
the non-camera sensors capture sensor data for the monitored space; and
the AI/ML entities detecting events from the captured images and the captured sensor data.
18 . The workflow method of claim 1 further comprising:
receiving descriptions of capabilities of different entities; and
transmitting workflow control packages to the entities to implement the workflow method on the entities.
19 . A system comprising:
a plurality of non-human technological entities, the entities including cameras that view a space and further including artificial intelligence and/or machine learning (AI/ML) entities; and
a service that configures the entities to implement a workflow to develop a situational awareness of the space, wherein the workflow includes:
capturing images of the space;
detecting events from the captured images;
generating SceneMarks with attributes that are descriptive of the detected events;
at least one AI/ML entity performing analysis of the SceneMarks; and adding information to the attributes of at least one of the SceneMarks based on said analysis, and/or detecting an event based on said analysis and generating a new SceneMark for said detected event;
wherein the workflow contextualizes the captured images from the SceneMarks to provide awareness of situations in the space; and a first of the AI/ML entities is trained based on Scenemarks generated by execution of the workflow.
20 . The system of claim 19 wherein the first AI/ML entity is trained to determine when a person or object identified in two different SceneMarks are the same person or object.