Efficient event-driven object detection at the forklifts at the edge in warehouse environments
An event driven detection model is disclosed. A model operates at a node to identify relevant video data from video streams generated by cameras. Video data that is not relevant is discarded. An objectness score is generated for the relevant video data. The objectness score and position data from position sensors is used to infer an event. When an event is inferred by the model, a decision may be made and performed.
1 . A method, comprising:
receiving position data at a first model operating on a node, wherein the first model is trained using a set of historical data that includes positional data and video data, wherein the node includes sensors configured to generate the position data at the node, the position data including time series data, wherein the position data determine a position of the node in an environment, a direction of node movement in the environment, an anticipated trajectory of the node, and a velocity of the node;
generating, by an object model, a set of cues from video data generated at the node or in the environment, wherein the set of cues includes information associated with the video data including one or more of color contrast, edge density, superpixel straddling, and number of edges, or combinations thereof;
determining, by the object model, an objectness score for the video data;
selecting first video frames from the video data that have objectness scores greater than a threshold objectness score and discarding second video frames from the video data that have objectness scores lower than the threshold objectness score;
generating an event, by a first model, based on a most recent position data and first video frames that correlate to the most recent position data;
providing the event to a pipeline;
making a decision by the pipeline based on the event generated by the first model;
performing the decision at the node and auditing the event based on the first video frames.
2 . The method of claim 1 , further comprising determining the objectness score using multiple models including the object model, wherein at least one of the multiple models is configured to detect objects in the video data.
3 . The method of claim 1 , wherein the first model uses a single collection of the position data when generating the event, wherein the single collection of the position data corresponds in time the first video frames.
4 . The method of claim 1 , further comprising deploying the object model and the first model to the multiple nodes in the environment including the node.
5 . The method of claim 1 , further comprising:
storing the video data whose objectness score is above the threshold objectness score in a video data sensor database at a central node;
storing the position data generated at the node in a position sensor database at the central node; and
storing the event in an event database at the central node.
6 . The method of claim 5 , further comprising training the first model, prior to deployment, using video data in the video sensor database and the position data in the position sensor database at the central node.
7 . The method of claim 1 , wherein the object model and/or the first model operate at the node or at a central node.
8 . The method of claim 1 , wherein the decision includes generating an alarm.
9 . The method of claim 1 , wherein the object model is configured to generate an object indicator, wherein the position data includes position data from multiple objects.
10 . The method of claim 9 , further comprising training the first model with labels associated with events in an event database at a central node.
11 . A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising:
receiving position data at a first model operating on a node, wherein the first model is trained using a set of historical data that includes positional data and video data, wherein the node includes sensors configured to generate the position data at the node, the position data including time series data, wherein the position data determine a position of the node in an environment, a direction of node movement in the environment, an anticipated trajectory of the node, and a velocity of the node;
generating, by an object model, a set of cues from video data generated at the node or in the environment, wherein the set of cues includes information associated with the video data including one or more of color contrast, edge density, superpixel straddling, and number of edges, or combinations thereof;
determining, by the object model, an objectness score for the video data;
selecting first video frames from the video data that have objectness scores greater than a threshold objectness score and discarding second video frames from the video data that have objectness scores lower than the threshold objectness score;
generating an event, by a first model, based on a most recent position data and first video frames that correlate to the most recent position data;
providing the event to a pipeline;
making a decision by the pipeline based on the event generated by the first model;
performing the decision at the node and auditing the event based on the first video frames.
12 . The non-transitory storage medium of claim 11 , further comprising determining the objectness score using multiple models including the object model, wherein at least one of the multiple models is configured to detect objects in the video data.
13 . The non-transitory storage medium of claim 11 , wherein the first model uses a single collection of the position data when generating the event, wherein the single collection of the position data corresponds in time the first video frames.
14 . The non-transitory storage medium of claim 11 , further comprising deploying the object model and the first model to the multiple nodes in the environment including the node.
15 . The non-transitory storage medium of claim 11 , further comprising:
storing the video data whose objectness score is above the threshold objectness score in a video data sensor database at a central node;
storing the position data generated at the node in a position sensor database at the central node; and
storing the event in an event database at the central node.
16 . The non-transitory storage medium of claim 15 , further comprising training the first model, prior to deployment, using video data in the video sensor database and the position data in the position sensor database at the central node.
17 . The non-transitory storage medium of claim 11 , wherein the object model and/or the first model operate at the node or at a central node.
18 . The non-transitory storage medium of claim 11 , wherein the decision includes generating an alarm.
19 . The non-transitory storage medium of claim 11 , wherein the object model is configured to generate an object indicator, wherein the position data includes position data from multiple objects.
20 . The non-transitory storage medium of claim 19 , further comprising training the first model with labels associated with events in an event database at a central node.