Sparse agent re-identification
Disclosed are systems and methods for tracking and re-identification of agents through sparce and disparate event areas (entry, exit, inventory locations, etc.) within a materials handling facility that includes untracked areas. With the disclosed implementations, only certain areas of a materials handling facility need cameras while other areas may be devoid of cameras. With the disclosed implementations, re-identification of an agent may only occur when the agent enters an authentication event area, such as an exit event area and/or performs a defined action, such as an item pick from an inventory location or an item place to an inventory location. In other instances, while an agent may be tracked while the agent is located within an event area, the tracklet generated for that agent while in that event area may remain unlinked or otherwise disconnected from any other tracklet until re-identification of the agent is performed (e.g., when the agent exits the materials handling facility).
1 . A computer-implemented method, comprising:
detecting a first agent in a first event area of a plurality of event areas within a materials handling facility, wherein:
the materials handling facility includes the plurality of event areas and at least one untracked area that separates each of the plurality of event areas; and
each event area of the plurality of event areas includes one or more cameras;
generating, based at least in part on first image data from at least a first camera of the first event area, a first sub-session tracklet indicative of the first agent while the first agent is in the first event area;
including the first sub-session tracklet in an unlinked sub-session tracklet list that indicates a plurality of tracklets indicative of agents detected within the materials handling facility;
detecting a second agent in a second event area of the plurality of event areas within the materials handling facility;
generating, based at least in part on second image data from at least a second camera of the second event area, a second sub-session tracklet indicative of the second agent while the second agent is in the second event area;
including the second sub-session tracklet in the unlinked sub-session tracklet list;
detecting a third agent in a third event area of the materials handling facility that corresponds to an exit of the materials handling facility;
generating, based at least in part on third image data from at least a third camera of the third event area, an exit tracklet indicative of the third agent while the third agent is in the third event area;
defining a plurality of nodes of a node graph based at least in part on the first sub-session tracklet, the second sub-session tracklet, and the exit tracklet;
determining, based at least in part on the nodes of the node graph, an edge of the node graph linking a first node representative of the first sub-session tracklet and a second node representative of the exit tracklet; and
associating the first sub-session tracklet and the exit tracklet with a first agent session of the first agent.
2 . The computer-implemented method of claim 1 , further comprising:
determining, based at least in part on the nodes of the node graph, that a second edge does not exist between a third node representative of the second sub-session tracklet and the second node and that the second sub-session tracklet does not correspond to the first agent.
3 . The computer-implemented method of claim 1 , further comprising:
determining that an action of an item pick of an item is associated with the first sub-session tracklet;
updating an agent account of the first agent to include an item identifier of the item; and
charging the first agent a fee for the item.
4 . The computer-implemented method of claim 1 , further comprising:
comparing the exit tracklet with each of a plurality of entry tracklets to determine an entry tracklet of the plurality of entry tracklets that corresponds to the first agent, each of the plurality of entry tracklets indicative of an agent while the agent is within an entry event area of the materials handling facility;
determining, based at least in part on the entry tracklet, an entry time during which the first agent was in the entry event area of the materials handling facility;
determining, based at least in part on the exit tracklet, an exit time at which the first agent was within the third event area of the materials handling facility; and
defining the plurality of nodes of the node graph as nodes of unlinked sub-session tracklets corresponding to a period of time between the entry time and the exit time.
5 . A system, comprising:
one or more processors; and
a memory storing program instructions that, when executed by the one or more processors, cause the one or more processors to at least:
determine that an agent at a first event area of a plurality of event areas within a materials handling facility is to be identified, wherein:
the materials handling facility includes the plurality of event areas and at least one untracked area that separates each of the plurality of event areas; and
each event area of the plurality of event areas includes one or more cameras;
define a node graph that includes a plurality of nodes, each node of the plurality of nodes representative of a plurality of unlinked sub-session tracklets, wherein each sub-session tracklet includes at least a feature embedding representative of the agent and generated based on one or more images of the agent generated while the agent is in an event area of the plurality of event areas;
define an edge between a first node of the node graph corresponding to a first sub-session tracklet of the first event area and a second node of the node graph corresponding to a second sub-session tracklet of a second event area; and
associate the first sub-session tracklet with the second sub-session tracklet as corresponding to the agent.
6 . The system of claim 5 , wherein:
the first event area is an exit event area; and
the program instructions that, when executed by the one or more processors, further cause the one or more processors to at least:
determine an entry time corresponding to the first sub-session tracklet;
determine an exit time corresponding to the second sub-session tracklet;
determine a plurality of candidate sub-session tracklets that were each generated between the entry time and the exit time, each candidate sub-session tracklet of the plurality of candidate sub-session tracklets indicative of an agent positioned within one of the plurality of event areas; and
define nodes of the node graph for each of the plurality of candidate sub-session tracklets.
7 . The system of claim 5 , wherein the program instructions that, when executed by the one or more processors, further cause the one or more processors to at least:
determine that an action of an item pick of an item from an inventory location is associated with the second sub-session tracklet; and
associate an item identifier of the item with the agent.
8 . The system of claim 5 , wherein the program instructions that define the edge, further include program instructions that, when executed by the one or more processors, further cause the one or more processors to at least:
define the edge based on one or more of:
a duration of time between a first time corresponding to the first node and a second time corresponding to the second node;
a spatial distance between a first position corresponding to the first node and a second position corresponding to the second node; or
a feature distance between a first feature embedding corresponding to the first node and a second feature embedding corresponding to the second node.
9 . The system of claim 5 , wherein:
the second event area is an inventory area; and
wherein the program instructions that, when executed by the one or more processors, further cause the one or more processors to at least:
determine that an item pick of an item from an inventory location within the second event area has been performed by the agent;
associate an item identifier of the item with the second sub-session tracklet; and
in response to defining the edge between the first node and the second node:
generate an agent session for the agent; and
associate at least one of the item identifier, the first sub-session tracklet, or the second sub-session tracklet with the agent session.
10 . The system of claim 5 , wherein the program instructions that, when executed by the one or more processors, further cause the one or more processors to at least:
detect the agent at the first event area;
obtain one or more images of the agent while the agent is at the first event area;
generate, based at least in part on the one or more images, a first feature representative of one or more embedding vectors indicative of the agent; and
generate the first sub-session tracklet, wherein the first sub-session tracklet includes at least the first feature embedding and an indication of the first event area.
11 . The system of claim 5 , wherein the program instructions that, when executed by the one or more processors, further cause the one or more processors to at least:
determine an agent account corresponding to at least one of the first sub-session tracklet or the second sub-session tracklet; and
associate at least one of the first sub-session tracklet, the second sub-session tracklet, or the agent with the agent account.
12 . The system of claim 5 , wherein the program instructions that, when executed by the one or more processors, further cause the one or more processors to at least:
remove the first sub-session tracklet and the second sub-session tracklet from the plurality of unlinked sub-session tracklets.
13 . The system of claim 5 , wherein the program instructions that, when executed by the one or more processors, further cause the one or more processors to at least:
determine a plurality of candidate sub-session tracklets;
define a second edge between the second node corresponding to the sub-session tracklet and a third node corresponding to a third sub-session tracklet of the plurality of candidate sub-session tracklets; and
associate the third sub-session tracklet with the agent.
14 . The system of claim 5 , wherein the program instructions that, when executed by the one or more processors, further cause the one or more processors to at least:
reduce, based at least in part on at least one of the first sub-session tracklet or the second sub-session tracklet, a plurality of candidate sub-session tracklets to generate a reduced plurality of candidate sub-session tracklets that include less than all of the candidate sub-session tracklets; and
define the plurality of nodes as corresponding to the reduced plurality of candidate sub-session tracklets.
15 . The system of claim 14 , wherein the program instructions that cause the one or more processors to reduce the plurality of candidate sub-session tracklets further include instructions that, when executed by the one or more processors, further cause the one or more processors to at least:
determine, based at least in part on the first sub-session tracklet, at least a third candidate sub-session tracklet of the plurality of candidate sub-session tracklets that could not correspond to the agent; and
exclude the third candidate sub-session tracklet from the reduced plurality of candidate sub-session tracklets.
16 . A computer-implemented method, comprising:
generating, for each of a plurality of agents located within a materials handling facility, one or more sub-session tracklets indicative of the agent while the agent is at an event area within the materials handling facility, wherein:
the materials handling facility includes a plurality of event areas and at least one untracked area that separates each of the plurality of event areas; and
each event area of the plurality of event areas includes one or more cameras;
including each of the sub-session tracklets in an unlinked sub-session tracklet list;
determining an agent at an exit event area of the materials handling facility;
generating, for the agent, an exit tracklet indicative of the agent at the exit event area;
defining, based at least in part on the unlinked sub-session tracklet list, a plurality of nodes of a node graph, each node of the plurality of nodes corresponding to a sub-session tracklet indicated on the unlinked sub-session tracklet list;
defining an edge between a first node of the node graph corresponding to a first sub-session tracklet and a second node of the node graph corresponding to the exit tracklet; and
associating the first sub-session tracklet with an agent session generated for the agent.
17 . The computer-implemented method of claim 16 , wherein defining the edge further includes:
determining the edge based at least in part on one or more of a time and spatial similarity between the exit tracklet and the first sub-session tracklet or a feature embedding similarity between a first feature embedding of the first sub-session tracklet and a second feature embedding of the exit tracklet.
18 . The computer-implemented method of claim 16 , wherein generating further includes:
determining at least one candidate sub-session tracklet to exclude from the plurality of sub-session tracklets based at least in part on a time associated with the first sub-session tracklet or an event area of the first sub-session tracklet.
19 . The computer-implemented method of claim 16 , wherein each sub-session tracklet includes, at least:
an event area indication corresponding to an event area at which the sub-session tracklet was generated;
a feature embedding indicative of the agent and generated based at least in part on a plurality of embedding vectors generated for the agent; and
a time corresponding to the sub-session tracklet.
20 . The computer-implemented method of claim 16 , further comprising:
determining, for each node of the plurality of nodes of the node graph, one entry edge connecting the node to another node of the node graph and one exit edge connecting the node to another node of the node graph.