Determining associations between objects and persons using machine learning models
In various examples, sensor data—such as masked sensor data—may be used as input to a machine learning model to determine a confidence for object to person associations. The masked sensor data may focus the machine learning model on particular regions of the image that correspond to persons, objects, or some combination thereof. In some embodiments, coordinates corresponding to persons, objects, or combinations thereof, in addition to area ratios between various regions of the image corresponding to the persons, objects, or combinations thereof, may be used to further aid the machine learning model in focusing on important regions of the image for determining the object to person associations.
1. A method comprising:
determining, from an image, one or more persons associated with an object;
for each person of the one or more persons, performing operations comprising:
determining an overlap region of the image corresponding to an overlap in the image between an object region of the object and a person region of the person;
applying a mask to portions of the image not included in the overlap region to generate a masked image;
applying data representative of the masked image to a neural network trained to predict confidences for associations between objects and persons; and
computing, using the neural network and based at least in part on the data, a confidence for an association between the object and the person; and
based on the confidence for each person of the one or more persons, associating the object to the person of the one or more persons having a highest associated confidence.
2. The method of claim 1 , wherein the determining the one or more persons associated with an object further comprises:
generating an association region for the object; and
determining that the one or more persons or one or more bounding shapes corresponding to the one or more persons at least partially overlap with the association region.
3. The method of claim 2 , wherein the association region is defined by dimensions extending from a centroid of the object or a bounding shape corresponding to the object, and the association region is larger than the object or the bounding shape corresponding to the object.
4. The method of claim 1 , wherein the person region includes a first portion of the image corresponding to a person bounding shape of the person, the object region includes a second portion of the image corresponding to an object bounding shape of the object, and the overlap region includes a third portion of the image corresponding to an overlap of the person bounding shape with the object bounding shape.
5. The method of claim 1 , wherein the image is a most recent image in a sequence of images, and temporal smoothing is used to weight the confidence in view of prior confidences predicted for associations between the person and the object to generate a final confidence, further wherein the final confidence is used for the associating the object to the person.