Automated recalibration of sensors for monitoring an area of real space
Automated techniques provide for recalibrating cameras configured for monitoring an area of real space. The method includes first processing one or more selected images selected from a plurality of sequences of images received from a plurality of cameras calibrated using a set of calibration images that were used to calibrate the cameras previously. Transformation information between the selected images and the set of calibration images is obtained based on one or more features extracted from the selected images using a trained neural network classifier corresponding to points located at relatively immobile structures. The features extracted from the selected images match features in the set of calibration images. Camera calibrations can be updated when transform information between features matched meets or exceeds a threshold.
1 . A method for recalibrating cameras in a real space for tracking puts and takes of items by subjects, the method including:
first processing one or more images selected from a plurality of sequences of images received from a plurality of cameras, in which selected images in the plurality of sequences of images have respective fields of view in the real space, to:
obtain, based upon one or more features (i) corresponding to points located at displays or relatively immobile structures extracted from the selected images using a trained neural network classifier and (ii) matched with features from a set of calibration images, transformation information between the selected images and the set of calibration images; and
update calibration of a camera with the transformation information whenever the transformation information for the camera meets or exceeds a first threshold.
2 . The method of claim 1 , wherein the trained neural network classifier has been trained using a synthetic shapes dataset created by a second neural network.
3 . The method of claim 2 , wherein the second neural network has been trained using a plurality of synthetic shapes having no ambiguity in interest point locations, wherein the synthetic shapes comprise three-dimensional models created automatically, and a plurality of viewpoints generated for the three-dimensional models for matching features; and
wherein three-dimensional models are finetuned by data collected from like real space environments having matching features annotated between different images captured from different viewpoints.
4 . The method of claim 1 , wherein feature descriptors corresponding to points located at displays or structures that remain substantially immobile are extracted using a scale invariant feature transform.
5 . The method of claim 1 , further including second processing sequences of images of the plurality of sequences of images, to track puts and takes of items by subjects within respective fields of view in the real space; and
wherein first processing and second processing occur substantially contemporaneously, thereby enabling cameras to be calibrated without clearing subjects from the real space or interrupting tracking puts and takes of items by subjects.
6 . The method of claim 5 , wherein second processing at least one sequence of images of the plurality of sequences of images to track a take or put event, further includes, detecting the take or put event using a trained neural network.
7 . The method of claim 6 , wherein second processing to track puts and takes of items by subjects includes tracking inventory caches involved in an exchange that move over time having locations in three dimensions.
8 . The method of claim 7 , wherein locations of the inventory caches include locations corresponding to hands of identified subjects, and wherein processing the plurality of sequences of images includes using an image recognition engine to detect an inventory item in hands of a subject identified in the exchange as detected.
9 . The method of claim 5 , wherein second processing at least one sequence of images of the plurality of sequences of images to track a take or put event, further including, detecting the take or put event using a trained random forest.
10 . The method of claim 1 , further including determining whether the transformation information is valid and storing the transformation information in a database if the transformation information is determined to be valid.
11 . The method of claim 1 , wherein the transformation information is determined relative to an origin point that is selected as a reference point for calibration.
12 . The method of claim 1 , wherein updating calibration of a camera with the transformation information further includes updating calibration of a camera with the transformation information whenever the transformation information obtained for the camera meets or exceeds a second threshold of at least a 1 centimeter change in camera translation value.
13 . The method of claim 1 , wherein updating calibration of a camera with the transformation information further includes updating calibration of a camera with the transformation information whenever the transformation information obtained for the camera meets or exceeds a third threshold of at least a 1 degree change in camera rotation value.
14 . A non-transitory computer readable storage medium impressed with computer program instructions to recalibrating cameras in a real space for tracking puts and takes of items by subjects between inventory caches which can act as at least one of sources and sinks of inventory items in exchanges of inventory items, which computer program instructions when executed implement a method according to claim 1 .
15 . A system including one or more processors and memory accessible by the processors, the memory loaded with computer instructions recalibrating cameras in a real space for tracking puts and takes of items by subjects between inventory caches which can act as at least one of sources and sinks of inventory items in exchanges of inventory items, which computer instructions, when executed on the processors, implement actions comprising:
first processing one or more images selected from a plurality of sequences of images received from a plurality of cameras, in which selected images in the plurality of sequences of images have respective fields of view in the real space, to:
obtain, based upon one or more features (i) corresponding to points located at displays or relatively immobile structures extracted from the selected images using a trained neural network classifier and (ii) matched with features from a set of calibration images, transformation information between the selected images and the set of calibration images; and
update calibration of a camera with the transformation information whenever the transformation information for the camera meets or exceeds a first threshold.
16 . The system of claim 15 , wherein the trained neural network classifier has been trained using a synthetic shapes dataset created by a second neural network.
17 . The system of claim 16 , wherein the second neural network has been trained using a plurality of synthetic shapes having no ambiguity in interest point locations, wherein the synthetic shapes comprise three-dimensional models created automatically, and a plurality of viewpoints generated for the three-dimensional models for matching features; and
wherein three-dimensional models are finetuned by data collected from like real space environments having matching features annotated between different images captured from different viewpoints.
18 . A method for recalibrating cameras in a real space for tracking puts and takes of items by subjects, the method including:
first processing one or more images selected from a plurality of sequences of images received from a plurality of cameras, in which selected images in the plurality of sequences of images have respective fields of view in the real space, to:
obtain, based upon one or more features (i) corresponding to points located at displays or relatively immobile structures extracted from the selected images using a trained neural network classifier and (ii) matched with features from a set of calibration images, transformation information between the selected images and the set of calibration images; and
update calibration of a camera with the transformation information whenever the transformation information for the camera meets or exceeds a threshold; and
second processing sequences of images of the plurality of sequences of images, to track puts and takes of items by subjects within respective fields of view in the real space; and
wherein first processing and second processing occur substantially contemporaneously, thereby enabling cameras to be calibrated without clearing subjects from the real space or interrupting tracking puts and takes of items by subjects.
19 . A system including one or more processors and memory accessible by the processors, the memory loaded with computer instructions recalibrating cameras in a real space for tracking puts and takes of items by subjects between inventory caches which can act as at least one of sources and sinks of inventory items in exchanges of inventory items, which computer instructions, when executed on the processors, implement a method according to claim 18 .
20 . A non-transitory computer readable storage medium impressed with computer program instructions to recalibrating cameras in a real space for tracking puts and takes of items by subjects between inventory caches which can act as at least one of sources and sinks of inventory items in exchanges of inventory items, which computer program instructions when executed implement a method according to claim 18 .