Localization processing service and observed scene reconstruction service
Systems, methods, and computer-readable media for providing a localization processing service for enabling localization of a navigation network-restricted subsystem and for providing an observed scene reconstruction service are provided.
1. A method of localizing an object in an environment using an observing subsystem comprising an image sensor component, a memory component, and a processing module communicatively coupled to the image sensor component and the memory component, the method comprising:
storing, with the memory component, a map feature database comprising a plurality of map feature entries, wherein:
each map feature entry of the plurality of map feature entries is respectively associated with a rendered map image of a plurality of rendered map images rendered from a georeferenced three-dimensional map; and
each rendered map image of the plurality of rendered map images is associated with a respective map location and a respective map orientation;
capturing, at a moment in time with the image sensor component, an observing image of the object;
determining, with the processing module, an observing orientation of the image sensor component at the moment in time;
determining, with the processing module, an observing location of the image sensor component at the moment in time;
defining, with the processing module, a similar image set, wherein:
the similar image set comprises the captured observing image and a particular rendered map image of the plurality of rendered map images; and
the defining comprises determining that:
the observing orientation of the captured observing image satisfies an orientation similarity comparison with the map orientation of the particular rendered map image; and
the observing location of the captured observing image satisfies a location similarity comparison with the map location of the particular rendered map image;
extracting, with a feature extractor model of the processing module, the following:
an image feature map from the captured observing image of the similar image set; and
a map feature map from the particular rendered map image of the similar image set;
performing, with a correlation module of the processing module, a tensor correlation between the extracted image feature map and the extracted map feature map to determine the position of the object on the particular rendered map image; and
based on the determined position of the object on the particular rendered map image, determining, with a raycasting module of the processing module, georeferenced three-dimensional coordinates of the object in the environment.
2. The method of claim 1 , wherein the particular rendered map image has a lower resolution than the captured observing image.
3. The method of claim 1 , wherein the feature extractor model comprises a convolutional neural network.
4. The method of claim 1 , wherein the determined position of the object on the particular rendered map image is indicative of the two-dimensional position of the object on the particular rendered map image.
5. The method of claim 1 , further comprising analyzing, with the processing module, the captured observing image to determine that the object is a pre-defined object of interest.
6. The method of claim 5 , wherein the extracting is in response to the determination that the object is a predefined object of interest.
7. The method of claim 1 , wherein the observing subsystem is navigation network-restricted throughout the duration between the capturing and the determining georeferenced three-dimensional coordinates of the object in the environment.
8. The method of claim 1 , further comprising reconstructing a scene comprising the object in the environment using the observing subsystem and a reconstruction subsystem that is remote from the observing subsystem, wherein:
the observing subsystem further comprises a communication component communicatively coupled to the processing module;
the captured observing image comprises an image of the scene comprising the object; and
the reconstructing comprises:
based on the captured observing image, determining, with the processing module, the following attributes of the scene:
a state-based attribute comprising the determined georeferenced three-dimensional coordinates of the object in the environment; and
a categorical attribute of the object; and
transmitting, with the communication component from the observing subsystem to the reconstruction subsystem, the determined attributes of the scene.
9. The method of claim 8 , wherein no pixel data of the scene is transmitted from the observing subsystem to the reconstruction subsystem.
10. The method of claim 1 , further comprising reconstructing a scene comprising the object in the environment using the observing subsystem and a reconstruction subsystem that is remote from the observing subsystem, wherein:
the reconstruction subsystem comprises a reconstruction memory component, a reconstruction communication component, and a reconstruction processing module communicatively coupled to the reconstruction memory component and the reconstruction communication component;
the captured observing image comprises an image of the scene comprising the object; and
the reconstructing comprises:
receiving, with the reconstruction communication component at the reconstruction subsystem from the observing subsystem, the following attributes of the scene:
a state-based attribute comprising the determined georeferenced three-dimensional coordinates of the object in the environment; and
a categorical attribute of the object;
identifying, with the reconstruction processing module from a plurality of three-dimensional models stored in the reconstruction memory component, a particular three-dimensional model associated with the categorical attribute of the object; and
rendering, with the reconstruction processing module, a visualization of the scene by positioning the particular three-dimensional model in a reconstruction georeferenced three-dimensional map at the georeferenced three-dimensional coordinates of the state-based attribute.
11. The method of claim 10 , wherein no pixel data of the scene is transmitted from the observing subsystem to the reconstruction subsystem.
12. The method of claim 1 , wherein the image sensor component comprises at least one of an infrared sensor or a thermal sensor.
13. A method of reconstructing a scene comprising an object in an environment using a reconstruction subsystem and an observing subsystem that is remote from the reconstruction subsystem, the observing subsystem comprising an image sensor component, a communication component, and a processing module communicatively coupled to the image sensor component and the communication component, the method comprising:
capturing, at a moment in time with the image sensor component, an observing image of the scene comprising the object;
based on the captured observing image, determining, with the processing module, the following attributes of the scene:
a state-based attribute comprising georeferenced three-dimensional coordinates of the object in the environment with a raycasting module of the processing module; and
a categorical attribute of the object; and
transmitting, with the communication component from the observing subsystem to the reconstruction subsystem, the determined attributes of the scene.
14. The method of claim 13 , wherein the observing subsystem is navigation network-restricted during the capturing, the determining, and the transmitting.
15. The method of claim 13 , wherein the categorical attribute of the object comprises an object class of the object.
16. The method of claim 13 , wherein the categorical attribute of the object comprises a color of the object.
17. The method of claim 13 , wherein the object comprises an entity moving within the environment at the moment in time.
18. The method of claim 13 , wherein the object comprises a vehicle.
19. The method of claim 18 , wherein the categorical attribute of the object comprises a license plate number of the vehicle.
20. The method of claim 18 , wherein the categorical attribute of the object comprises at least one of a make or model of the vehicle.
21. The method of claim 13 , wherein the object comprises a human being.
22. The method of claim 13 , wherein the observing subsystem is a mobile subsystem.
23. The method of claim 22 , wherein the observing subsystem is a vehicle.
24. The method of claim 13 , wherein no pixel data of the scene is transmitted from the observing subsystem to the reconstruction subsystem.
25. The method of claim 13 , wherein the image sensor component comprises at least one of an infrared sensor or a thermal sensor.
26. A method of reconstructing a scene comprising an object in an environment using a reconstruction subsystem and an observing subsystem that is remote from the reconstruction subsystem, the reconstruction subsystem comprising a memory component, a communication component, and a processing module communicatively coupled to the memory component and the communication component, the method comprising:
receiving, with the communication component at the reconstruction subsystem from the observing subsystem, the following attributes of the scene:
a state-based attribute comprising georeferenced three-dimensional coordinates of the object in the environment; and
a categorical attribute of the object;
identifying, with the processing module from a plurality of three-dimensional models stored in the memory component, a particular three-dimensional model associated with the categorical attribute of the object;
rendering, with the processing module, a visualization of the scene by positioning the particular three-dimensional model in a georeferenced three-dimensional map at the georeferenced three-dimensional coordinates of the state-based attribute; and
presenting, with an output component of the reconstruction subsystem, at least a portion of the rendered visualization.
27. The method of claim 26 , wherein no pixel data of the scene is received from the observing subsystem by the reconstruction subsystem.