Granted Patent B2

US 12,380,592 · App. 17/943,963 · Granted Aug 5, 2025

Inventors: Kentaro Wada (London, GB); Edgar Antonio Sucar Escamilla (London, GB); Stephen Lloyd James (London, GB); Daniel James Lenton (London, GB); Andrew Davison (London, GB)

Assignee: Imperial College Innovations Limited

G06T7/70G06T7/11G06T17/10G06V10/44G06V10/761G06T2207/10028

Abstract

Claims (71)

1. A method of determining a pose of each of a plurality of objects in a three-dimensional environment, the method comprising, using one or more processors:

for each given object of the plurality of objects:

obtaining image data and associated depth information representing a view of the three-dimensional environment in which at least a part of the given object is visible; and

processing the image data and the associated depth information to estimate a pose of the given object; and

iteratively updating the estimated poses of the plurality of objects, wherein the updating comprises:

sampling, for each given object of the plurality of objects, a plurality of points from a predetermined model of the given object transformed in accordance with the estimated pose of the given object;

determining respective first occupancy data for each given object of the plurality of objects dependent on positions of the points sampled from the predetermined model of the given object, relative to a voxel grid containing the given object, the first occupancy data for the given object being dependent on minimum distances between voxels of the voxel grid containing the given object and the points sampled from the predetermined model of the given object;

determining respective second occupancy data for each given object of the plurality of objects dependent on positions of the points sampled from the predetermined models of the other objects of the plurality of objects, relative to the voxel grid containing the given object, the second occupancy data for the given object being dependent on minimum distances between voxels of the voxel grid containing the given object and the points sampled from the predetermined models of the other objects of the plurality of objects; and

updating the estimated poses of the plurality of objects to reduce an occupancy penalty depending on the respective first occupancy data and the respective second occupancy data for each of the plurality of objects,

wherein the occupancy penalty comprises a collision component for each given object of the plurality of object which increases when a point sampled from the predetermined model of the given object and a point sampled from the predetermined model of a different object are simultaneously brought closer to a voxel of the voxel grid containing the given object.

2. The method of claim 1 , wherein updating the estimated poses of the plurality of objects comprises:

determining a gradient of the occupancy penalty with respect to the estimated poses of the plurality of objects; and

updating the estimated poses of the plurality of objects using the determined gradient of the occupancy penalty.

3. The method of claim 1 , wherein the dependences on the minimum distances saturate at a predetermined distance threshold.

4. The method of claim 1 , comprising processing the image data and the associated depth information to generate a volumetric reconstruction for each given object of the plurality of objects.

5. The method of claim 4 , comprising generating, using the generated volumetric reconstructions, third occupancy data indicating portions of the voxel grid containing the given object which are occupied by free space and portions of the voxel grid containing the given object which are occupied by objects other than the given object.

6. The method of claim 5 , wherein the occupancy penalty comprises a collision component for each given object of the plurality of objects which increases when a point sampled from the predetermined model of the given object is brought closer to a voxel of the voxel grid containing the given object which is occupied by free space or by objects other than the given object.

7. The method of claim 5 , wherein estimating the pose of each given object comprises determining the estimated pose of the given object using the generated third occupancy data and pointwise feature data for a plurality of points on the surface of the given object.

8. The method of claim 4 , wherein the occupancy penalty comprises a surface alignment component for each given object of the plurality of objects which decreases when a point sampled from the predetermined model of the given object is brought closer to a voxel of the voxel grid containing the given object which is occupied by the volumetric reconstruction for the given object.

9. The method of claim 1 , wherein:

the view of the three-dimensional environment is a first view of the three-dimensional environment; and

the determined pose of each given object is a first pose of the given object,

the method comprising:

obtaining further image data and further associated depth information representing a second view of the three-dimensional environment different to the first view of the three-dimensional environment; and

for each given object of the plurality of objects:

processing the further image data and the further associated depth information to determine a second pose for the given object;

transforming at least one of the first pose and the second pose of the given object to determine pose comparison data;

processing the pose comparison data to determine whether a consistency condition is met; and

when the consistency condition is determined to be met, generating a predetermined object model for the given object transformed consistently with the first pose and the second pose of the given object.

10. The method of claim 1 , comprising generating the predetermined model of each given object of the plurality of objects from a respective computer-aided design (CAD) model.

11. A system comprising:

one or more processors; and

one or more non-transitory storage media comprising instructions which, when executed by the one or more processors, cause the one or more processors to carry out operations comprising:

for each given object of a plurality of objects in a three-dimensional environment:

obtaining image data and associated depth information representing a view of the three-dimensional environment in which at least a part of the given object is visible; and

processing the image data and the associated depth information to estimate a pose of the given object; and

iteratively updating the estimated poses of the plurality of objects, wherein the updating comprises:

sampling, for each given object of the plurality of objects, a plurality of points from a predetermined model of the given object transformed in accordance with the estimated pose of the given object;

determining respective first occupancy data for each given object of the plurality of objects dependent on positions of the points sampled from the predetermined model of the given object, relative to a voxel grid containing the given object, the first occupancy data for the given object being dependent on minimum distances between voxels of the voxel grid containing the given object and the points sampled from the predetermined model of the given object;

determining respective second occupancy data for each given object of the plurality of objects dependent on positions of the points sampled from the predetermined models of the other objects of the plurality of objects, relative to the voxel grid containing the given object, the second occupancy data for the given object being dependent on minimum distances between voxels of the voxel grid containing the given object and the points sampled from the predetermined models of the other objects of the plurality of objects; and

updating the estimated poses of the plurality of objects to reduce an occupancy penalty depending on the respective first occupancy data and the respective second occupancy data for each of the plurality of objects,

wherein the occupancy penalty comprises a collision component for each given object of the plurality of object which increases when a point sampled from the predetermined model of the given object and a point sampled from the predetermined model of a different object are simultaneously brought closer to a voxel of the voxel grid containing the given object.

12. The system of claim 11 , wherein updating the estimated poses of the plurality of objects comprises:

determining a gradient of the occupancy penalty with respect to the estimated poses of the plurality of objects; and

updating the estimated poses of the plurality of objects using the determined gradient of the occupancy penalty.

13. The system of claim 11 , wherein for each given object of the plurality of objects:

the first occupancy data for the given object is dependent on minimum distances between voxels of the voxel grid containing the given object and the points sampled from the predetermined model of the given object; and

the second occupancy data for the given object is dependent on minimum distances between voxels of the voxel grid containing the given object and the points sampled from the predetermined models of the other objects of the plurality of objects.

14. The system of claim 11 , wherein the operations further comprise processing the image data and the associated depth information to generate a volumetric reconstruction for each given object of the plurality of objects.

15. The system of claim 11 , further comprising engaging means for engaging a given object of the plurality of objects in dependence on the determined pose of the given object.

16. The system of claim 11 , further comprising one or more sensors configured to capture the image data and the associated depth information.

17. The system of claim 16 , further comprising one or more actuators configured to move the one or more sensors, wherein:

the view of the three-dimensional environment is a first view of the three-dimensional environment;

the determined pose of each given object is a first pose of the given object; and the operations further comprise:

moving the one or more sensors using the one or more actuators;

capturing further image data and further associated depth information representing a second view of the three-dimensional environment resulting from the moving of the one or more sensors; and

for each given object of the plurality of objects:

processing the further image data and the further associated depth information to determine a second pose for the given object;

transforming at least one of the first pose and the second pose of the given object to determine pose comparison data;

processing the pose comparison data to determine whether a consistency condition is met; and

when the consistency condition is determined to be met, generating a predetermined object model for the given object transformed consistently with the first pose and the second pose of the given object.

18. A non-transitory storage medium comprising instructions which, when executed by one or more processors, cause the one or more processors to carry out operations comprising:

for each given object of a plurality of objects in a three-dimensional environment:

obtaining image data and associated depth information representing a view of the three-dimensional environment in which at least a part of the given object is visible; and

processing the image data and the associated depth information to estimate a pose of the given object; and

iteratively updating the estimated poses of the plurality of objects, wherein the updating comprises:

sampling, for each given object of the plurality of objects, a plurality of points from a predetermined model of the given object transformed in accordance with the estimated pose of the given object;

determining respective first occupancy data for each given object of the plurality of objects dependent on positions of the points sampled from the predetermined model of the given object, relative to a voxel grid containing the given object, the first occupancy data for the given object being dependent on minimum distances between voxels of the voxel grid containing the given object and the points sampled from the predetermined model of the given object;

determining respective second occupancy data for each given object of the plurality of objects dependent on positions of the points sampled from the predetermined models of the other objects of the plurality of objects, relative to the voxel grid containing the given object, the second occupancy data for the given object being dependent on minimum distances between voxels of the voxel grid containing the given object and the points sampled from the predetermined models of the other objects of the plurality of objects; and

updating the estimated poses of the plurality of objects to reduce an occupancy penalty depending on the respective first occupancy data and the respective second occupancy data for each of the plurality of objects,

wherein the occupancy penalty comprises a collision component for each given object of the plurality of object which increases when a point sampled from the predetermined model of the given object and a point sampled from the predetermined model of a different object are simultaneously brought closer to a voxel of the voxel grid containing the given object.

Assignments (1)

Priority Claims (1)

Continuity (2)

References Cited (15)