Scene understanding using occupancy grids
This document describes scene understanding for cross reality systems using occupancy grids. In one aspect, a method includes recognizing one or more objects in a model of a physical environment generated using images of the physical environment. For each object, a bounding box is fit around the object. An occupancy grid that includes a multiple cells is generated within the bounding box around the object. A value is assigned to each cell of the occupancy grid based on whether the cell includes a portion of the object. An object representation that includes information describing the occupancy grid for the object is generated. The object representations are sent to one or more devices.
1. A computer-implemented method performed by one or more data processing apparatus, the method, comprising:
recognizing one or more objects in a model of a physical environment generated using images of the physical environment;
for each object of the one or more objects:
fitting a bounding box around each object;
generating an occupancy grid within the bounding box around each object, wherein the occupancy grid includes a plurality of cells;
assigning a value to each cell of the occupancy grid based on whether the cell includes a portion of each object; and
generating an object representation that includes information describing the occupancy grid for each object; and
sending the object representations to one or more devices.
2. The computer-implemented method of claim 1 , wherein assigning a value to each cell of the occupancy grid based on whether the cell includes a portion of each object, comprises:
assigning a first value to each cell that includes a portion of each object; and
assigning a second value different from the first value to each cell that does not include any portion of each object.
3. The computer-implemented method of claim 1 , comprising:
detecting a change to a given object of the one or more objects;
generating a new occupancy grid for the given object; and
sending the new occupancy grid for the given object to the one or more devices rather than an updated model of the physical environment.
4. The computer-implemented method of claim 3 , wherein each device of the one or more devices update a local mesh for the physical environment using the new occupancy grid for the given object.
5. The computer-implemented method of claim 1 , wherein the model comprises a plurality of voxels that represent the physical environment.
6. The computer-implemented method of claim 5 , further comprising assigning a semantic label to each voxel based on a type of object recognized in the voxel.
7. The computer-implemented method of claim 6 , further comprising clustering voxels based on the semantic label for each voxel.
8. A computer-implemented system, comprising:
one or more computers; and
one or more computer memory devices interoperably coupled with the one or more computers and having tangible, non-transitory, machine-readable media storing one or more instructions that, when executed by the one or more computers, perform operations comprising:
recognizing one or more objects in a model of a physical environment generated using images of the physical environment;
for each object of the one or more objects:
fitting a bounding box around each object;
generating an occupancy grid within the bounding box around each object, wherein the occupancy grid includes a plurality of cells;
assigning a value to each cell of the occupancy grid based on whether the cell includes a portion of each object; and
generating an object representation that includes information describing the occupancy grid for each object; and
sending each object representations to one or more devices.
9. The computer-implemented system of claim 8 , wherein assigning a value to each cell of the occupancy grid based on whether the cell includes a portion of each object, comprises:
assigning a first value to each cell that includes a portion of each object; and
assigning a second value different from the first value to each cell that does not include any portion of each object.
10. The computer-implemented system of claim 8 , wherein the operations comprise:
detecting a change to a given object of the one or more objects;
generating a new occupancy grid for the given object; and
sending the new occupancy grid for the given object to the one or more devices rather than an updated model of the physical environment.
11. The computer-implemented system of claim 10 , wherein each device of the one or more devices update a local mesh for the physical environment using the new occupancy grid for the given object.
12. The computer-implemented system of claim 8 , wherein the model comprises a plurality of voxels that represent the physical environment.
13. The computer-implemented system of claim 12 , wherein the operations comprise assigning a semantic label to each voxel based on a type of object recognized in the voxel.
14. A non-transitory, computer-readable medium storing one or more instructions executable by a computer system to perform operations, comprising:
recognizing one or more objects in a model of a physical environment generated using images of the physical environment;
for each object of the one or more objects:
fitting a bounding box around each object;
generating an occupancy grid within the bounding box around each object, wherein the occupancy grid includes a plurality of cells;
assigning a value to each cell of the occupancy grid based on whether the cell includes a portion of each object; and
generating an object representation that includes information describing the occupancy grid for each object; and
sending each object representations to one or more devices.
15. The non-transitory, computer-readable medium of claim 14 , wherein assigning a value to each cell of the occupancy grid based on whether the cell includes a portion of each object, comprises:
assigning a first value to each cell that includes a portion of each object; and
assigning a second value different from the first value to each cell that does not include any portion of each object.
16. The non-transitory, computer-readable medium of claim 14 , wherein the operations comprise:
detecting a change to a given object of the one or more objects;
generating a new occupancy grid for the given object; and
sending the new occupancy grid for the given object to the one or more devices rather than an updated model of the physical environment.
17. The non-transitory, computer-readable medium of claim 16 , wherein each device of the one or more devices update a local mesh for the physical environment using the new occupancy grid for the given object.
18. The non-transitory, computer-readable medium of claim 14 , wherein the model comprises a plurality of voxels that represent the physical environment.
19. The non-transitory, computer-readable medium of claim 18 , wherein the operations comprise assigning a semantic label to each voxel based on a type of object recognized in the voxel.
20. The non-transitory, computer-readable medium of claim 19 , wherein the operations comprise clustering voxels based on the semantic label for each voxel.