Object detection with instance detection and general scene understanding
Various implementations disclosed herein include devices, systems, and methods that determine a particular object instance in CGR environments. In some implementations, an object type of an object depicted in an image of a physical environment is identified. Then, a particular instance is determined based on the object type and the image. In some implementations, objects of the particular instance have a set of characteristics that differs from sets of characteristics associated with other instances of the object type. Then, the set of characteristics of the particular instance of the object depicted in the physical environment is obtained.
1. A method comprising:
at an electronic device having a processor:
identifying an object type of an object depicted in an image of a physical environment;
determining a particular instance of the object based on the object type and the image, objects of the particular instance having a set of characteristics that differs from sets of characteristics of other instances of the object type;
obtaining the set of characteristics of the particular instance of the object depicted in the physical environment; and
combining the set of characteristics of the particular instance of the object with a CGR environment depicting the physical environment.
2. The method of claim 1 , wherein identifying the object type of the object depicted in an image includes detecting the object type of a plurality of object types of the object in the image using a first machine learning model.
3. The method of claim 1 , wherein determining a particular instance of the object based on the object type and the image comprises using features extracted from a portion of the image depicting the object to generate a representation of the particular instance of the object; and
determining the particular instance using the representation.
4. The method of claim 3 , wherein determining the particular instance using the representation comprises:
querying a first database of instances of the object type using the representation; and
receiving the particular instance from the first database.
5. The method of claim 4 , wherein obtaining the set of characteristics of the particular instance of the object depicted in the physical environment comprises accessing a database to receive information on materials, dimensions, physical properties, or visual properties of the particular instance of the object.
6. The method of claim 3 , wherein determining the particular instance using the representation comprises using a second machine learning model that inputs the object type and the representation of the particular instance of the object and outputs the particular instance of the object.
7. The method of claim 6 , wherein obtaining the set of characteristics of the particular instance of the object depicted in the physical environment comprises accessing a database to receive information on materials, dimensions, physical properties, or visual properties of the particular instance of the object.
8. The method of claim 1 , wherein combining the set of characteristics of the particular instance of the object with the CGR environment depicting the physical environment comprises using the set of characteristics of the particular instance of the object to perform environment lighting in the CGR environment depicting the physical environment.
9. The method of claim 1 , wherein combining the set of characteristics of the particular instance of the object with the CGR environment depicting the physical environment comprises using the set of characteristics of the particular instance of the object to perform scene understanding of the CGR environment depicting the physical environment.
10. The method of claim 1 , wherein combining the set of characteristics of the particular instance of the object with the CGR environment depicting the physical environment comprises using the set of characteristics of the particular instance of the object to perform scene reconstruction in the CGR environment depicting the physical environment.
11. The method of claim 1 , wherein combining the set of characteristics of the particular instance of the object with the CGR environment depicting the physical environment comprises using the set of characteristics of the particular instance of the object to perform material detection in the CGR environment depicting the physical environment.
12. The method of claim 1 , wherein combining the set of characteristics of the particular instance of the object with the CGR environment depicting the physical environment comprises using the set of characteristics of the particular instance of the object to perform environment texturing in the CGR environment depicting the physical environment.
13. The method of claim 1 , wherein combining the set of characteristics of the particular instance of the object with the CGR environment depicting the physical environment comprises using the set of characteristics of the particular instance of the object to generate reflections of virtual objects in the CGR environment or reflections of real objects of the CGR environment in the virtual objects of the CGR environment depicting the physical environment.
14. The method of claim 1 , wherein combining the set of characteristics of the particular instance of the object with the CGR environment depicting the physical environment comprises using the set of characteristics of the particular instance of the object to perform physics simulations in the CGR environment depicting the physical environment.
15. The method of claim 1 , wherein combining the set of characteristics of the particular instance of the object with the CGR environment depicting the physical environment comprises using the set of characteristics of the particular instance of the object to reconstruct object portions in the CGR environment that are not in the image depicting the physical environment.
16. The method of claim 1 , wherein combining the set of characteristics of the particular instance of the object with the CGR environment depicting the physical environment comprises using the set of characteristics of the particular instance of the object to determine object or plane boundaries in the CGR environment depicting the physical environment.
17. The method of claim 1 , wherein combining the set of characteristics of the particular instance of the object with the CGR environment depicting the physical environment comprises using the set of characteristics of the particular instance of the object to effect removal of real objects from the CGR environment depicting the physical environment.
18. A system comprising:
a non-transitory computer-readable storage medium; and
one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the system to perform operations comprising:
identifying an object type of an object depicted in an image of a physical environment;
determining a particular instance of the object based on the object type and the image, objects of the particular instance having a set of characteristics that differs from sets of characteristics of other instances of the object type;
obtaining the set of characteristics of the particular instance of the object depicted in the physical environment; and
combining the set of characteristics of the particular instance of the object with a CGR environment depicting the physical environment.
19. A non-transitory computer-readable storage medium, storing program instructions computer-executable on a computer to perform operations comprising:
identifying an object type of an object depicted in an image of a physical environment;
determining a particular instance of the object based on the object type and the image, objects of the particular instance having a set of characteristics that differs from sets of characteristics of other instances of the object type;
obtaining the set of characteristics of the particular instance of the object depicted in the physical environment; and
combining the set of characteristics of the particular instance of the object with a CGR environment depicting the physical environment.
20. The non-transitory computer-readable storage medium of claim 19 , wherein determining a particular instance of the object based on the object type and the image comprises using features extracted from a portion of the image depicting the object to generate a representation of the particular instance of the object; and
determining the particular instance using the representation.