Object detection with instance detection and general scene understanding
Various implementations disclosed herein include devices, systems, and methods that determine a particular object instance in CGR environments. In some implementations, an object type of an object depicted in an image of a physical environment is identified. Then, a particular instance is determined based on the object type and the image. In some implementations, objects of the particular instance have a set of characteristics that differs from sets of characteristics associated with other instances of the object type. Then, the set of characteristics of the particular instance of the object depicted in the physical environment is obtained.
1. A method comprising:
at an electronic device having a processor:
identifying an object type of an object depicted in an image of a physical environment;
determining candidate instances of the object based on the object type, each of the candidate instances of the object being associated with at least one characteristic different from characteristics associated with another candidate instance of the object;
selecting a particular instance of the object from the candidate instances of the object based on a first characteristic of the object depicted in the image, wherein the particular instance of the object is associated with a set of characteristics that includes a second characteristic corresponding to the first characteristic; and
obtaining, via a database, the set of characteristics associated with the particular instance of the object depicted in the image of the physical environment.
2. The method of claim 1 , wherein identifying an object type of an object depicted in an image includes detecting the object type of a plurality of object types of the object in the image using a first machine learning model.
3. The method of claim 1 , wherein determining the candidate instances of the object based on the object type comprises:
using features extracted from a portion of the image depicting the object to generate a representation of the object; and
determining the candidate instances using the representation.
4. The method of claim 3 , wherein determining the candidate instances using the representation comprises:
querying a database of instances of the object type using the representation; and
receiving the candidate instances from the first database of the instances of the object type.
5. The method of claim 1 , wherein obtaining the set of characteristics associated with the particular instance of the object depicted in the image of the physical environment comprises accessing the database to receive information on materials, dimensions, physical properties, or visual properties of the particular instance of the object.
6. The method of claim 1 , wherein selecting the particular instance of the object comprises using a machine learning model that inputs an instance type and a representation of the object and outputs the particular instance of the object.
7. The method of claim 1 , further comprising:
combining the set of characteristics associated with the particular instance of the object with a CGR environment depicting the physical environment.
8. The method of claim 7 , wherein the set of characteristics associated with the particular instance of the object is used to perform scene reconstruction in the CGR environment depicting the physical environment.
9. The method of claim 7 , wherein the set of characteristics associated with the particular instance of the object is used to perform material detection in the CGR environment depicting the physical environment.
10. The method of claim 7 , wherein the set of characteristics associated with the particular instance of the object is used to perform environment texturing in the CGR environment depicting the physical environment.
11. The method of claim 7 , wherein the set of characteristics associated with the particular instance of the object is used to generate reflections of virtual objects in the CGR environment or reflections of real objects of the CGR environment in the virtual objects of the CGR environment depicting the physical environment.
12. The method of claim 7 , wherein the set of characteristics associated with the particular instance of the object is used to perform physics simulations in the CGR environment depicting the physical environment.
13. The method of claim 7 , wherein the set of characteristics associated with the particular instance of the object is used to reconstruct object portions in the CGR environment that are not in the image depicting the physical environment.
14. The method of claim 7 , wherein the set of characteristics associated with the particular instance of the object is used to determine object or plane boundaries in the CGR environment depicting the physical environment.
15. The method of claim 7 , wherein the set of characteristics associated with the particular instance of the object is used to effect removal of real objects from the CGR environment depicting the physical environment.
16. The method of claim 1 , wherein the set of characteristics associated with the particular instance of the object are characteristics associated with a specific brand and model of the object.
17. The method of claim 1 , wherein selecting the particular instance of the object from the candidate instances of the object comprises:
determining, based on the object type and the first characteristic, a respective confidence value for each of the candidate instances of the object; and
comparing the respective confidence value for each of the candidate instances of the object to a threshold.
18. A system comprising:
a non-transitory computer-readable storage medium; and
one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the system to perform operations comprising:
identifying an object type of an object depicted in an image of a physical environment;
determining candidate instances of the object based on the object type, each of the candidate instances of the object being associated with at least one characteristic different from characteristics associated with another candidate instance of the object;
selecting a particular instance from the candidate instances of the object based on a first characteristic of the object depicted in the image, wherein the particular instance of the object is associated with a set of characteristics that includes a second characteristic corresponding to the first characteristic; and
obtaining, via a database, the set of characteristics of associated with the particular instance of the object depicted in the image of the physical environment.
19. A non-transitory computer-readable storage medium, storing program instructions computer-executable on a computer to perform operations comprising:
identifying an object type of an object depicted in an image of a physical environment;
determining candidate instances of the object based on the object type, each of the candidate instances of the object being associated with at least one characteristic different from characteristics associated with another candidate instance of the object;
selecting a particular instance from the candidate instances of the object based on a first characteristic of the object depicted in the image, wherein the particular instance of the object is associated with a set of characteristics that includes a second characteristic corresponding to the first characteristic; and
obtaining, via a database, the set of characteristics associated with the particular instance of the object depicted in the image of the physical environment.
20. The non-transitory computer-readable storage medium of claim 19 , wherein determining the candidate instances of the object based on the object type comprises:
using features extracted from a portion of the image depicting the object to generate a representation of the object; and
determining the candidate instances using the representation.