IP Library › Granted Patent US 12682483
Granted Patent B2
US 12682483 · App. 18/384,352 · Granted Jul 14, 2026

Methods and devices for detecting and identifying features in an AR/VR scene

Inventors: Jeffrey S. Norris (Saratoga, CA); Alexandre Da Veiga (San Francisco, CA); Bruno M. Sommer (Sunnyvale, CA); Ye Cong (Santa Clara, CA); Tobias Eble (Sunnyvale, CA); Moinul Khan (San Jose, CA); Nicolas Bonnier (Campbell, CA); Hao Pan (Sunnyvale, CA)
Assignee: Apple Inc.
G06T7/73G02B27/017G06V10/245G06V10/462G06V20/20G06T2207/30204
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12682483
App. No.
18/384,352
Granted
Jul 14, 2026
Kind
B2
Abstract

A method includes obtaining first pass-through image data characterized by a first pose. The method includes obtaining respective pixel characterization vectors for pixels in the first pass-through image data. The method includes identifying a feature of an object within the first pass-through image data in accordance with a determination that pixel characterization vectors for the feature satisfy a feature confidence threshold. The method includes displaying the first pass-through image data and an AR display marker that corresponds to the feature. The method includes obtaining second pass-through image data characterized by a second pose. The method includes transforming the AR display marker to a position associated with the second pose in order to track the feature. The method includes displaying the second pass-through image data and maintaining display of the AR display marker that corresponds to the feature of the object based on the transformation.

Claims (50)

1 . A method comprising:

at an electronic device with one or more processors, a non-transitory memory, and a display:

generating, from pass-through image data characterized by a plurality of poses of a space, a three-dimensional (3D) point cloud for the space, wherein each of the plurality of poses of the space is associated with a respective field of view of an image sensor;

obtaining characterization vectors for points of the 3D point cloud, wherein each of the characterization vectors includes one or more labels;

disambiguating a first group of points from the 3D point cloud corresponding to a first object in the space, wherein characterization vectors for the first group of points satisfy an object confidence threshold based on the labels included in the characterization vectors for the first group of points;

generating a first volumetric region for the first group of points, wherein the first volumetric region corresponds to a 3D representation of the first object; and

synthesizing a two-dimensional (2D) floorplan of the space including a 2D boundary of the space and a 2D representation of the first object within the 2D boundary of the space corresponding to a top-down view of the first volumetric region.

2 . The method of claim 1 , further comprising displaying, on the display, at least a portion of the 2D floorplan of the space.

3 . The method of claim 2 , further comprising displaying, on the display AR content overlaid on the portion of the 2D floorplan of the space.

4 . The method of claim 3 , wherein displaying the AR content includes:

computing a measurement associated with the first object based on the first group of points; and

displaying a measurement indicator overlaid on the portion of the 2D floorplan of the space and proximate to the 2D representation of the first object, wherein the measurement indicator indicates the measurement associated with the object.

5 . The method of claim 1 , further comprising:

disambiguating a second group of points from the 3D point cloud corresponding to a second object in the space, wherein characterization vectors for the second group of points satisfy the object confidence threshold based on the labels included in the characterization vectors for the second group of points;

generating a second volumetric region for the second group of points, wherein the second volumetric region corresponds to a 3D representation of the second object; and

resynthesizing the 2D floorplan of the space to further include a 2D representation of the second object within the 2D boundary of the space corresponding to a top-down view of the second volumetric region.

6 . The method of claim 2 , wherein the portion of the 2D floorplan satisfies one or more display criteria.

7 . The method of claim 6 , wherein the portion of the 2D floorplan is within a threshold distance from the image sensor.

8 . The method of claim 1 , wherein the characterization vectors are obtained from a point characterizer.

9 . The method of claim 1 , further comprising obtaining, from the image sensor, the pass-through image data.

10 . The method of claim 1 , wherein the electronic device corresponds to a mobile device.

11 . The method of claim 1 , wherein the display corresponds to a head-mounted display (HMD).

12 . The method of claim 1 , wherein the display is separate from the image sensor.

13 . An electronic device, comprising:

a display;

one or more processors;

a non-transitory memory; and

one or more programs stored in the non-transitory memory and configured to be executed by the one or more processors, the one or more programs including instructions, which, when executed by the electronic device, cause the electronic device to perform operations comprising:

generating, from pass-through image data characterized by a plurality of poses of a space, a three-dimensional (3D) point cloud for the space, wherein each of the plurality of poses of the space is associated with a respective field of view of an image sensor;

obtaining characterization vectors for points of the 3D point cloud, wherein each of the characterization vectors includes one or more labels;

disambiguating a first group of points from the 3D point cloud corresponding to a first object in the space, wherein characterization vectors for the first group of points satisfy an object confidence threshold based on the labels included in the characterization vectors for the first group of points;

generating a first volumetric region for the first group of points, wherein the first volumetric region corresponds to a 3D representation of the first object; and

synthesizing a two-dimensional (2D) floorplan of the space including a 2D boundary of the space and a 2D representation of the first object within the 2D boundary of the space corresponding to a top-down view of the first volumetric region.

14 . The electronic device of claim 13 , wherein the operations further comprise displaying, on the display, at least a portion of the 2D floorplan of the space.

15 . The electronic device of claim 14 , wherein the operations further comprise displaying, on the display, AR content overlaid on the portion of the 2D floorplan of the space.

16 . The electronic device of claim 15 , wherein displaying the AR content includes:

computing a measurement associated with the first object based on the first group of points; and

displaying a measurement indicator overlaid on the portion of the 2D floorplan of the space and proximate to the 2D representation of the first object, wherein the measurement indicator indicates the measurement associated with the object.

17 . A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which, when executed by an electronic device with a display, cause the electronic device to:

generate, from pass-through image data characterized by a plurality of poses of a space, a three-dimensional (3D) point cloud for the space, wherein each of the plurality of poses of the space is associated with a respective field of view of an image sensor;

obtain characterization vectors for points of the 3D point cloud, wherein each of the characterization vectors includes one or more labels;

disambiguate a first group of points from the 3D point cloud corresponding to a first object in the space, wherein characterization vectors for the first group of points satisfy an object confidence threshold based on the labels included in the characterization vectors for the first group of points;

generate a first volumetric region for the first group of points, wherein the first volumetric region corresponds to a 3D representation of the first object; and

synthesize a two-dimensional (2D) floorplan of the space including a 2D boundary of the space and a 2D representation of the first object within the 2D boundary of the space corresponding to a top-down view of the first volumetric region.

18 . The non-transitory computer readable storage medium of claim 17 , wherein the instructions further cause the electronic device to:

disambiguate a second group of points from the 3D point cloud corresponding to a second object in the space, wherein characterization vectors for the second group of points satisfy the object confidence threshold based on the labels included in the characterization vectors for the second group of points;

generate a second volumetric region for the second group of points, wherein the second volumetric region corresponds to a 3D representation of the second object; and

resynthesize the 2D floorplan of the space to further include a 2D representation of the second object within the 2D boundary of the space corresponding to a top-down view of the second volumetric region.

19 . The non-transitory computer readable storage medium of claim 17 , wherein the instructions further cause the electronic device to display at least a portion of the 2D floorplan, wherein the portion of the 2D floorplan satisfies one or more display criteria.

20 . The non-transitory computer readable storage medium of claim 19 , wherein the portion of the 2D floorplan is within a threshold distance from the image sensor.