IP Library Granted Patent US 11,657,586
Granted Patent B1
US 11,657,586 · App. 18/064,658 · Granted May 23, 2023

Systems and methods for augmented reality viewing based on directly mapped point cloud overlays

Inventor: Alexandre Leuckert Klein (Novo Hamburgo, BR)
Assignee: Illuscio, Inc.
G06T19/006G06T7/20G06T7/70G06T17/00G06V10/44G06T2207/10024G06V2201/07
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,657,586
App. No.
18/064,658
Granted
May 23, 2023
Kind
B1
Abstract

An augmented reality system enhances or augments the view of a real-world object by replacing different portions of the object with visualizations of non-visible data encoded for those different portions. The system may receive an image that captures visible properties of an object from a particular perspective. The system may select a model with data points that are generated from scanning the object and that encode visible and non-visible properties of the object. The system may align the model to match the object at the particular perspective, may map the non-visible properties from a set of the data points to an object part that is represented and aligned with the set of data points, and may present an augmented reality view in which the visible properties at the object part from the particular perspective are replaced with a visualization of the non-visible properties from the set of data points.

Claims (79)

1. A method comprising:

obtaining an image of an object at a first position;

selecting a model of the object, wherein the model comprises a plurality of data points, wherein each data point of the plurality of data points comprises positional elements and non-positional elements, wherein the positional elements of a particular data point define a position of a corresponding point on the object, and wherein the non-positional elements of the particular data point define non-positional properties of the corresponding point on the object;

establishing one or more points of reference between different subsets of the plurality of data points and corresponding points on the object;

mapping a set of the plurality of data points corresponding to a particular feature of the object based on the one or more points of reference; and

enhancing the image of the object by presenting values from one or more of the non-positional elements of the set of data points over the particular feature of the object.

2. The method of claim 1 further comprising:

obtaining a second image of the object at a second position;

tracking movement of the one or more points of reference in response to the object moving from the first position to the second position; and

updating the second image of the object by presenting the values from the one or more of the non-positional elements of the set of data points over the particular feature of the object at the second position based on said tracking of the movement of the one or more points of reference.

3. The method of claim 1 , wherein mapping the set of data points comprises:

aligning the different subsets of data points over the corresponding points on the object in the image;

determining positioning of the set of data points relative to the different subsets of data points established as the one or more points of reference; and

aligning the set of data points over the particular feature of the object in the image based on aligning the different subsets of data points and determining the positioning of the set of data points relative to the different subsets of data points.

4. The method of claim 3 ,

wherein aligning the different subsets of data points comprises adjusting the positional elements of the different subsets of data points to match positioning of the corresponding points in the image; and

wherein aligning the set of data points comprises adjusting the positional elements of the set of data points equal to said adjusting of the positional elements of the different subsets of data points.

5. The method of claim 1 further comprising:

tracking movement of the one or more points of reference from a first image to a second image;

computing a positional offset between the one or more points of reference from the first image to the second image; and

realigning the set of data points with the particular feature by applying the positional offset to the positional elements of the set of data points.

6. The method of claim 1 further comprising:

adjusting the positional elements of the plurality of data points based on differences in the positional elements of the different subsets of data points and the corresponding points in the image.

7. The method of claim 6 further comprising:

tracking an offset between the one or more points of reference in the image and a subsequent second image;

readjusting the positional elements of the set of data points according to the offset; and

enhancing the subsequent second image by presenting the values from the one or more of the non-positional elements of the set of data points at positional values resulting from readjusting the positional elements according to the offset.

8. The method of claim 1 ,

wherein obtaining the image of the object comprises performing a hyperspectral capture of the object; and

wherein enhancing the image comprises presenting hyperspectral properties of the particular feature over a visible light image of the particular feature.

9. The method of claim 1 further comprising:

providing an interactive tool for selecting the one or more non-positional elements; and

wherein enhancing the image comprises:

modifying a visualization of the particular feature by presenting values from a first set of the non-positional elements of the set of data points over the particular feature in response to a first selection that is made using the interactive tool; and

modifying the visualization of the particular feature by presenting values from a different second set of the non-positional elements of the set of data points over the particular feature in response to a second selection that is made using the interactive tool.

10. The method of claim 1 , wherein mapping the set of data points comprises:

receiving a selection of the particular feature in the image;

determining an offset between the selection and the corresponding points on the object used as the one or more points of reference; and

selecting the set of data points based on the offset and the positional elements of the different subsets of data points.

11. The method of claim 1 , wherein establishing the one or more points of reference comprises:

detecting a distinctive structure or formation at the corresponding points of the image; and

determining that the positional elements of the different subsets of data points form the distinctive structure or formation.

12. The method of claim 1 , wherein establishing the one or more points of reference comprises:

detecting a unique pattern or variation in coloring at the corresponding points of the image; and

determining that the non-positional elements of the different subsets of data points create the unique pattern or variation in the coloring.

13. The method of claim 1 , wherein establishing the one or more points of reference comprises:

scaling or rotating the plurality of data points so that the different subsets of data points become aligned with the corresponding points on the object in the image.

14. A system comprising:

one or more hardware processors configured to:

obtain an image of an object at a first position;

select a model of the object, wherein the model comprises a plurality of data points, wherein each data point of the plurality of data points comprises positional elements and non-positional elements, wherein the positional elements of a particular data point define a position of a corresponding point on the object, and wherein the non-positional elements of the particular data point define non-positional properties of the corresponding point on the object;

establish one or more points of reference between different subsets of the plurality of data points and corresponding points on the object;

map a set of the plurality of data points corresponding to a particular feature of the object based on the one or more points of reference; and

enhance the image of the object by presenting values from one or more of the non-positional elements of the set of data points over the particular feature of the object.

15. The system of claim 14 , wherein the one or more hardware processors are further configured to:

obtain a second image of the object at a second position;

track movement of the one or more points of reference in response to the object moving from the first position to the second position; and

update the second image of the object by presenting the values from the one or more of the non-positional elements of the set of data points over the particular feature of the object at the second position based on said tracking of the movement of the one or more points of reference.

16. The system of claim 14 , wherein the one or more hardware processors are further configured to:

track movement of the one or more points of reference from a first image to a second image;

compute a positional offset between the one or more points of reference from the first image to the second image; and

realign the set of data points with the particular feature by applying the positional offset to the positional elements of the set of data points.

17. The system of claim 14 , wherein the one or more hardware processors are further configured to:

adjust the positional elements of the plurality of data points based on differences in the positional elements of the different subsets of data points and the corresponding points in the image.

18. The system of claim 17 , wherein the one or more hardware processors are further configured to:

track an offset between the one or more points of reference in the image and a subsequent second image;

readjust the positional elements of the set of data points according to the offset; and

enhance the subsequent second image by presenting the values from the one or more of the non-positional elements of the set of data points at positional values resulting from readjusting the positional element according to the offset.

19. The system of claim 14 , wherein the one or more hardware processors are further configured to:

provide an interactive tool for selecting the one or more non-positional elements; and

wherein enhancing the image comprises:

modifying a visualization of the particular feature by presenting values from a first set of the non-positional elements of the set of data points over the particular feature in response to a first selection that is made using the interactive tool; and

modifying the visualization of the particular feature by presenting values from a different second set of the non-positional elements of the set of data points over the particular feature in response to a second selection that is made using the interactive tool.

20. A non-transitory computer-readable medium storing program instructions that, when executed by one or more hardware processors, cause a computing system to perform operations comprising:

obtaining an image of an object at a first position;

selecting a model of the object, wherein the model comprises a plurality of data points, wherein each data point of the plurality of data points comprises positional elements and non-positional elements, wherein the positional elements of a particular data point define a position of a corresponding point on the object, and wherein the non-positional elements of the particular data point define non-positional properties of the corresponding point on the object;

establishing one or more points of reference between different subsets of the plurality of data points and corresponding points on the object;

mapping a set of the plurality of data points corresponding to a particular feature of the object based on the one or more points of reference; and

enhancing the image of the object by presenting values from one or more of the non-positional elements of the set of data points over the particular feature of the object.

Assignments (2)
CHANGE OF NAME Recorded Sep 18, 2025
From: ILLUSCIO, INC.
To: MIRIS, INC.
Reel/Frame 072896/0406 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 12, 2022
From: KLEIN, ALEXANDRE L
To: ILLUSCIO, INC.
Reel/Frame 062058/0860 →
Continuity (1)
Continuation 17822613 · Aug 26, 2022