IP Library › Granted Patent US 11,763,460
Granted Patent B2
US 11,763,460 · App. 17/245,767 · Granted Sep 19, 2023

Image segmentation confidence determination

Inventors: Jonathan Tung (Cupertino, CA); Jung W Suh (Palo Alto, CA); Advit Bhatt (Danville, CA)
G06T7/12G06F18/23G06V10/28G06V10/752G06V10/763G06V10/809G06T2207/10132G06T2207/30101
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,763,460
App. No.
17/245,767
Granted
Sep 19, 2023
Kind
B2
Abstract

Examples for determining a confidence level associated with image segmentation are disclosed. A confidence level associated with a collective image segmentation result can be determined by generating multiple individual segmentation results each from the same image data. These examples can then aggregate the individual segmentation results to form the collective image segmentation result and measure the spread of each individual segmentation result from the collective image segmentation result. The measured spread of each individual segmentation result can then be used to determine the confidence level associated with the collective image segmentation result. This can allow a confidence level associated with the collective image segmentation result to be determined. This confidence level may be determined without needing a ground truth to compare to the collective image segmentation result.

Claims (57)

1. A method comprising:

receiving, by a processor, an image that includes one or more objects;

for each of a plurality of models, generating, by the processor and using the respective model, a boundary prediction for each of the one or more objects within the image, wherein each boundary prediction comprises a plurality of boundary prediction portions and each model of the plurality of models comprises a distinct model;

generating, by the processor, and based on the boundary prediction for each model of the plurality of models, and for each object of the one or more objects a clustered boundary prediction for the respective object within the image, the clustered boundary prediction comprising a plurality of clustered boundary prediction portions that each correspond to one of the plurality of boundary prediction portions of each of the boundary predictions;

for each of the plurality of clustered boundary prediction portions for each object of the one or more objects, determining, by the processor, and based on a comparison between the respective clustered boundary prediction portion and the corresponding boundary prediction portion, a confidence level for the respective clustered boundary prediction portion for the respective object; and

for each object of the one or more objects, outputting, by the processor and for display on a display device, a graphical indication for each of the plurality of clustered boundary prediction portions for the respective object, wherein each clustered boundary prediction portion has a visual characteristic indicative of the confidence level for the respective clustered boundary prediction portion.

2. The method of claim 1 , wherein the plurality of models includes one or more of a machine learning model, a deep learning model, a pattern recognition model, a computer vision model, and a parameter-modified model.

3. The method of claim 1 , wherein determining the confidence level for the respective clustered boundary prediction portion comprises:

for each of the plurality of models, determining, by the processor, and based on a comparison between the respective clustered boundary prediction portion and the corresponding boundary prediction portion of the boundary predictions for the respective object, a spread for each of the corresponding boundary prediction portions of the boundary predictions for the respective object; and

determining, by the processor and based on the spread for each of the corresponding boundary prediction portions of the boundary predictions for the respective object, the confidence level for the respective clustered boundary prediction portion.

4. The method of claim 3 , wherein determining the spread for each of the corresponding boundary prediction portions of the boundary predictions comprises:

creating, by the processor and using the clustered boundary prediction portions and each of the corresponding boundary prediction portions of the boundary predictions for the respective object, a prediction distribution with the clustered boundary prediction portions as a mean of the prediction distribution;

determining, by the processor, a standard deviation for each of the corresponding boundary prediction portions of the boundary predictions for the respective object; and

determining, by the processor, a z-score for each of the corresponding boundary prediction portions of the boundary predictions for the respective object as a number of standard deviations away the respective corresponding boundary prediction portion is from the clustered boundary prediction for the respective object.

5. The method of claim 3 , further comprising:

determining, by the processor, that the spread for a first corresponding boundary prediction portion of a first boundary prediction for the respective object exceeds a deviation threshold for a prediction distribution, the first boundary prediction being associated with a first model of the plurality of models, wherein the deviation threshold comprises one of a user-defined threshold or a default threshold;

removing, by the processor, the first model from the plurality of models to generate a subset of models for the respective object; and

re-calculating, by the processor, at least the respective clustered boundary prediction portion for the respective object using only the subset of models.

6. The method of claim 1 , wherein the visual characteristic comprises one or more of a line color, a fill pattern, a line weight, a line segmentation, a textual indication, or a graphical icon.

7. The method of claim 1 , further comprising:

for each of the plurality of models, determining, by the processor, a weight for the respective model; and

for each of the one or more objects, generating, by the processor, and based on the corresponding boundary prediction portions of the boundary predictions and the weight for each model of the plurality of models, the clustered boundary prediction for the respective object as a weighted clustered boundary prediction for the respective object within the image.

8. The method of claim 1 , wherein the plurality of models comprises three or more distinct models.

9. The method of claim 1 , wherein the image comprises one or more of a cross-sectional view of the one or more objects, a longitudinal view of the one or more objects, or a three-dimensional view of the one or more objects.

10. A computing device comprising:

at least one processor; and

a storage component configured to store instructions executable by the at least one processor to:

receive an image that includes one or more objects;

for each of a plurality of models, generate, using the respective model, a boundary prediction for each of the one or more objects within the image, wherein each boundary prediction comprises a plurality of boundary prediction portions and each model of the plurality of models comprises a distinct model;

generate, based on the boundary prediction for each model of the plurality of models, and for each object of the one or more objects a clustered boundary prediction for the respective object within the image, the clustered boundary prediction comprising a plurality of clustered boundary prediction portions that each correspond to one of the plurality of boundary prediction portions of each of the boundary predictions;

for each object of the one or more objects, for each of the plurality of clustered boundary prediction portions for the respective object, determine, and based on a comparison between the respective clustered boundary prediction portion and the corresponding boundary prediction portion, a confidence level for the respective clustered boundary prediction portion for the respective object; and

for each object of the one or more objects, output, for display on a display device, a graphical indication for each of the plurality of clustered boundary prediction portions for the respective object, wherein each clustered boundary prediction portion has a visual characteristic indicative of the confidence level for the respective clustered boundary prediction portion.

11. The computing device of claim 10 , wherein the plurality of models includes one or more of a machine learning model, a deep learning model, a pattern recognition model, a computer vision model, and a parameter-modified model.

12. The computing device of claim 10 , wherein the instructions executable by the one or more processors to determine the confidence level for the respective clustered boundary prediction portion comprise instructions executable by the one or more processors to:

for each of the plurality of models, determine, based on the comparison between the respective clustered boundary prediction portion and the corresponding boundary prediction portions of the boundary predictions for the respective object, a spread for each of the corresponding boundary prediction portions of the boundary predictions for the respective object; and

determine, based on the spread for each of the corresponding boundary prediction portions of the boundary predictions for the respective object, the confidence level for the respective clustered boundary prediction portion.

13. The computing device of claim 12 , wherein the instructions executable by the one or more processors to determine the spread for each of the corresponding boundary prediction portions of the boundary predictions comprise instructions executable by the one or more processors to:

create, using the clustered boundary prediction portions and each of the corresponding boundary prediction portions of the boundary predictions for the respective object, a prediction distribution with the clustered boundary prediction portions as a mean of the prediction distribution;

determine a standard deviation for each of the corresponding boundary prediction portions of the boundary predictions for the respective object; and

determine a z-score for each of the corresponding boundary prediction portions of the boundary predictions for the respective object as a number of standard deviations away the corresponding boundary prediction portion is from the clustered boundary prediction portion for the respective object.

14. The computing device of claim 12 , wherein the instructions are further executable by the one or more processors to:

determine that the spread for a first corresponding boundary prediction portion of a first boundary prediction for the respective object exceeds a deviation threshold for a prediction distribution, the first boundary prediction being associated with a first model of the plurality of models;

remove the first model from the plurality of models to generate a subset of models for the respective object; and

re-calculate at least the corresponding clustered boundary prediction portion for the respective object using only the subset of models.

15. The computing device of claim 14 , wherein the deviation threshold comprises one of a user-defined threshold or a default threshold.

16. The computing device of claim 10 , wherein the visual characteristic comprises one or more of a line color, a fill pattern, a line weight, a line segmentation, a textual indication, or a graphical icon.

17. The computing device of claim 10 , wherein the instructions are further executable by the one or more processors to:

for each of the plurality of models, determine a weight for the respective model; and

for each of the one or more objects, generate, based on the corresponding boundary prediction portions of the boundary predictions and the weight for each model of the plurality of models, the clustered boundary prediction for the respective object as a weighted clustered boundary prediction for the respective object within the image.

18. The computing device of claim 10 , wherein the plurality of models comprises three or more distinct models.

19. The computing device of claim 10 , further comprising a camera configured to capture the image.

20. A non-transitory computer-readable storage medium is described having stored thereon instructions that, when executed, cause one or more processors of a computing device to:

receive an image that includes one or more objects;

for each of a plurality of models, generate, using the respective model, a boundary prediction for each of the one or more objects within the image, wherein each boundary prediction comprises a plurality of boundary prediction portions and each model of the plurality of models comprises a distinct model;

generate, based on the boundary prediction for each model of the plurality of models, and for each object of the one or more objects a clustered boundary prediction for the respective object within the image, the clustered boundary prediction comprising a plurality of clustered boundary prediction portions that each correspond to one of the plurality of boundary prediction portions of each of the boundary predictions;

for each of the plurality of clustered boundary prediction portions for each object of the one or more objects, determine, and based on a comparison between the respective clustered boundary prediction portion and the corresponding boundary prediction portion, a confidence level for the respective clustered boundary prediction portion for the respective object; and

for each object of the one or more objects, output, for display on a display device, a graphical indication for each of the plurality of clustered boundary prediction portions for the respective object, wherein each clustered boundary prediction portion has a visual characteristic indicative of the confidence level for the respective clustered boundary prediction portion.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 30, 2021
From: TUNG, JONATHAN; SUH, JUNG W.; BHATT, ADVIT
To: ACIST MEDICAL SYSTEMS, INC.
Reel/Frame 056100/0315 →
Continuity (3)
Continuation 16581577 · Sep 24, 2019
Provisional Application 62869793 · Jul 2, 2019
Related Publication 20210264610A1 · Aug 26, 2021