IP Library Granted Patent US 10,957,045
Granted Patent B2
US 10,957,045 · App. 16/346,533 · Granted Mar 23, 2021

Segmenting ultrasound images

Inventors: Danny Ziyi Chen (Granger, IN); Yizhe Zhang (South Bend, IN); Lin Yang (Mishawaka, IN); Michael Tin-Cheung Ying (Hong Kong, HK); Anil Tejbhan Ahuja (Hong Kong, HK)
Assignees: University of Notre Dame du Lac; Honk Kong Polytechnic University; Chinese University of Hong Kong
G06T7/12G06K9/0014G06K9/00147G06K9/4628G06K9/627G06K9/6271G06K2209/05G06T2200/24G06T2207/10024G06T2207/10132G06T2207/20084G06T2207/20116G06T2207/30004
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,957,045
App. No.
16/346,533
Granted
Mar 23, 2021
Kind
B2
Abstract

Optimizations are provided for segmenting tissue objects included in an ultrasound image. Initially, raw pixel data is received. Here, each pixel corresponds to ultrasound information. This raw pixel data is processed through a first fully convolutional network to generate a first segmentation label map. This first map includes a first set of objects that have been segmented into a coarse segmentation class. Then, this first map is processed through a second fully convolutional network to generate a second segmentation label map. This second map is processed using the raw pixel data as a base reference. Further, this second map includes a second set of objects that have been segmented into a fine segmentation class. Then, a contour optimization algorithm is applied to at least one of the second set of objects in order to refine that object's contour boundary. Subsequently, that object is identified as corresponding to a lymph node.

Claims (52)

1. A computer system comprising:

one or more processors; and

one or more computer-readable hardware storage devices having stored thereon computer-executable instructions that, when executed by the one or more processors, cause the computer system to:

receive raw image data that is comprised of an array of pixels, each pixel within the array of pixels comprising ultrasound information;

process the raw image data through a first fully convolutional network to generate a first segmentation label map, wherein:

the first segmentation label map comprises a first set of objects that have been segmented into at least a coarse segmentation class, and

each object within the first set of objects corresponds to a group of pixels from the array of pixels;

process the first segmentation label map through a second fully convolutional network to generate a second segmentation label map, wherein:

processing the first segmentation label map through the second fully convolutional network is performed using the raw image data as a base reference,

the second segmentation label map comprises a second set of objects that have been segmented into a fine segmentation class, and

each object within the second set of objects corresponds to a group of pixels from the array of pixels;

apply a contour optimization algorithm to at least one object within the second set of objects, wherein the contour optimization algorithm refines a corresponding contour boundary for the at least one object; and

generate an identification that the at least one object corresponds to a lymph node.

2. The computer system of claim 1 , wherein the array of pixels includes (1) pixels that correspond to a lymph node and (2) pixels that correspond to tissues that are not lymph nodes.

3. The computer system of claim 2 , wherein objects within the first set have associated therewith a similarity probability that satisfies a first threshold level, the similarity probability being based on an estimated similarity in visual appearance between each of the objects in the first set and an identifiable lymph node.

4. The computer system of claim 3 , wherein objects within the second set are objects whose similarity probability satisfies a second threshold level, the second threshold level being stricter than the first threshold level.

5. The computer system of claim 1 , wherein the contour optimization algorithm refines the corresponding contour boundary for the at least one object as a function of convexity.

6. The computer system of claim 1 , wherein the array of pixels defines (1) an image length of the raw image data, (2) an image width of the raw image data, and (3) a number of color channels that are included in the raw image data.

7. The computer system of claim 6 , wherein the number of color channels is 1.

8. The computer system of claim 1 , wherein the first segmentation label map is displayed on a user interface of the computer system, and wherein all of the objects in the first set are displayed using a first format while all other objects are displayed using a different format.

9. The computer system of claim 1 , wherein the second segmentation label map is displayed on a user interface of the computer system, and wherein all of the objects in the second set are displayed using a first format while all other objects are displayed using a different format.

10. The computer system of claim 1 , wherein the first segmentation label map and the second segmentation label map are displayed simultaneously with each other on a user interface of the computer system.

11. One or more hardware storage devices having stored thereon computer-executable instructions that, when executed by one or more processors of a computer system, cause the computer system to:

receive raw image data that is comprised of an array of pixels, each pixel within the array of pixels comprising ultrasound information;

process the raw image data through a first fully convolutional network to generate a first segmentation label map, wherein:

the first segmentation label map comprises a first set of objects that have been segmented into at least a coarse segmentation class, and

each object within the first set of objects corresponds to a group of pixels from the array of pixels;

process the first segmentation label map through a second fully convolutional network to generate a second segmentation label map, wherein:

processing the first segmentation label map through the second fully convolutional network is performed using the raw image data as a base reference,

the second segmentation label map comprises a second set of objects that have been segmented into a fine segmentation class, and

each object within the second set of objects corresponds to a group of pixels from the array of pixels;

apply a contour optimization algorithm to at least one object within the second set of objects, wherein the contour optimization algorithm refines a corresponding contour boundary for the at least one object; and

generate an identification that the at least one object corresponds to a lymph node.

12. The one or more hardware storage devices of claim 11 , wherein applying the contour optimization algorithm includes applying a smoothness constraint on the corresponding boundary for the at least one object.

13. The one or more hardware storage devices of claim 11 , wherein applying the contour optimization algorithm includes applying a convexity shape constraint on the corresponding boundary for the at least one object.

14. The one or more hardware storage devices of claim 11 , wherein the first segmentation label map is a coarse label map, and wherein the second segmentation label map is a fine label map.

15. The one or more hardware storage devices of claim 11 , wherein the corresponding contour boundary for the at least one object initially includes an irregular portion, and wherein refining the corresponding contour boundary for the at least one object results in smoothing the irregular portion.

16. A method for segmenting tissue objects that are included within an ultrasound image, the method being implemented by one or more processors of a computer system, the method comprising:

receiving raw image data that is comprised of an array of pixels, each pixel within the array of pixels comprising ultrasound information;

processing the raw image data through a first fully convolutional network to generate a first segmentation label map, wherein:

the first segmentation label map comprises a first set of objects that have been segmented into at least a coarse segmentation class, and

each object within the first set of objects corresponds to a group of pixels from the array of pixels;

processing the first segmentation label map through a second fully convolutional network to generate a second segmentation label map, wherein:

processing the first segmentation label map through the second fully convolutional network is performed using the raw image data as a base reference,

the second segmentation label map comprises a second set of objects that have been segmented into a fine segmentation class, and

each object within the second set of objects corresponds to a group of pixels from the array of pixels;

applying a contour optimization algorithm to at least one object within the second set of objects, wherein the contour optimization algorithm refines a corresponding contour boundary for the at least one object; and

generating an identification that the at least one object corresponds to a lymph node.

17. The method of claim 16 , wherein all objects in the second set correspond only to lymph nodes.

18. The method of claim 16 , wherein applying the contour optimization algorithm includes applying a soft convex-shape constraint.

19. The method of claim 16 , wherein (1) the raw image data, (2) the first segmentation label map, and (3) the second segmentation label map are displayed simultaneously with each other on a user interface of the computer system.

20. The method of claim 19 , wherein a format of all objects in the second set is different than a format of all objects in the first set on the user interface.

Assignments (4)
CONFIRMATORY LICENSE Recorded May 23, 2019
From: UNIVERSITY OF NOTRE DAME
To: NATIONAL SCIENCE FOUNDATION
Reel/Frame 049283/0509 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 30, 2019
From: CHEN, DANNY ZIYI; YANG, LIN; ZHANG, YIZHE
To: UNIVERSITY OF NOTRE DAME DU LAC
Reel/Frame 049041/0208 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 30, 2019
From: YING, MICHAEL TIN-CHEUNG
To: HONG KONG POLYTECHNIC UNIVERSITY
Reel/Frame 049041/0243 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 30, 2019
From: AHUJA, ANIL TEJBHAN
To: CHINESE UNIVERSITY OF HONG KONG
Reel/Frame 049041/0253 →
Continuity (2)
Provisional Application 62432849 · Dec 12, 2016
Related Publication 20190304098A1 · Oct 3, 2019
Cited By (1)
US 12,488,469