IP Library Granted Patent US 12,249,062
Granted Patent B2
US 12,249,062 · App. 16/430,175 · Granted Mar 11, 2025

Techniques for segmentation of lymph nodes, lung lesions and other solid or part-solid objects

Inventors: Yongqiang Tan (Forest Hills, NY); Binsheng Zhao (Forest Hills, NY); Lawrence H. Schwartz (New York, NY)
Assignee: The Trustees of Columbia University in the City of New York
G06T7/0012A61B5/4887G06T7/10G06T7/12G06T7/136G06T7/149G06T7/155G06T7/62G06T7/66G06T7/70G06T7/77G06T11/008G06T15/06G06V10/22G06V10/225G06V10/245G06V10/26G06V10/267G06V10/42G06V10/44G06V10/755G06V20/695A61B5/004A61B5/4222A61B5/4244A61B5/4842A61B5/489A61B8/08A61B8/0808A61B8/0833A61B8/085A61B2576/02A61B2576/026G06F18/295G06T5/40G06T5/70G06T7/13G06T7/143G06T2200/04G06T2207/10056G06T2207/10072G06T2207/10081G06T2207/10116G06T2207/10136G06T2207/20008G06T2207/20012G06T2207/20036G06T2207/20041G06T2207/20112G06T2207/20116G06T2207/20152G06T2207/20161G06T2207/20168G06T2207/30004G06T2207/30008G06T2207/30016G06T2207/30024G06T2207/30032G06T2207/30056G06T2207/30061G06T2207/30064G06T2207/30084G06T2207/30096G06T2207/30101G06T2207/30208G06T2210/41G06V10/50G06V10/764G06V10/85G06V20/698G06V2201/03G06V2201/031G06V2201/032
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,249,062
App. No.
16/430,175
Granted
Mar 11, 2025
Kind
B2
Abstract

Techniques for segmentation include determining an edge of voxels in a range associated with a target object. A center voxel is determined. Target size is determined based on the center voxel. In some embodiments, edges near the center are suppressed, markers are determined based on the center, and an initial boundary is determined using a watershed transform. Some embodiments include determining multiple rays originating at the center in 3D, and determining adjacent rays for each. In some embodiments, a 2D field of amplitudes is determined on a first dimension for distance along a ray and a second dimension for successive rays in order. An initial boundary is determined based on a path of minimum cost to connect each ray. In some embodiments, active contouring is performed using a novel term to refine the initial boundary. In some embodiments, boundaries of part-solid target objects are refined using Markov models.

Claims (68)

1. A method comprising:

obtaining, on a processor, image data that indicates amplitude values at each of a plurality of voxels for a particular measurement modality;

determining, on a processor, an outer edge of a contiguous subset of voxels based on voxels that have an amplitude in a first amplitude range associated with a target object in image data of the particular measurement modality;

determining, on a processor, a center voxel, wherein a center distance from the center voxel to a closest voxel of the outer edge is greater than a distance from any neighbor of the center voxel to a corresponding closest voxel of the outer edge;

generating new image data by causing, on a processor, a location and size of the target object to be marked in the image data based on the center voxel and the center distance further comprising

determining, on a processor, a gradient value at a voxel; and

determining, on a processor, a reduced gradient equal to a product of the gradient with a fraction f that decreases with decreasing distance of the voxel from the center voxel,

wherein the new image data is based at least in part on the reduced gradient; and

presenting on a display device output image data based at least in part on the new image data with the location and size of the target object marked.

2. The method as recited in claim 1 , further comprising determining the first amplitude range associated with the target object based on a Gaussian mixture model.

3. The method as recited in claim 1 , further comprising:

determining, on a processor, a marker based on the location and size of the target object; and

determining on a processor a watershed boundary based on the marker,

wherein the new image data is based at least in part on the watershed boundary.

4. A method comprising:

obtaining, on a processor, image data that indicates amplitude values at each of a plurality of voxels for a particular measurement modality;

determining, on a processor, an outer edge of a contiguous subset of voxels based on voxels that have an amplitude in a first amplitude range associated with a target object in image data of the particular measurement modality;

determining, on a processor, a center voxel, wherein a center distance from the center voxel to a closest voxel of the outer edge is greater than a distance from any neighbor of the center voxel to a corresponding closest voxel of the outer edge;

generating new image data by causing, on a processor, a location and size of the target object to be marked in the image data based on the center voxel and the center distance further comprising

determining, on a processor, a gradient value at each voxel;

determining, on a processor, a reduced gradient equal to a product of the gradient with a fraction f that decreases with decreasing distance of each voxel from the center voxel; and

determining, on a processor, a watershed boundary based, at least in part, on the reduced gradient instead of the gradient at voxels within the center distance from the center voxel; and

presenting on a display device output image data based at least in part on the new image data with the location and size of the target object marked on the watershed boundary.

5. A method comprising:

obtaining, on a processor, image data that indicates amplitude values at each of a plurality of voxels for a particular measurement modality;

determining, on a processor, an outer edge of a contiguous subset of voxels based on voxels that have an amplitude in a first amplitude range associated with a target object in image data of the particular measurement modality;

determining, on a processor, a center voxel, wherein a center distance from the center voxel to a closest voxel of the outer edge is greater than a distance from any neighbor of the center voxel to a corresponding closest voxel of the outer edge;

determining, on a processor, a location and size of the target object in the image data based on the center voxel and the center distance further comprising determining, on a processor, a refined boundary by moving an initial boundary based on active contouring using as the initial boundary at least one of a marker as the center voxel and the center distance or a watershed boundary; and

presenting on a display device output image data based at least in part on the refined boundary,

wherein

the active contouring of the initial boundary further comprises determining a change in boundary position based, at least in part, on a potential well term of an equation comprising the potential well term and one or more other terms, and wherein the potential well term is weighted by a parameter α independently of the other terms of the equation wherein α has a value greater than 1; and

the potential well term comprises a product of a gradient of an edge indicator function g and a gradient of a level set function ϕ.

6. The method as recited in claim 5 , wherein:

the active contouring of the initial boundary further comprises determining a change in boundary position based, at least in part, on a topographical distance term, (Li-Le), wherein

topographical distance L is a minimum accumulated weighted change in amplitude,

Li is a topographical distance to a boundary based on eroding the initial boundary by a plurality of voxels, and Le is a topographical distance to a boundary based on dilating the initial boundary by a plurality of voxels.

7. A method comprising:

obtaining, on a processor, image data that indicates amplitude values at each of a plurality of voxels for a particular measurement modality;

determining, on a processor, an outer edge of a contiguous subset of voxels based on voxels that have an amplitude in a first amplitude range associated with a target object in image data of the particular measurement modality;

determining, on a processor, a center voxel, wherein a center distance from the center voxel to a closest voxel of the outer edge is greater than a distance from any neighbor of the center voxel to a corresponding closest voxel of the outer edge;

determining, on a processor, a location and size of the target object in the image data based on the center voxel and the center distance further comprising determining, on a processor, a refined boundary by moving an initial boundary based on active contouring using as the initial boundary at least one of a marker as the center voxel and the center distance or a watershed boundary; and

presenting on a display device output image data based at least in part on the location and size of the target object or the refined boundary,

wherein:

the active contouring of the initial boundary further comprises determining a change in boundary position based, at least in part, on a potential well term independently weighted by a parameter α,

the potential well term comprises a product of a gradient of an edge indicator function g and a gradient of a level set function ϕ,

the active contouring of the initial boundary further comprises determining a change in boundary position based, at least in part, on a volume preserving mean curvature term of an equation comprising the volume preserving mean curvature term and the potential well term and wherein the volume preserving mean curvature term is independently weighted by an independent parameter β such that the potential well term is not weighted by the independent parameter β; and

the volume preserving mean curvature term comprises a product of a magnitude of the gradient of the level set function ϕ and a difference between a local mean curvature κ and a mean curvature for the initial boundary κmean.

8. A method comprising:

obtaining, on a processor, image data that indicates amplitude values at each of a plurality of voxels for a particular measurement modality;

determining, on a processor, an outer edge of a contiguous subset of voxels based on voxels that have an amplitude in a first amplitude range associated with a target object in image data of the particular measurement modality;

determining, on a processor, a center voxel, wherein a center distance from the center voxel to a closest voxel of the outer edge is greater than a distance from any neighbor of the center voxel to a corresponding closest voxel of the outer edge;

determining, on a processor, a location and size of the target object in the image data based on the center voxel and the center distance further comprising determining, on a processor, a refined boundary by moving an initial boundary based on active contouring using as the initial boundary at least one of a marker as the center voxel and the center distance or a watershed boundary; and

presenting on a display device output image data based at least in part on the location and size of the target object or the refined boundary,

wherein performing active contouring of the initial boundary comprises using an evolution equation that includes a first term that indicates deviations in amplitude for all voxels inside the boundary from a statistical metric of amplitude of all voxels inside the boundary and a second term that indicates deviations in amplitude for local voxels outside the boundary from a statistical metric of amplitude of the local voxels outside the boundary, wherein the local voxels outside the boundary exclude a larger number of voxels outside the initial boundary.

9. A non-transitory computer-readable medium carrying one or more sequences of instructions, wherein execution of the one or more sequences of instructions by one or more processors causes the one or more processors to perform the steps of claim 1 .

10. A system comprising:

a medical imaging device;

a display device;

at least one processor; and

at least one memory including one or more sequences of instructions,

the at least one memory and the one or more sequences of instructions configured to, with the at least one processor, cause an apparatus to perform at least the steps of claim 1 .

11. A non-transitory computer-readable medium carrying one or more sequences of instructions, wherein execution of the one or more sequences of instructions by one or more processors causes the one or more processors to perform the steps of claim 4 .

12. A system comprising:

a medical imaging device;

a display device;

at least one processor; and

at least one memory including one or more sequences of instructions,

the at least one memory and the one or more sequences of instructions configured to, with the at least one processor, cause an apparatus to perform at least the steps of claim 4 .

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 6, 2019
From: TAN, YONGQIANG; ZHAO, BINSHENG; SCHWARTZ, LAWRENCE H.
To: THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITY OF NEW YORK
Reel/Frame 050291/0056 →
Continuity (4)
Continuation 14394098
Provisional Application 61622989 · Apr 11, 2012
Provisional Application 61622998 · Apr 11, 2012
Related Publication 20190355117A1 · Nov 21, 2019
References Cited (40)
US 5551431A · Wells, III et al. · 1996 [cited by applicant]
US 6259802B1 · Jolly et al. · 2001 [cited by applicant]
US 6694163B1 · Vining · 2004 [cited by applicant]
US 6898303B2 · Armato, III et al. · 2005 [cited by applicant]
US 7454045B2 · Yao et al. · 2008 [cited by applicant]
US 7822266B2 · Wellington et al. · 2010 [cited by applicant]
US 8150120B2 · Gindele et al. · 2012 [cited by applicant]
US 8355552B2 · Schwartz et al. · 2013 [cited by applicant]
US 8435181B2 · Yang et al. · 2013 [cited by applicant]
US 8884618B2 · Mahfouz · 2014 [cited by applicant]
US 10354377B2 · Tan · 2019 [cited by examiner]
US 20080317314A1 · Schwartz et al. · 2008 [cited by applicant]
US 20100027861A1 · Shekhar · 2010 [cited by examiner]
US 20100183225A1 · Vantaram et al. · 2010 [cited by applicant]
US 20110026798A1 · Madabhushi et al. · 2011 [cited by applicant]
Shang et al., Liver segmentation by an active Contour model with embedded Gaussian mixture model based classifiers, May 5, 2010 [retrieved Dec. 31, 2020], Proceedings, Optics, Photonics, and Digital Technologies for Mul… [cited by examiner]
Xiao et al., Fast level set image and video segmentation using new evolution indicator operators, published on-line Feb. 9, 2012 [retrieved Aug. 20, 2021], The Visual Computer, vol. 29, pp. 27-39. Retrieved: https://lin… [cited by examiner]
Weiguang et al., Deformable registration of digital images, May 1998 [retrieved Aug. 20, 2021], Journal of Computer Science and Technology, vol. 13, issue 3, Article No. 246, pp. 246-260. Retrieved: https://link.springe… [cited by examiner]
Maintz et al., A survey of medical image registration, Mar. 1998 [retrieved Aug. 20, 2021], Medical Image Analysis, vol. 2, Issue: 2, pp. 1-36. Retrieved: https://www.sciencedirect.com/science/article/pii/S1361841501800… [cited by examiner]
Rogowska et al., Evaluation of Selected Two-Dimensional Segmentation Techniques for Computed Tomography Quantitation of Lymph Nodes [text-version], Mar. 1996 [retrieved Aug. 20, 2021], Investigative Radiology, vol. 31, … [cited by examiner]
[Item X continued] https://journals.lww.com/investigativeradiology/Fulltext/1996/03000/Evaluation_of_Selected_Two_Dimensional.4. aspx (Year: 1996). [cited by examiner]
Kass et al., Snakes: Active Contour Models, Jan. 1988 [retrieved Aug. 20, 2021], International Journal of Computer Vision , vol. 1, pp. 321-331. Retrieved: https://link.springer.com/article/10.1007/BF00133570 (Year: 198… [cited by examiner]
Huisken, The volume preserving mean curvature flow, 1987 [retrieved Aug. 20, 2021], Journal fur die reine und angewandte Mathematik, vol. 382, 15 pages (Year: 1987). [cited by examiner]
Xie et al., Image Gradient Based Level Set Methods in 2D and 3D, 2013 [1st Online Oct. 30, 2012 & retrieved Oct. 27, 2024], Deformation Models, vol. 7, 20 pages. DOI: 10.1007/978-94-007-5446-1_4 (Year: 2012). [cited by examiner]
Dufour et al., Segmenting and Tracking Fluorescent Cells in Dynamic 3-D Microscopy With Coupled Active Surfaces, Sep. 2005 [retrieved Oct. 27, 2024], IEEE Transactions on Image Processing, vol. 14, Issue: 9, 16 pages. D… [cited by examiner]
Gibou, et al., “A Fast Hybrid k-Means Level Set Algorithm For Segmentation”, 2002, pp. 1-11, Publisher: 4th Annual Hawaii International Conference on Statistics and Mathematics. [cited by applicant]
ISA/US, “International Search Report and Written Opinion for the corresponding PCT application No. 2013/036258”, Mar. 6, 2013, pp. 1-11. [cited by applicant]
Lu, et al., “Fast image motion segmentation for surveillance applications”, Image and Vision Computing, 2011, pp. 104-116, vol. 29. [cited by applicant]
Zhu, et al., “Automatic Segmentation of Ground-Glass Opacities in Lung CT Images by Using Markov Random Field-Based Algorithms”, J Digit Imaging, 2011, pp. 409-422, vol. 25. [cited by applicant]
Yang et al., “Automatic centerline extraction of coronary arteries in coronary computed tomographic angiography,” The International Journal of Cardiovascular Imaging, 2012, pp. 921-933, vol. 28, Issue: 5. [cited by applicant]
Delingette, et al. “Shape and Topology Constraints on Parametric Active Contours”, Computer Vision and Image Understanding, 2001, pp. 140-171, vol. 83. [cited by applicant]
Elter, et al., “Contour tracing for segmentation of mammographic masses”, Phys. Med. Biol, 2010, pp. 5299-5315, vol. 55. [cited by applicant]
Esedoglu, et al., “A Variational Formulation for a Level Set Representation of Multiphase Flow and Area Preserving Curvature Flow”, Commun. Math. Sci., 2008, pp. 125-148, vol. 6, No. 1. [cited by applicant]
Geets, et al, “A gradient-based method for segmenting FDG-PET images: methodology and validation”, Eur. J. Nucl. Med. Mol. Imaging, 2007, pp. 1427-1428, vol. 34. [cited by applicant]
Nguyen, et al., “Watersnakes: Energy-Driven Watershed Segmentation”, IEEE Transactions on Patter Analysis and Machine Intelligence, 2003, pp. 330-342, vol. 25, No. 3. [cited by applicant]
Strickland, et al., “Development of Subject-Specific Geometric Spine Model through Use of Automated Active Contour Segmentation and Kinematic Constraint-Limited Registration”, Journal of Digital Imaging, 2011, pp. 926-9… [cited by applicant]
Thurey, et al. “A Multiscale Approach to Mesh-based Surface Tensions Flows”, ACM Transactions on Graphics, 2010, pp. 1-10, vol. 29, No. 4. [cited by applicant]
Yezzi, et al., “A Geometric Snake Model for Segmentation of Medical Imagery”, IEEE Transactions on Medical Imaging, 1997, pp. 199-209, vol. 16, No. 2. [cited by applicant]
Zhang, et al. “Active contours driven by local image fitting energy”, Pattern Recognition, 2010, pp. 1199-1206, vol. 43. [cited by applicant]
Vincent, et al., “Watersheds in Digital Spaces: An Efficient Algorithm Based on Immersion Simulations”, IEEE Transactions on Medical Imaging, 1991, pp. 583-598, vol. 13, No. 6. [cited by applicant]
Cited By (1)
US 12,469,159