IP Library › Granted Patent US 12,693,512
Granted Patent B2
US 12,693,512 · App. 18/909,298 · Granted Jul 28, 2026

Microscope-based system and method for image-guided microscopic illumination

Inventors: Jung-Chi Liao (Taipei City, TW); Yi-De Chen (Taipei, TW); Chih-Wei Chang (Taipei, TW); Weng Man Chong (Taipei City, TW)
Assignee: Academia Sinica
G02B21/365G02B21/06G02B21/26G06V10/141G06V10/25G06V20/693H04N23/56H04N23/74
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,693,512
App. No.
18/909,298
Filed
Oct 8, 2024
Granted
Jul 28, 2026
Kind
B2
Art Unit
2483
USPC
348/79
Abstract

A system and method for image-guided microscopic illumination are provided. A processing module controls an imaging assembly such that a camera acquires an image or images of a sample in multiple fields of view, and the image or images are automatically transmitted to a processing module and processed by the first processing module automatically in real-time based on a predefined criterion so as to determine coordinate information of an interested region in each field of view. The processing module also controls an illuminating assembly to illuminate the interested region of the sample according to the received coordinate information regarding to the interested region, with the illumination patterns changing among the fields of view.

Claims (34)

1 . A system for image-guided microscopic laser scanning of varying patterns on a sample through two or more fields of view of the sample, the system comprising at least one processing module configured to:

control an imaging assembly to acquire a first image of a first field of view of the sample, the first field of view being a subset of the sample;

process the first image automatically based on a predefined criterion, so as to determine coordinate information of a first region of interest in the first field of view;

control an illuminating assembly to direct a laser beam at the first field of view following a first pattern corresponding to the coordinate information of the first region of interest in the first field of view; and

control the imaging assembly to acquire a second image of a second field of view, the second field of view being a subset of the sample;

determine coordinate information of a second region of interest in the second field of view automatically based on the predefined criterion; and

control the illuminating assembly to direct a laser beam at the second field of view following a second pattern corresponding to coordinate information of the second region of interest in the second field of view,

wherein the second pattern is different from the first pattern.

2 . The system according to claim 1 , wherein the illuminating assembly is adapted to induce a photo-activated reaction in the first region of interest and the second region of interest.

3 . The system according to claim 1 , wherein the first region of interest and the second region of interest each comprises a plurality of regions of interest.

4 . The system according to claim 1 , wherein the imaging assembly comprises a camera and an imaging light source.

5 . The system according to claim 1 , wherein the illuminating assembly comprises a laser light source and a pattern illumination device.

6 . The system according to claim 1 , wherein the microscope comprises an objective, the at least one processing module being further configured to control the objective to move among the first field of view and the second field of view, and wherein the at least one processing module is further configured to move a stage of a microscope among the first field of view and the second field of view.

7 . The system according to claim 1 , wherein the sample comprises live cells, the at least one processing module being configured to direct the laser beam at the live cells according to the coordinate information of the first region of interest in the first field of view and the second region of interest in the second field of view, and wherein the coordinate information of the first region of interest in first field of view is different from the coordinate information of the second region of interest in the second field of view.

8 . The system according to claim 1 , wherein the sample comprises fixed cells, the at least one processing module being configured to direct the laser beam at the fixed cells according to the coordinate information of the first region of interest in the first field of view and the coordinate information of the second region of interest in the second field of view, and wherein the coordinate information of the first region of interest in first field of view is different from the coordinate information of the second region of interest in the second field of view.

9 . The system according to claim 1 , wherein the predefined criterion comprises a morphology, intensity, contrast, or specific features of the sample.

10 . The system according to claim 1 , wherein the sample comprises a cell or a tissue sample.

11 . The system according to claim 1 , wherein the illuminating assembly is adapted to induce a photochemical reaction in the first region of interest and the second region of interest.

12 . A method for image-guided microscopic laser scanning of varying patterns on a sample through two or more fields of view of the sample, comprising:

acquiring a first image of a first field of view, the first field of view being a subset of the sample;

based on a predefined criterion, processing the first image automatically to determine coordinate information of a first region of interest in the first field of view;

directing a laser beam at the first field of view following a first pattern corresponding to the coordinate information of the first region of interest in the first field of view;

acquiring a second image of a second field of view;

determining coordinate information of a second region of interest in the second field of view automatically based on the predefined criterion; and

directing a laser beam at the second field of view following a second pattern corresponding to the coordinate information of the second region of interest in the second field of view;

wherein the second pattern is different from the first pattern.

13 . The method according to claim 12 , wherein the predefined criterion comprises a morphology, intensity, contrast, or specific features of the sample.

14 . The method according to claim 12 , wherein the sample comprise a cell or a tissue sample.

15 . The method according to claim 12 , wherein the sample is disposed on a stage of a microscope, the method further comprising controlling an objective of the microscope to move among the first field of view and the second field of view.

16 . The method according to claim 12 , wherein the sample comprises a live cell, wherein the method comprises directing the laser beam at the live cell according to the coordinate information of the first region of interest in the first field of view and the second region of interest in the second field of view, and wherein the coordinate information of the first region of interest in first field of view is different from the coordinate information of the second region of interest in the second field of view.

17 . The method according to claim 12 , wherein the sample comprises a fixed cell, wherein the method comprises directing the laser beam at the fixed cell according to the coordinate information of the first region of interest in the first field of view and the second region of interest in the second field of view, and wherein the coordinate information of the first region of interest in first field of view is different from the coordinate information of the second region of interest in the second field of view.

18 . The method according to claim 12 , wherein the first region of interest and the second region of interest each comprises a plurality of regions of interest.

19 . The method according to claim 12 , wherein directing the laser beam at the first field of view comprises inducing a photochemical reaction in the first region of interest, and wherein directing the laser beam at the second field of view comprises inducing a photochemical reaction in the second region of interest.

20 . The method according to claim 12 , wherein directing the laser beam at the first field of view comprises inducing a photo-activated reaction in the first region of interest, and wherein directing the laser beam at the second field of view comprises inducing a photo-activated reaction in the second region of interest.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 15, 2025
From: LIAO, JUNG-CHI; CHEN, YI-DE; CHANG, CHIH-WEI; CHONG, WENG MAN
To: ACADEMIA SINICA
Reel/Frame 071718/0512 →
Continuity (5)
Continuation 18464125 · Sep 8, 2023
Continuation 17592211 · Feb 3, 2022
Continuation 16013663 · Jun 20, 2018
Provisional Application 62522265 · Jun 20, 2017
Related Publication 20250028162A1 · Jan 23, 2025
References Cited (124)
US 5646413A · Nishi · 1997 [cited by applicant]
US 6522774B1 · Bacus et al. · 2003 [cited by applicant]
US 6614031B2 · Engelhardt et al. · 2003 [cited by applicant]
US 6697526B1 · Abe · 2004 [cited by applicant]
US 6947583B2 · Ellis et al. · 2005 [cited by applicant]
US 6956647B2 · Foster et al. · 2005 [cited by applicant]
US 7133543B2 · Verwoerd et al. · 2006 [cited by applicant]
US 7460732B2 · Recht · 2008 [cited by applicant]
US 7639357B2 · Okugawa · 2009 [cited by applicant]
US 7817273B2 · Bahatt et al. · 2010 [cited by applicant]
US 8187866B2 · Duer · 2012 [cited by applicant]
US 8288157B2 · Duer · 2012 [cited by applicant]
US 8923568B2 · Olson et al. · 2014 [cited by applicant]
US 9019598B2 · Tsurumune · 2015 [cited by applicant]
US 9030547B2 · Vizi et al. · 2015 [cited by applicant]
US 9125562B2 · Spencer et al. · 2015 [cited by applicant]
US 9217694B2 · Sieckmann et al. · 2015 [cited by applicant]
US 9528939B2 · Duer · 2016 [cited by applicant]
US 9557156B2 · Kankaria · 2017 [cited by applicant]
US 9581801B2 · Takamizawa · 2017 [cited by applicant]
US 9696535B2 · Prakash et al. · 2017 [cited by applicant]
US 9788790B2 · Black et al. · 2017 [cited by applicant]
US 9810892B2 · Prakash et al. · 2017 [cited by applicant]
US 9939381B1 · Kimmel et al. · 2018 [cited by applicant]
US 10517482B2 · Sato et al. · 2019 [cited by applicant]
US 10729326B2 · Spencer et al. · 2020 [cited by applicant]
US 10746980B2 · Kenny et al. · 2020 [cited by applicant]
US 10932670B2 · Smith et al. · 2021 [cited by applicant]
US 11265449B2 · Liao et al. · 2022 [cited by applicant]
US 11408821B2 · Boamfa et al. · 2022 [cited by applicant]
US 11513328B2 · Pannhoff et al. · 2022 [cited by applicant]
US 11789251B2 · Liao et al. · 2023 [cited by applicant]
US 20020008904A1 · Engelhardt · 2002 [cited by applicant]
US 20030189676A1 · Kato et al. · 2003 [cited by applicant]
US 20030231791A1 · Torre-Bueno et al. · 2003 [cited by applicant]
US 20040105000A1 · Yuri · 2004 [cited by applicant]
US 20050072920A1 · Inada · 2005 [cited by applicant]
US 20050082494A1 · Motomura · 2005 [cited by applicant]
US 20050089208A1 · Dong et al. · 2005 [cited by applicant]
US 20050179899A1 · Wasserman et al. · 2005 [cited by applicant]
US 20060028717A1 · Dunn · 2006 [cited by applicant]
US 20060226375A1 · Maruo · 2006 [cited by applicant]
US 20070152130A1 · Fomitchov · 2007 [cited by applicant]
US 20080206752A1 · Balakirev et al. · 2008 [cited by applicant]
US 20090195646A1 · Ganser et al. · 2009 [cited by applicant]
US 20090316259A1 · Kenny et al. · 2009 [cited by applicant]
US 20100079857A1 · Sasaki et al. · 2010 [cited by applicant]
US 20100278399A1 · Bednarkiewicz et al. · 2010 [cited by applicant]
US 20110050745A1 · Jung et al. · 2011 [cited by applicant]
US 20120242817A1 · Pan · 2012 [cited by applicant]
US 20130250391A1 · Kato · 2013 [cited by examiner]
US 20130294645A1 · Sibarita · 2013 [cited by applicant]
US 20140022373A1 · Kanarowski et al. · 2014 [cited by applicant]
US 20140152794A1 · Takahashi · 2014 [cited by applicant]
US 20140293407A1 · Amano et al. · 2014 [cited by applicant]
US 20150168702A1 · Harris · 2015 [cited by applicant]
US 20150185456A1 · Kishima · 2015 [cited by applicant]
US 20150258352A1 · Lin et al. · 2015 [cited by applicant]
US 20160124208A1 · Best et al. · 2016 [cited by applicant]
US 20160302740A1 · Iyer et al. · 2016 [cited by applicant]
US 20170059407A1 · Shiraiwa · 2017 [cited by applicant]
US 20170090176A1 · Pospiech et al. · 2017 [cited by applicant]
US 20170154236A1 · Masuura et al. · 2017 [cited by applicant]
US 20170212342A1 · Rozsa et al. · 2017 [cited by applicant]
US 20170248837A1 · Sato et al. · 2017 [cited by applicant]
US 20170299784A1 · Mikkelsen et al. · 2017 [cited by applicant]
US 20180136451A1 · Soenksen · 2018 [cited by applicant]
US 20180177401A1 · Yang et al. · 2018 [cited by applicant]
US 20180180642A1 · Shetty et al. · 2018 [cited by applicant]
US 20180180885A1 · Holland et al. · 2018 [cited by applicant]
US 20180203221A1 · Dai et al. · 2018 [cited by applicant]
US 20180210183A1 · Nishikawa · 2018 [cited by applicant]
US 20180284419A1 · Peschka et al. · 2018 [cited by applicant]
US 20180348500A1 · Naaman et al. · 2018 [cited by applicant]
US 20190324240A1 · Shroff et al. · 2019 [cited by applicant]
US 20190339456A1 · Ruggles · 2019 [cited by applicant]
US 20190391078A1 · Cohen et al. · 2019 [cited by applicant]
US 20210224954A1 · Guo et al. · 2021 [cited by applicant]
US 20230418041A1 · Liao et al. · 2023 [cited by applicant]
US 20240219703A1 · Liao et al. · 2024 [cited by applicant]
US 20240219704A1 · Liao et al. · 2024 [cited by applicant]
US 20240219705A1 · Liao et al. · 2024 [cited by applicant]
US 20240411122A1 · Liao et al. · 2024 [cited by applicant]
US 20250035910A1 · Liao et al. · 2025 [cited by applicant]
US 20250052993A1 · Liao et al. · 2025 [cited by applicant]
TW 201013817A · 2010 [cited by applicant]
WO WO2016133787A1 · 2016 [cited by applicant]
Alqattan et al.; Direct laser writing of nanophotonic structures on contact lenses; ACS nano; 12(6); pp. 5130-5140; Apr. 2018. [cited by applicant]
Aznar et al.; Gated materials for on-command release of guest molecules; Chemical reviews; 116(2); pp. 561-718; Jan. 27, 2016. [cited by applicant]
Bar et al.; Biotinylation by antibody recognition—a method for proximity labeling; Nature Methods; 15(2); pp. 127-133; (author Manuscript); 19 pages; Feb. 2018. [cited by applicant]
Brieke et al.; Light?controlled tools; Angewandte Chemie International Edition; 51(34); pp. 8446-8476; Aug. 20, 2012. [cited by applicant]
Chen et al.; Lattice light-sheet microscopy: imaging molecules to embryos at high spatiotemporal resolution; Science; 346(6208); 1257998; doi:10.1126/scince.1257998; (Author Manuscript); 28 pages; Oct. 2014. [cited by applicant]
Ezzoukhry et al.; Combining laser capture microdissection and proteomics reveals an active translation machinery controlling formation; Nature Communications; 9(1); doi:10.1038/s41467-018-04461-9; 11 pages; May 2018. [cited by applicant]
Hadley et al.; Determining composition of micron-scale protein deposits in neurodegenerative disease by spatially targeted optical microproteomics; eLife; 21 pages. doi: 10.7554/eLife.09579; Sep. 2015. [cited by applicant]
Hirsch et al.; Easily reversible desthiobiotin binding to streptavidin, avidin, and other bitin-binding proteins: uses for protein labeling, detection, and isolation; Analytical Biochemistry; 308(2); pp. 343-357; Sep. 2… [cited by applicant]
Hung et al.; Protein localization in disease and therapy; Journal of Cell Science; 124(20); pp. 3381-3392; Oct. 2011. [cited by applicant]
Hung et al.; Spatially resolved proteomic mapping in living cells with the engineered peroxidase APEX2; Nature Protocols; 11(3); pp. 456-475; (Author Manuscript); 39 pages; Mar. 2016. [cited by applicant]
Itzhak et al.; A mass spectrometry-based approach for mapping protein neurons; Cell Reports; 20(11); pp. 2706-2718; Sep. 2017. [cited by applicant]
Jain et al.; ATPase-modulated stress granules contain a diverse proteome and substructure; Cell; 164(3); pp. 487-498; Jan. 2016. [cited by applicant]
Kamburov et al.; The consensusPathDB interaction database: 2013 update; Nucleic Acids Research; 41(D1); pp. D793-D800; Jan. 2013. [cited by applicant]
Kepiro et al.; Molecular tattoo: subcellular confinement of drug effects; Chemistry and Biology; 22(4); pp. 548-558; Apr. 2015. [cited by applicant]
Lam et al.; Directed evolution of APEX2 for electron microscopy and proximity labeling; Nature Methods; 12(1); pp. 51-54; (author Manuscript); 15 pages; Jan. 2015. [cited by applicant]
Liao et al.; A pioneering high content cell image-registered protein labeling system; 2017 Bio Taiwan Exhibition and Conference; 2 pages; Jun. 29-Jul. 2, 2017. [cited by applicant]
Liao et al.; High content cell image-registered protein labeling system; 2017 Bio Taiwan Exhibition and Conference; 14 pages; Jun. 29-Jul. 2, 2017. [cited by applicant]
Liu et al; Supplementary materials: super-resolution labeling with action-paint; Nature Chemistry; Supplementary Information; doi.org/10.1038/s41557-019-0325-7; 43 pages retrieved from the internet (https://yin.hms.harv… [cited by applicant]
Prier et. al.; Visible light photoredox cataysis with transition metal complexes: applications in organic synthesis; Chemical reviews; 113(7); pp. 5322-5363; (author Manuscript) 141 pages; Jul. 2013. [cited by applicant]
Rhee et al.; Proteomic mapping of mitochondria in living cells via spatially restricted enzymatic tagging; Science; 339(6125); pp. 1328-1331; (Author Manuscript); 8 pages; Mar. 2013. [cited by applicant]
Rost et al.; Multiplexed ion beam imaging analysis for quanitation of protein expression in cancer tissue sections; Laboratory Investigation; 97(8); pp. 992-1003; Aug. 2017. [cited by applicant]
Roux et al.; A promiscuous biotin ligase fusion protein identifies proximal and interacting proteins in mammalian cells; Journal of Cell Biology; 196(6); pp. 801-810; Mar. 2012. [cited by applicant]
Roux et al.; BioID: A screen for protein-protein interactions: Current Protocols in protein Science; 91(1); pp. 19-23; (Author Manuscript); 20 pages; Jan. 2018. [cited by applicant]
Thermofisher Scientific: Sulfo-SBED biotin label transfer reagent; (Product Discription); retrieved from the internet (https://www.thermofisher.com/order/catalog/product/33033#/33033); 5 pages; on May 5, 2021. [cited by applicant]
Thul et al.; A subcellular map of the human proteome; Science; 356(6340) eaal3321; 14 pages; May 2017. [cited by applicant]
Zipfel et al.; Nonlinear magic: multiphoton microscopy in the biosciences; Nature Biotechnolgy: 21(11); pp. 1369-1377; Nov. 2003. [cited by applicant]
Huang et al.; U.S. Appl. No. 18/748,710 entitled “Adjustable Support and Optical Module,” filed Jun. 20, 2024. [cited by applicant]
Liao et al.; U.S. Appl. No. 18/909,323 entitled “Microscope-based system and method for image-guided microscope illumination,” filed Oct. 8, 2024. [cited by applicant]
Ronneberger et al.; U-net: Convolutional networks for biomedical image segmentation. InMedical image computing and computer-assisted interventionRMICCAI 2015; 18th international conference; Springer international publis… [cited by applicant]
Stegmaier et al.; Real-time three-dimensional cell segmentation in large-scale microscopy data of developing embryos. Developmental Cell; 36(2); pp. 225-240; Jan. 25, 2016. [cited by applicant]
Wikipedia; Biotinylation; 4 pages; retrieved from the internet (https://en.wikipedia.org/w/index.php?title=Biotinylation&oldid=841539236) on May 16, 2018. [cited by applicant]
Liao et al.; U.S. Appl. No. 18/864,988 entitled “Method of calibrating a microscope system,” filed Nov. 12, 2024. [cited by applicant]
Liao et al.; U.S. Appl. No. 18/864,997 entitled “Method of calibrating a microscope system,” filed Nov. 12, 2024. [cited by applicant]
Chen et al.; U.S. Appl. No. 18/877,609 entitled “Microscope-based system and method using a UV-transmission mirror,” filed Dec. 20, 2024. [cited by applicant]
Liao et al.; U.S. Appl. No. 18/997,408 entitled “Training AI model for a microscope-based pattern photoillumination system,” filed Jan. 21, 2025. [cited by applicant]
Liao et al.; U.S. Appl. No. 18/997,412 entitled “Method of standardizing image pixel values in a microscope-based system,” filed Jan. 21, 2025. [cited by applicant]
Liao et al.; U.S. Appl. No. 19/110,325 entitled “Microscope-based system and method of determining beam processing path,” filed Mar. 10, 2025. [cited by applicant]