IP Library › Granted Patent US 12694544

Granted Patent B2

US 12694544 · App. 18/689,443 · Granted Jul 28, 2026

Image processing device, image processing method, and surgical microscope system

Inventors: Junichiro Enoki (Tokyo, JP); Izumu Hosoi (Tokyo, JP); Tomoyuki Ootsuki (Tokyo, JP); Koji Kashima (Tokyo, JP)

Assignee: SONY GROUP CORPORATION

G06T7/337 A61B90/20 A61B90/37 G06T3/14 G06T3/20 G06T3/40 G06T5/73 G06T5/90 G06V10/25 G06T2207/10056 G06T2207/30041 G06T2207/30101 G06V2201/03

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 12694544
App. No.: 18/689,443
Granted: Jul 28, 2026
Kind: B2

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

There is provided an image processing device including a displayed-image generation unit that generates a displayed image in which a first reference image indicating reference coordinates of a preoperative image that is a captured image of an eyeball of a patient before surgery is superimposed on the preoperative image, and a second reference image indicating reference coordinates of an intraoperative image that is a captured image of the eyeball of the patient at the start of the surgery or during the surgery is superimposed on the intraoperative image, wherein the displayed-image generation unit arranges the first reference image and the second reference image such that the respective reference coordinates of the preoperative image and the intraoperative image correspond to each other, and such that a scale, a position, or an orientation of at least one of the preoperative image or the intraoperative image is changeable, to generate the displayed image.

Claims (75)

1 . An image processing device, comprising:

a displayed-image generation unit configured to:

arrange each of a first reference image, that indicates a first set of reference coordinates of a preoperative image, and a second reference image that indicates a second set of reference coordinates of an intraoperative image, wherein

the preoperative image is a first captured image of an eyeball of a patient before surgery,

the intraoperative image is a second captured image of the eyeball of the patient one of at a start of the surgery or during the surgery, and

the first reference image and the second reference image are arranged based on the first set of reference coordinates and the second set of reference coordinates;

generate a display image, based on the arrangement of the first reference image and the second reference image, in which the first reference image is superimposed on the preoperative image and the second reference image is superimposed on the intraoperative image, wherein

the first set of reference coordinates correspond to the second set of reference coordinates, and

at least one of a scale of at least one of the preoperative image or the intraoperative image, a position of the at least one of the preoperative image or the intraoperative image, or an orientation of the at least one of the preoperative image or the intraoperative image is changeable;

acquire at least one feature point of a plurality of feature points based on an image recognition operation on each of the preoperative image and the intraoperative image, wherein

the at least one feature point is on each of the preoperative image and the intraoperative image, and

the at least one feature point is common between the preoperative image and the intraoperative image;

superimpose a mark on the at least one feature point based on a result of the image recognition operation on each of the preoperative image and the intraoperative image; and

arrange a third reference image between the preoperative image on which the first reference image is superimposed in the generated display image, and the intraoperative image on which the second reference image is superimposed in the generated display image, wherein the third reference image indicates a position of the at least one feature point.

2 . The image processing device according to claim 1 , wherein the displayed-image generation unit is further configured to:

match a scale of the first set of reference coordinates of the preoperative image with a scale of the second set of reference coordinates of the intraoperative image;

match an orientation of the first set of reference coordinates of the preoperative image with an orientation of the second set of reference coordinates of the intraoperative image; and

arrange the preoperative image and the intraoperative image based on:

the match of the scale of the first set of reference coordinates with the scale of the second set of reference coordinates, and

the match of the orientation of the first set of reference coordinates with the orientation of the second set of reference coordinates.

3 . The image processing device according to claim 1 , wherein the displayed-image generation unit is further configured to arrange each of the preoperative image and the intraoperative image along one of a longitudinal direction of a display screen or a horizontal direction of the display screen.

4 . The image processing device according to claim 1 , wherein the displayed-image generation unit is further configured to convert each of the preoperative image and the intraoperative image into a vascular structure image.

5 . The image processing device according to claim 1 , wherein the displayed-image generation unit is further configured to at least one of turn, translate, enlarge, or reduce the at least one of the preoperative image or the intraoperative image based on one of the at least one feature point or a user operation.

6 . The image processing device according to claim 1 , wherein the displayed-image generation unit is further configured to superimpose the mark on the at least one feature point based on a user operation.

7 . The image processing device according to claim 1 , wherein the at least one feature point is a pattern of at least one of a scar of the eyeball of the patient or a blood vessel of the eyeball of the patient.

8 . The image processing device according to claim 1 , wherein the displayed-image generation unit is further configured to adjust, based on the result of the image recognition operation on each of the preoperative image and the intraoperative image, at least one of a contrast of the at least one of the preoperative image or the intraoperative image, a sharpness of the at least one of the preoperative image or the intraoperative image, a brightness of the at least one of the preoperative image or the intraoperative image, or a color of the at least one of the preoperative image or the intraoperative image.

9 . The image processing device according to claim 1 , wherein each of the first reference image and the second reference image includes one of a plurality of polar coordinates that includes a plurality of graduations along a circular ring, or a plurality of coordinates that includes one of a polygonal grid or a spider-web grid.

10 . The image processing device according to claim 1 , wherein the displayed-image generation unit is further configured to:

divide at least a part of each of the preoperative image and the intraoperative image into a plurality of regions; and

arrange the plurality of regions of each of the preoperative image and the intraoperative image, wherein

based on the arrangement of the plurality of regions, at least one of the scale of the preoperative image with respect to the first set of reference coordinates, the position of the preoperative image with respect to the first set of reference coordinates, or the orientation of the preoperative image with respect to the first set of reference coordinates corresponds to at least one of the scale of the intraoperative image with respect to the second set of reference coordinates, the position of the intraoperative image with respect to the second set of reference coordinates, or the orientation of the intraoperative image with respect to the second set of reference coordinates.

11 . The image processing device according to claim 10 , wherein the displayed-image generation unit is further configured to arrange the plurality of regions in at least one of a ring shape or a grid shape.

12 . The image processing device according to claim 1 , wherein the displayed-image generation unit is further configured to:

extract a first region of interest from the preoperative image;

extract a second region of interest from the intraoperative image; and

arrange the first region of interest adjacent to the second region of interest at respective positions.

13 . The image processing device according to claim 12 , wherein the displayed-image generation unit is further configured to arrange the first region of interest and the second region of interest alternatively.

14 . The image processing device according to claim 12 , wherein the displayed-image generation unit is further configured to extract each of the first region of interest and the second region of interest based on the result of the image recognition operation on each of the preoperative image and the intraoperative image.

15 . The image processing device according to claim 10 , wherein the displayed-image generation unit is further configured to:

arrange the plurality of regions in each of the preoperative image and the intraoperative image in a band shape;

adjust at least one of a scale of each of the plurality of regions, a position of each of the plurality of regions, or an orientation of each of the plurality of regions;

based on the adjustment of the at least one of the scale of each of the plurality of regions, the position of each of the plurality of regions, or the orientation of each of the plurality of regions, at least one of the scale of the preoperative image with respect to the first set of reference coordinates, the position of the preoperative image with respect to the first set of reference coordinates, or the orientation of the preoperative image with respect to the first set of reference coordinates corresponds to at least one of the scale of the intraoperative image with respect to the second set of reference coordinates, the position of the intraoperative image with respect to the second set of reference coordinates, or the orientation of the intraoperative image with respect to the second set of reference coordinates; and

generate a developed image that includes the plurality of regions, arranged in the band shape, in each of the preoperative image and the intraoperative image.

16 . An image processing method, comprising:

in an image processing device,

arranging each of a first reference image, that indicates a first set of reference coordinates of a preoperative image, and a second reference image that indicates a second set of reference coordinates of an intraoperative image, wherein

the preoperative image is a first captured image of an eyeball of a patient before surgery,

the intraoperative image is a second captured image of the eyeball of the patient one of at a start of the surgery or during the surgery, and

the first reference image and the second reference image are arranged based on the first set of reference coordinates and the second set of reference coordinates;

generating a display image, based on the arrangement of the first reference image and the second reference image, in which the first reference image is superimposed on the preoperative image and the second reference image is superimposed on the intraoperative image, wherein

the first set of reference coordinates correspond to the second set of reference coordinates, and

acquiring at least one feature point of a plurality of feature points based on an image recognition operation on each of the preoperative image and the intraoperative image, wherein

the at least one feature point is on each of the preoperative image and the intraoperative image, and

the at least one feature point is common between the preoperative image and the intraoperative image;

superimposing a mark on the at least one feature point based on a result of the image recognition operation on each of the preoperative image and the intraoperative image; and

arranging a third reference image between the preoperative image, on which the first reference image is superimposed in the generated display image, and the intraoperative image on which the second reference image is superimposed, wherein the third reference image indicates a position of the at least one feature point.

17 . A surgical microscope system, comprising:

a surgical microscope configured to acquire an operative-field image for an eye of a patient;

an image processing device configured to generate a display image based on the operative-field image; and

a display unit configured to display the display image, wherein

the image processing device includes a displayed-image generation unit that is configured to:

the operative-field image includes the preoperative image and the intraoperative image,

the preoperative image is a first captured image of an eyeball of the patient before surgery,

the intraoperative image is a second captured image of the eyeball of the patient one of at a start of the surgery or during the surgery, and

the first reference image and the second reference image are arranged based on the first set of reference coordinates and the second set of reference coordinates;

generate the display image, based on the arrangement of the first reference image and the second reference image, in which the first reference image is superimposed on the preoperative image and the second reference image is superimposed on the intraoperative image, wherein

the first set of reference coordinates correspond to the second set of reference coordinates, and

acquire at least one feature point of a plurality of feature points based on an image recognition operation on each of the preoperative image and the intraoperative image, wherein

the at least one feature point is on each of the preoperative image and the intraoperative image, and

the at least one feature point is common between the preoperative image and the intraoperative image;

superimpose a mark on the at least one feature point based on a result of the image recognition operation on each of the preoperative image and the intraoperative image; and

arrange a third reference image between the preoperative image, on which the first reference image is superimposed in the generated display image, and the intraoperative image on which the second reference image is superimposed, wherein the third reference image indicates a position of the at least one feature point.