Granted Patent B2

US 12,626,338 · App. 18/249,549 · Granted May 12, 2026

Inventors: Kai Walter (Schriesheim, DE); Constantin Kappel (Schriesheim, DE)

Assignee: LEICA MICROSYSTEMS CMS GMBH

G06T5/70G06T2207/10016G06T2207/10056G06T2207/20084G06T2207/20182

Abstract

Claims (31)

1 . A method for improving signal-to-noise of image frames, the method comprising:

estimating a representative velocity of an optical flow in an image frame sequence;

determining an interpolation factor from the representative velocity of the optical flow, wherein a ratio of the representative velocity and the interpolation factor is equal to or less than one pixel per frame;

employing a trained artificial neural network for generating an expanded image frame sequence, wherein the expanded image frame sequence comprises a number of interpolating image frames, wherein each interpolating image frame interpolates between subsequent image frames of the image frame sequence, wherein the number of interpolating image frames corresponds to the interpolation factor;

generating an output image frame sequence by computing a time-dependent combination of image frames from the expanded image frame sequence.

2 . The method of claim 1 , wherein estimating the representative velocity of the optical flow comprises:

calculating a histogram of the optical flow between successive image frames in the image frame sequence; and

analyzing the histogram to determine the representative velocity.

3 . The method of claim 2 , wherein analyzing the histogram to determine the representative velocity comprises employing the histogram to determine the representative velocity as a quantile for a predetermined threshold value.

4 . The method of claim 2 , wherein calculating the histogram is based on estimating a pixel-wise optical flow.

5 . The method of claim 4 , wherein estimating the pixel-wise optical flow is based on Farnebäck's algorithm.

6 . The method of claim 1 , wherein the image frames of the image frame sequence are microscopy image frames.

7 . The method of claim 1 , wherein the trained artificial neural network is configured for generating the interpolating image frames by applying a feature reshaping operation with channel attention.

8 . The method of claim 7 , wherein the feature reshaping operation comprises a pixel shuffle operation.

9 . The method of claim 1 , wherein the interpolation factor is a power of two and wherein the trained artificial neural network is configured for recursively generating and adding the interpolating image frames to the image frame sequence, wherein a number of recursions corresponds to the power.

10 . A method for improving signal-to-noise of image frames, the method comprising

estimating a representative velocity of an optical flow in an image frame sequence;

determining an interpolation factor based on the representative velocity of the optical flow;

employing a trained artificial neural network for generating an expanded image frame sequence, wherein the expanded image frame sequence comprises a number of interpolating image frames, wherein each interpolating image frame interpolates between subsequent image frames of the image frame sequence, wherein the number of interpolating image frames corresponds to the interpolation factor;

generating an output image frame sequence by computing a time-dependent combination of image frames by applying a rolling average to the expanded image frame sequence.

11 . The method of claim 10 , wherein the rolling average is a weighted rolling average and wherein applying the weighted rolling average comprises determining parameters for the weighted rolling average from the representative velocity.

12 . The method of claim 11 , wherein the parameters for the weighted rolling average are a window size determining a group of image frames from the expanded image frame sequence that contribute to the weighted rolling average, and a width determining a weight by which each image frame from the group of image frames contributes to the weighted rolling average.

13 . The method of claim 1 , further comprising applying a de-noising algorithm to the output image frame sequence.

14 . The method of claim 1 , wherein the trained artificial neural network has been trained by:

pre-training an artificial neural network with image frame sequences that are not domain specific; and

training the artificial neural network with domain-specific image frames.

15 . An image processing device for improving signal-to-noise of image frames by carrying out the method of claim 1 , the image processing device comprising:

a memory configured for saving an image frame sequence at least temporarily;

processing circuitry configured for estimating a representative velocity of an optical flow in the image frame sequence and for determining an interpolation factor from the representative velocity of the optical flow; and

a trained artificial neural network for generating an expanded image frame sequence, wherein the expanded image frame sequence comprises a predetermined number of interpolating image frames, wherein each interpolating image frame interpolates between subsequent image frames of the image frame sequence, wherein the predetermined number corresponds to the interpolation factor,

wherein the processing circuitry is further configured for computing a time-dependent combination of image frames from the expanded image frame sequence to generate an output image frame sequence.

Assignments (1)

Priority Claims (1)

Continuity (1)

References Cited (35)