Method for identifying analytes in an image series

1 . A method for identifying analytes in an image series, the image series being generated by marking analytes with markers in multiple coloring rounds and detecting the markers using a camera, the camera recording an image of the image series in each coloring round, the markers being selected in such a way that image signals of an analyte in an image area over the image series include colored signals and uncolored signals, comprising:

extracting multiple signal series of an image area of the image series in each case,

filtering out candidate signal series from the extracted signal series, wherein a signal series is assessed as a candidate signal series when:

(i) a ratio of at least one signal of the signal series selected from the group consisting of a colored signal and an uncolored signal to at least one other signal of the signal series selected from the group consisting of a colored signal and an uncolored signal is a characteristic ratio; or

(ii) the signal series has a characteristic signature that comprises at least one characteristic ratio; or

(iii) a ratio of at least one signal of the signal series selected from the group consisting of a colored signal and an uncolored signal to at least one other signal of the signal series selected from the group consisting of a colored signal and an uncolored signal is a characteristic ratio and the signal series has a characteristic signature that comprises the at least one characteristic ratio, and

assigning the candidate signal series to a type of analyte or to the background.

2 . The method according to claim 1 , wherein the filtering out of candidate signal series is carried out by using a candidate extraction model, the candidate extraction model being selected from a set of candidate extraction models based, for example, on a type of sample, a type of experiment, or a user ID.

3 . The method according to claim 2 , wherein the candidate extraction model has been trained to use at least one specified ratio, the at least one specified ratio being a ratio of one of the colored or uncolored signals of the particular signal series to at least one other of the colored or uncolored signals of the particular signal series, and wherein the candidate extraction model has been trained (i) to identify the colored and uncolored signals based on the at least one specified ratio, or (ii) to identify the candidate signal series based on a characteristic signature that has the at least one specified ratio.

4 . The method according to claim 2 , wherein the candidate extraction model is a semantic segmentation model that assigns to each image area a value that indicates whether the image area captures an analyte, the value indicating, for example, a probability or a point value that the image area is an analyte image area, or the value being a bit that indicates whether or not the image area is an analyte image area.

5 . The method according to preceding claim 4 , wherein the candidate extraction model is a patch classifier, and assigns the value to each image area by use of a sliding window method.

6 . The method according to claim 2 , wherein the candidate extraction model is a fully convolutional network, and as a classification model has been trained with fully connected layers using signal series of individual image areas, and after the training, the classification model is converted into the fully convolutional network by replacing the fully connected layers with convolutional layers, the signal series of all image areas of the image series being simultaneously processed in the fully convolutional network.

7 . The method according to claim 2 , wherein the candidate extraction model is an image-to-image model and carries out image-to-image mapping that assigns to each image area a distance value that indicates the distance of the image area from a closest image area that includes a candidate signal series, or that assigns to each pixel a probability of being an image area that includes a candidate signal series.

8 . The method according to claim 2 , wherein the candidate extraction model is implemented as a detection model, and outputs a list of the candidate areas that include a candidate signal series.

9 . The method according to claim 1 , wherein the method also comprises, prior to the checking for whether the signal series is a candidate signal series, a step of transforming the signal series by means of principal axis transformation or singular value decomposition, and the transformed signal series is used in checking whether the signal series is a candidate signal series, for example only one subset of the components of the transformed signal series being used in the cluster analysis, and for example the first component, or the first component and the last component, or the first and the last two components preferably being omitted.

10 . The method according to claim 1 , wherein the image areas each include only one pixel, an area of contiguous pixels, or a contiguous volume in an image stack, and are input into the candidate extraction model as a tensor, for example.

11 . The method according to claim 1 , also comprising determining an image region, the determining of an image region including:

combining neighboring image areas into an image region when the neighboring image areas have candidate signal series, the combining of neighboring image areas including non-maximum suppression, for example.

12 . The method according to preceding claim 11 , wherein the determining of an image region also includes: checking the image regions, the checking of an image region including at least one of the following:

separating the image region into two or more image regions when the image region exceeds a maximum size, including determining an image region signal series for each of the image regions that arises from the separation,

separating the image regions into two or more image regions when the image regions in each case are connected to one another by only a few bridge pixels, or, based on a shape of the image region, it is apparent that two image regions intersect here, including determining an image region signal series for each of the image regions that arises from the separation;

separating the image region based on analyte context information, analyte context information including, for example: information concerning a size of an image region as a function of the type of analyte, information concerning a location of an image region in a sample, information concerning colocations of certain types of analytes in certain regions or in a location in a sample, or expected analyte densities as a function of a location in a sample or a location of an image area;

discarding image regions when an image region falls below a minimum size or has a shape that cannot be assigned to an analyte with certainty.

13 . The method according to claim 11 , wherein the determining of an image region also includes determining an image region signal series based on corresponding candidate signal series of the image region, and

the assigning of the candidate signal series to a type of analyte or to the background takes place based on the image region signal series.

14 . The method according to claim 11 , wherein the determining of an image region is carried out after checking whether the signal series is a candidate signal series, and before assigning the candidate signal series to a type of analyte or to the background and/or after assigning the candidate signal series to a type of analyte or to the background.

15 . The method according to claim 11 , further comprising:

using the identified type of analyte as analyte context information when determining the image region, the following in particular being included, based on the analyte context information:

information concerning a size of an image region as a function of the type of analyte,

information concerning a location of a type of analyte in a sample, or

information concerning colocation of certain types of analytes.

16 . The method according to claim 1 , wherein the identifying of a type of analyte based on the signal series includes:

determining a result bit sequence from the signal series, a true value being assigned to the colored signals and a false value being assigned to the uncolored signals in order to obtain the result bit sequence,

identifying the type of analyte based on the result bit sequence, the result bit sequence being compared to target bit sequences of a codebook which includes at least one target bit sequence for each of the types of analytes to be identified.

17 . The method according to claim 16 , wherein the determining of a result bit sequence is carried out by use of a binarization model.

18 . The method according to preceding claim 17 , wherein the binarization model is a classification model, the classification model having been trained to output an output bit sequence as the result output, either a “hard” assignment to the result output taking place by the processing model assigning a true value in the result bit sequence to the colored signals of the input signal series and assigning a false value in the result bit sequence to the uncolored signals, or a “soft” assignment taking place by the processing model outputting a probability distribution in which a probability is assigned to each image signal of the signal series, which indicates the probability that the image signal of the signal series is a colored signal.

19 . The method according to claim 1 , wherein the identifying of a type of analyte based on the signal series includes:

inputting the candidate signal series into an assignment model,

outputting a type of analyte of the candidate signal series, the outputting of a type of analyte either taking place “hard,” in which exactly one type of analyte is assigned to each candidate signal series, or taking place “soft,” in which a probability distribution with assignments to the various types of analytes is output for each candidate signal series.

20 . The method according to claim 1 , further comprising adapting a sensitivity, by means of the sensitivity it being possible to adapt the number of identified analytes to an expected number of analytes to be identified.

21 . The method according to claim 20 , wherein the adapting of the sensitivity includes in particular:

adapting a model sensitivity of the candidate extraction model, via which the number of candidate signal series identified in the extracted signal series may be set, or

adapting an identification sensitivity, via which during the identification of the type of analyte, a threshold value is set, above which a candidate signal series as background is discarded.

22 . The method according to claim 1 , wherein assigning the candidate signal series to a type of analyte or to the background takes place according to a method for identifying analytes in an image series by means of a cluster analysis, the image series being generated by marking the analytes with markers in multiple coloring rounds and detecting the markers using a camera, the camera recording an image of the image series in each coloring round, the markers being selected in such a way that image signals of an analyte in an image area over the image series include colored signals and uncolored signals, comprising:

extracting multiple signal series of an image area of the image series in each case,

clustering the extracted signal series by means of a cluster analysis algorithm, the number of predefined clusters being at least equal to the number of types of analytes to be identified, and one of the clusters being assigned to each of the signal series,

determining a cluster center for each of the clusters,

determining at least one target cluster center for each of the types of analytes to be identified, based on target series of a codebook, the codebook including a target series for each type of analyte to be identified,

determining, for each of the cluster centers, distances of the cluster center from the target cluster centers, and

assigning, based on the distances, the clusters to one of the types of analytes or to a background, and

assigning the signal series to the type of analyte or to the background, based on the assignment of the determined cluster, wherein only the identified candidate signal series are extracted.

23 . The method according to claim 1 , further comprising:

generating an expanded annotated data set, based on the extracted signal series and the assignment of the signal series to the type of analyte or to the background, and

carrying out the method for training a machine learning system according to one claim 7 , using at least the expanded annotated data set as the annotated data set.

24 . The method according to claim 1 , wherein image signals of the signal series are situated in an experiment feature space and target image signals of the target bit sequences are situated in a reference feature space, and the method also comprises, prior to comparing the signal series to the target bit sequences of the codebook:

converting the image signals of the signal series from the experiment feature space into the reference feature space.

25 . The method according to claim 1 , wherein the method also comprises, prior to checking whether the signal series is a candidate signal series, a step of carrying out a background correction of the image signals of the image series, the carrying out of the background correction including one or more of the following:

a rolling ball method,

filtering, for example a top hat method, homomorphous filtering, low pass filtering, wherein the result of the low pass filtering is subtracted from the signal, or temporal filtering,

background correction by use of an image-to-image model,

background correction by use of mixed models,

background correction by use of a mean shift method,

background correction by use of principal component analysis,

background correction by use of non-negative matrix factorization, or

background correction by the excitation of autofluorescence by use of a nonspecific laser for all image areas of the image series.

26 . The method according to claim 1 , further comprising determining registration information of images of an image series.

27 . The method according to preceding claim 26 , wherein at least two sets of various registration information are determined according to at least two different methods, which are carried out using each of the various sets of registration information, and after the analytes are identified, a best result for the identification of analytes is selected from the at least two results, an assessment of the results taking place, for example, based on an expected distribution of the analytes.

28 . The method according to preceding claim 27 , wherein the at least two sets of various registration information are determined by employing a method for determining registration information of images of an image series by use of a registration model, the registration information including at least translation information and rotation information, the registration model being trained to recognize registration structures in images, and the registration structures being structures that occur in images, and to which a location may be unambiguously assigned, comprising:

inputting the image series into the registration model,

determining the registration information.