Granted Patent B1

US 10,769,501 · App. 16/133,542 · Granted Sep 8, 2020

Inventors: Dale M. Ando (South San Francisco, CA); Marc Berndl (Mountain View, CA); Lusann Yang (Mountain View, CA); Michelle Dimon (Woodside, CA)

Assignee: Google LLC

G06K9/6267G06K9/6215G06K9/6228G06K9/6257G06K9/6288G06K9/726G06N3/08G01N33/20G01N33/48G06K9/00147G06K2209/27

Abstract

Claims (97)

1. A method comprising:

applying a respective perturbation to each of a plurality of subjects in a controlled environment;

producing a respective visual representation for each of the perturbed subjects using at least one imaging modality;

obtaining, by a computing device for each of the respective visual representations, a corresponding semantic embedding associated with the respective visual representation,

wherein the semantic embedding associated with the respective visual representation is generated using a machine-learned, deep metric network model,

wherein obtaining the corresponding semantic embedding associated with the respective visual representations comprises incorporating data captured during one or more respective supplementary measurements of a given measurement type,

wherein the machine-learned, deep metric network model was trained using training data that includes data of the given measurement type, and

wherein incorporating data captured during one or more respective supplementary measurements comprises:

generating a first preliminary semantic embedding for the respective visual representation using the machine-learned, deep metric network model;

generating a second preliminary semantic embedding for the data captured during one or more respective supplementary measurements using the machine-learned, deep metric network model; and

generating a composite semantic embedding:

(i) using an additional machine-learned model, wherein the composite semantic embedding comprises a hybrid of the first preliminary semantic embedding and the second preliminary semantic embedding; or

(ii) by applying a mathematical operation to corresponding dimensions of the first preliminary semantic embedding and the second preliminary semantic embedding; and

classifying, by the computing device based on the corresponding semantic embedding, each of the visual representations into one or more groups.

2. The method of claim 1 , wherein the at least one imaging modality comprises computed tomography, magnetic resonance imaging, positron emission tomography, ultrasound, x-ray computed tomography, x-ray diffraction, fluoroscopy, projectional radiography, single-photon emission computed tomography, scintigraphy, elastography, photoacoustic imaging, near-infrared spectroscopy, magnetic particle imaging, optoacoustic imaging, diffuse optical tomography, Raman spectroscopy, fluorescent microscopy, confocal microscopy, two-photon microscopy, hyperspectral analysis, transmission microscopy, electromagnetic scanning, differential interference contrast microscopy, multiphoton microscopy, dark-field microscopy, quantitative phase-contrast microscopy, near-field scanning optical microscopy, photo-activated localization microscopy, second harmonic imaging, holography, scanning electron microscopy, or tunneling electron microscopy.

3. The method of claim 1 , wherein the plurality of subjects comprises biological cells, fibroblasts, malaria cells, yeast cells, yeast cultures, bacteria, bacterial cultures, fungus, fungal cultures, cancer cells, blood cells, malarial parasites, mitochondria, nuclei, axons, dendrites, induced pluripotent stem cells, biological cells from a given region of an organism, biological cells from a given tissue of an organism, biological cells from a given organ of an organism, biological cells from a given system of an organism, biological cell ensembles, tissues, organs, organoids, biological systems, organisms, groups of organisms, ecosystems, chemical compounds, crystals, metallic glasses, mixtures of metallic salts, semiconductors, metals, dielectrics, graphene, microelectromechanical systems (MEMS), or nanoelectromechanical systems (NEMS).

4. The method of claim 1 , wherein applying the respective perturbation to each of the plurality of subjects comprises at least one of:

adding one or more small molecule compounds to at least one of the plurality of subjects;

applying a candidate treatment compound to at least one of the plurality of subjects;

applying a genetic modification to at least one of the plurality of subjects;

allowing a predetermined amount of time to elapse for at least one of the plurality of subjects;

illuminating at least one of the plurality of subjects with light of a predetermined wavelength;

heating or cooling at least one of the plurality of subjects to a predetermined temperature;

exposing at least one of the plurality of subjects to another subject of the plurality of subjects;

introducing at least one of the plurality of subjects to a different environment; or

applying a predetermined force to one or more regions of at least one of the plurality of subjects.

5. The method of claim 1 , wherein each of the respective visual representations comprises a three-dimensional visual representation.

6. The method of claim 5 , wherein the machine-learned, deep metric network model was trained using three-dimensional training data.

7. The method of claim 1 , wherein each of the respective visual representations comprises a videographic representation.

8. The method of claim 7 , wherein the machine-learned, deep metric network model was trained using videographic training data.

9. The method of claim 1 , wherein producing the respective visual representations comprises performing a patch selection to identify each of the respective perturbed subjects.

10. The method of claim 9 , wherein the patch selection is machine-learned and comprises an attention mechanism used to define which regions of a preliminary visual representation are of threshold interest level to be selected by the patch selection.

11. The method of claim 1 , further comprising, for each of the semantic embeddings:

detecting, by the computing device, a background effect present within the semantic embedding; and

removing, by the computing device, the background effect from the semantic embedding.

12. The method of claim 11 , wherein the data captured during one or more respective supplementary measurements comprises experimental metadata, transcriptomic data, genomic data, proteomic data, metabolomic data, lipidomic data, bulk semiconductor material properties, or patient diagnostic data.

13. The method of claim 11 ,

wherein the machine-learned, deep metric network model was trained using training data that includes data of the given measurement type,

wherein each visual representation comprises a plurality of channels, and

wherein incorporating data captured during one or more respective supplementary measurements comprises:

retrieving, by the computing device, each channel of the respective visual representation;

generating, by the computing device using the machine-learned, deep metric network model, a respective single-channel semantic embedding for each channel of the respective visual representation;

generating, by the computing device using the machine-learned, deep metric network model, an additional single-channel semantic embedding for the data captured during one or more respective supplementary measurements; and

concatenating, by the computing device, the single-channel semantic embeddings for each channel of the respective visual representation with the additional single-channel semantic embedding into a multi-channel semantic embedding.

14. The method of claim 1 ,

wherein the composite semantic embedding is generated (i) using the additional machine-learned model, and wherein the composite semantic embedding comprises the hybrid of the first preliminary semantic embedding and the second preliminary semantic embedding.

15. The method of claim 1 ,

wherein the composite semantic embedding is generated (ii) by applying the mathematical operation to corresponding dimensions of the first preliminary semantic embedding and the second preliminary semantic embedding.

16. The method of claim 1 , wherein classifying each of the visual representations into one or more groups comprises performing vector arithmetic on the corresponding semantic embedding to determine to which of the one or more groups the respective visual representation belongs.

17. The method of claim 1 ,

wherein classifying each of the visual representations into one or more groups comprises applying a classification model to each of the corresponding semantic embeddings, and

wherein the classification model is trained to identify responses to the respective perturbations and disregard statistical variations present across the plurality of subjects.

18. The method of claim 17 , wherein the classification model was trained using a human-in-the-loop procedure to identify which variations correspond to notable differences and which variations are attributable to randomness.

19. The method of claim 1 , wherein classifying the visual representations into one or more groups comprises:

identifying one or more values corresponding to one or more respective dimensions of the semantic embedding associated with the respective visual representation; and

determining, based on the one or more values, to which of an enumerated list of groups the respective visual representation belongs.

20. The method of claim 1 ,

wherein classifying the visual representations into one or more groups comprises extracting information about the respective visual representation based on a topology within the semantic embedding associated with the respective visual representation, and

wherein the topology was enforced during training of the machine-learned, deep metric network model.

21. The method of claim 20 , wherein the topology is linear, elliptical, circular, polygonal, spherical, ellipsoidal, cylindrical, conical, hyperboloidal, paraboloidal, toroidal, pyramidal, polyhedral, or defined in a dimensional space having greater than three dimensions.

22. The method of claim 1 ,

wherein the machine-learned, deep metric network model is trained using a plurality of photographic images as training data,

wherein the photographic images are query results ranked based on selections, and

wherein rankings of the photographic images are used in the training of the machine-learned, deep metric network model to determine image similarity between two images.

23. The method of claim 1 , further comprising training the machine-learned, deep metric network model, wherein training the machine-learned, deep metric network model comprises:

receiving, by the computing device, a series of three-image sets as training data,

wherein each three-image set comprises a query image, a positive image, and a negative image,

wherein the query image, the positive image, and the negative image are query results ranked in comparison with one another based on selections, and

wherein the selections indicate that a similarity between the query image and the positive image is greater than a similarity between the query image and the negative image; and

refining, by the computing device, the machine-learned, deep metric network model based on each three-image set to account for image components of the query image, the positive image, and the negative image.

24. The method of claim 1 , further comprising determining, by the computing device, a preferred perturbation from among the respective perturbations based on the one or more groups into which each of the visual representations are classified.

25. A non-transitory, computer-readable medium having instructions stored thereon, wherein the instructions, when executed by a processor, cause the processor to execute a method comprising:

for each respective subject of a plurality of subjects, receiving a corresponding visual representation of the respective subject that was produced using at least one imaging modality after application of a respective perturbation to the respective subject in a controlled environment;

obtaining, for each of the visual representations, a corresponding semantic embedding associated with the visual representation,

wherein the semantic embeddings are generated using a machine-learned, deep metric network model,

wherein obtaining the semantic embeddings associated with the respective visual representations comprises incorporating data captured during one or more respective supplementary measurements of a given measurement type,

wherein the machine-learned, deep metric network model was trained using training data that includes data of the given measurement type, and

wherein incorporating data captured during one or more respective supplementary measurements comprises:

generating a first preliminary semantic embedding for the respective visual representation using the machine-learned, deep metric network model;

generating a second preliminary semantic embedding for the data captured during one or more respective supplementary measurements using the machine-learned, deep metric network model; and

generating a composite semantic embedding:

(i) using an additional machine-learned model, wherein the composite semantic embedding comprises a hybrid of the first preliminary semantic embedding and the second preliminary semantic embedding; or

(ii) by applying a mathematical operation to corresponding dimensions of the first preliminary semantic embedding and the second preliminary semantic embedding; and

classifying, based on the corresponding semantic embedding, each of the visual representations into one or more groups.

26. A method comprising:

for each respective subject of a plurality of subjects, receiving, by a computing device, a corresponding visual representation of the respective subject that was produced using at least one imaging modality after application of a respective perturbation to the respective subject in a controlled environment;

obtaining, by the computing device, a semantic embedding associated with the visual representation of the perturbed subject,

wherein the semantic embedding associated with the visual representation of the perturbed subject is generated using a machine-learned, deep metric network model,

wherein obtaining the corresponding semantic embedding associated with the respective visual representations comprises incorporating data captured during one or more respective supplementary measurements of a given measurement type,

wherein the machine-learned, deep metric network model was trained using training data that includes data of the given measurement type, and

wherein incorporating data captured during one or more respective supplementary measurements comprises:

generating a first preliminary semantic embedding for the respective visual representation using the machine-learned, deep metric network model;

generating a second preliminary semantic embedding for the data captured during one or more respective supplementary measurements using the machine-learned, deep metric network model; and

generating a composite semantic embedding:

(i) using an additional machine-learned model, wherein the composite semantic embedding comprises a hybrid of the first preliminary semantic embedding and the second preliminary semantic embedding; or

(ii) by applying a mathematical operation to corresponding dimensions of the first preliminary semantic embedding and the second preliminary semantic embedding; and

determining, by the computing device based on the semantic embedding, what type of perturbation was applied to the perturbed subject.

Assignments (1)

Continuity (2)

Cited By (9)

US 12,272,161 US 12,373,950 US 12,374,429 US 12,481,833 US 12,602,939 US 12,640,230 US 12,651,432 US 12,657,939 US 12,738,081