Systems and methods for visualizing a pattern in a dataset
A visualization system comprising a persistent memory, storing a dataset, and a non-persistent memory implements a pattern visualizing method. The dataset contains discrete attribute values for each first entity in a plurality of first entities for each second entity in a plurality of second entities. The dataset is compressed by blocked compression and represents discrete attribute values in both compressed sparse row and column formats. The discrete attribute values are clustered to assign each second entity to a cluster in a plurality of clusters. Differences in the discrete attribute values for the first entity across the second entities of a given cluster relative to the discrete attribute value for the same first entity across the other clusters are computed thereby deriving differential values. A heat map of these differential values for each first entity for each cluster is displayed to reveal the pattern in the dataset.
1. A visualization system, the visualization system comprising one or more processing cores, a persistent memory and a non-persistent memory, the persistent memory and the non-persistent memory collectively storing instructions for performing a method for visualizing a pattern in a discrete attribute value dataset, the method comprising:
storing the discrete attribute value dataset in persistent memory, wherein
the discrete attribute value dataset comprises a corresponding discrete attribute value for each first entity in a plurality of first entities for each respective second entity in a plurality of second entities, and
the discrete attribute value dataset redundantly represents the corresponding discrete attribute value for each first entity in the plurality of first entities for each respective second entity in the plurality of second entities in both a compressed sparse row format and a compressed sparse column format in which first entities for a respective second entity that have a null discrete attribute data value are discarded,
the discrete attribute value dataset is compressed in accordance with a blocked compression algorithm;
clustering the discrete attribute value dataset using the discrete attribute value for each first entity in the plurality of first entities, or principal components derived therefrom, for each respective second entity in the plurality of second entities thereby assigning each respective second entity in the plurality of second entities to a corresponding cluster in a plurality of clusters, wherein
each respective cluster in the plurality of clusters consists of a unique different subset of the second plurality of entities, and
the clustering loads less than the entirety of the discrete attribute value dataset into the non-persistent memory at any given time during the clustering, thereby allowing the clustering of the discrete attribute value dataset having a size that exceeds storage space in the non-persistent memory allocated to the discrete attribute value dataset;
computing, for each respective first entity in the plurality of first entities for each respective cluster in the plurality of clusters, a difference in the discrete attribute value for the respective first entity across the respective subset of second entities in the respective cluster relative to the discrete attribute value for the respective first entity across the plurality of clusters other than the respective cluster, thereby deriving a differential value for each respective first entity in the plurality of first entities for each respective cluster in the plurality of clusters; and
displaying in a first panel a heat map that comprises a representation of the differential value for each respective first entity in the plurality of first entities for each cluster in the plurality of clusters thereby visualizing the pattern in the discrete attribute value dataset.
2. The visualization system of claim 1 , wherein the differential value for each respective first entity in the plurality of first entities for each respective cluster in the plurality of clusters is a fold change in (i) a first measure of central tendency of the discrete attribute value for the first entity measured in each of the second entities in the plurality of second entities in the respective cluster and (ii) a second measure of central tendency of the discrete attribute value for the respective first entity measured in each of the second entities of all clusters other than the respective cluster.
3. The visualization system of claim 2 , wherein the fold change is a log 2 fold change.
4. The visualization system of claim 2 , wherein the fold change is a log 10 fold change.
5. The visualization system of claim 1 , wherein the method further comprises normalizing each discrete attribute value prior to computing the differential value for each respective first entity in the plurality of first entities for each respective cluster in the plurality of clusters.
6. The visualization system of claim 5 , wherein the normalizing comprises modeling the discrete attribute value of each first entity associated with each second entity in the plurality of entities with a negative binomial distribution having a consensus estimate of dispersion without loading the entire discrete attribute value dataset into non-persistent memory.
7. The visualization system of claim 1 , wherein the method further comprises:
applying a dimension reduction technique to a respective plurality of principal component values of each second entity in the plurality of second entities, wherein each said respective plurality of principal component values is derived from the discrete attribute values of each first entity in a corresponding second entity in the plurality of entities, thereby determining a two-dimensional data point for each second entity in the plurality of entities; and
plotting each respective second entity in the plurality of entities in a second panel based upon the two-dimensional data point for the respective second entity.
8. The visualization system of claim 7 , wherein
each cluster in the plurality of clusters is assigned a different graphic or color code, and
each respective second entity in the plurality of entities is coded in the second panel with the different graphic or color code for the cluster the respective second entity has been assigned.
9. The visualization system of claim 7 , wherein the dimension reduction technique is t-distributed stochastic neighbor embedding.
10. The visualization system of claim 7 , wherein the dimension reduction technique is Sammon mapping, curvilinear components analysis, stochastic neighbor embedding, Isomap, maximum variance unfolding, locally linear embedding, or Laplacian Eigenmaps.
11. The visualization system of claim 7 , wherein each said respective plurality of principal component values is derived from the discrete attribute values of each first entity in a corresponding second entity in the plurality of entities by principal component analysis that is performed on a computer system remote from the visualization system prior to storing the discrete attribute value dataset in persistent memory, and wherein the discrete attribute value dataset includes each said respective plurality of principal component values.
12. The visualization system of claim 1 , wherein the clustering of the discrete attribute value dataset is performed on a remote computer system remote from the visualization system prior to storing the discrete attribute value dataset in the persistent memory of the visualization system, wherein the clustering on the remote computer system loads less than the entirety of the discrete attribute value dataset into a non-persistent memory of the remote computer system at any given time during the clustering on the remote computer system.
13. The visualization system of claim 1 , wherein the clustering the discrete attribute value dataset comprises hierarchical clustering, agglomerative clustering using a nearest-neighbor algorithm, agglomerative clustering using a farthest-neighbor algorithm, agglomerative clustering using an average linkage algorithm, agglomerative clustering using a centroid algorithm, or agglomerative clustering using a sum-of-squares algorithm.
14. The visualization system of claim 1 , wherein the clustering the discrete attribute value dataset comprises application of a Louvain modularity algorithm, k-means clustering, a fuzzy k-means clustering algorithm, or Jarvis-Patrick clustering.
15. The visualization system of claim 1 , wherein the clustering the discrete attribute value dataset comprises k-means clustering of the discrete attribute value dataset into a predetermined number of clusters.
16. The visualization system of claim 15 , wherein the predetermined number of clusters is an integer between 2 and 50.
17. The visualization system of claim 1 , wherein the clustering the discrete attribute value dataset comprises k-means clustering of the discrete attribute value dataset into a number of clusters, wherein the number is provided by a user.
18. The visualization system of claim 1 , wherein
the clustering the discrete attribute value dataset comprises application of a Louvain modularity algorithm to a map, the map comprising a plurality of nodes and a plurality of edges,
each node in the plurality of nodes represents a second entity in the plurality of second entities, wherein the coordinates in N-dimensional space of a respective node in the plurality of nodes are a set of principal components of the corresponding second entity in the plurality of second entities, wherein the set of principal components is derived from the corresponding discrete attribute values of the plurality of first entities for the corresponding second entity, wherein N is the number of principal components in each set of principal components, and
an edge exists in the plurality of edges between a first node and a second node in the plurality of nodes when the first node is among the k nearest neighboring nodes of the second node in the first plurality of node, wherein the k nearest neighboring nodes to the second node is determined by computing a distance in the N-dimensional space between each node in the plurality of nodes, other than the second node, and the second node.
19. The visualization system of claim 18 , wherein the distance is a Euclidean distance.
20. The visualization system of claim 1 , wherein
each first entity in the plurality of first entities is a respective gene in a plurality of genes;
each discrete attribute value is a count of transcript reads within the second entity that map to a respective gene in the plurality of genes;
each second entity is a single cell; and
the discrete attribute value dataset represents a whole transcriptome shotgun sequencing experiment that quantifies gene expression from a single cell in counts of transcript reads mapped to the genes.
21. The visualization system of claim 1 , wherein each first entity in a particular second entity in the plurality of second entities is barcoded with a first barcode that is unique to the particular second entity.
22. The visualization system of claim 1 , wherein the discrete attribute value of each first entity in a particular second entity in the plurality of second entities is determined after the particular second entity has been separated from all the other second entities in the plurality of second entities into its own microfluidic partition.
23. The visualized system of claim 1 , wherein
each respective second entity in the plurality of second entities is barcoded with a unique barcode in a plurality of barcodes, the method further comprising indexing a clonotype dataset to the discrete attribute dataset, wherein
the clonotype dataset and the discrete attribute dataset are formed using a common plurality of second entities,
the clonotype dataset comprises a plurality of clonotypes, wherein each clonotype in the plurality of clonotypes is uniquely represented by a barcode in the plurality of barcodes; and,
the indexing the clonotype dataset to the discrete attribute dataset comprises identifying, for each clonotype in the plurality of clonotypes, a second entity in the discrete attribute dataset that has a matching barcode.
24. The visualized system of claim 23 , the method further comprising:
filtering the plurality of first entities in the discrete attribute dataset by one or more clonotypes in the clonotype dataset thereby producing a subset of the plurality of first entities; and
displaying the subset of the plurality of first entities.
25. The visualized system of claim 23 , the method further comprising:
filtering the plurality of first entities in the discrete attribute dataset by a union of (i) one or more clonotypes in the clonotype dataset and (ii) one or more clusters in the plurality of clusters thereby producing a subset of the plurality of first entities; and
displaying the subset of the plurality of first entities.
26. A method for visualizing a pattern in a discrete attribute value dataset, the method comprising:
at a computer system comprising a persistent memory and a non-persistent memory:
storing the discrete attribute value dataset in persistent memory, wherein
the discrete attribute value dataset comprises a corresponding discrete attribute value for each first entity in a plurality of first entities for each respective second entity in a plurality of second entities, and
the discrete attribute value dataset redundantly represents the corresponding discrete attribute value for each first entity in the plurality of first entities for each respective second entity in the plurality of second entities in both a compressed sparse row format and a compressed sparse column format in which first entities for a respective second entity that have a null discrete attribute data value are discarded, and
the discrete attribute value dataset is compressed in accordance with a blocked compression algorithm;
clustering the discrete attribute value dataset using the discrete attribute value for each first entity in the plurality of first entities, or principal components derived therefrom, for each respective second entity in the plurality of second entities thereby assigning each respective second entity in the plurality of second entities to a corresponding cluster in a plurality of clusters, wherein
each respective cluster in the plurality of clusters consists of a unique different subset of the second plurality of entities, and
the clustering loads less than the entirety of the discrete attribute value dataset into the non-persistent memory at any given time during the clustering, thereby allowing the clustering of the discrete attribute value dataset having a size that exceeds storage space in the non-persistent memory allocated to the discrete attribute value dataset;
computing, for each respective first entity in the plurality of first entities for each respective cluster in the plurality of clusters, a difference in the discrete attribute value for the respective first entity across the respective subset of second entities in the respective cluster relative to the discrete attribute value for the respective first entity across the plurality of clusters other than the respective cluster, thereby deriving a differential value for each respective first entity in the plurality of first entities for each respective cluster in the plurality of clusters; and
displaying in a first panel a heat map that comprises a representation of the differential value for each respective first entity in the plurality of first entities for each cluster in the plurality of clusters thereby visualizing the pattern in the discrete attribute value dataset.
27. A non-transitory computer readable storage medium, wherein the non-transitory computer readable storage medium stores instructions, which when executed by a computer system, cause the computer system to perform a method for visualizing a pattern in a discrete attribute value dataset, the method comprising:
storing the discrete attribute value dataset in persistent memory, wherein
the discrete attribute value dataset comprises a corresponding discrete attribute value for each first entity in a plurality of first entities for each respective second entity in a plurality of second entities, and
the discrete attribute value dataset redundantly represents the corresponding discrete attribute value for each first entity in the plurality of first entities for each respective second entity in the plurality of second entities in both a compressed sparse row format and a compressed sparse column format in which first entities for a respective second entity that have a null discrete attribute data value are discarded, and
the discrete attribute value dataset is compressed in accordance with a blocked compression algorithm;
clustering the discrete attribute value dataset using the discrete attribute value for each first entity in the plurality of first entities, or principal components derived therefrom, for each respective second entity in the plurality of second entities thereby assigning each respective second entity in the plurality of second entities to a corresponding cluster in a plurality of clusters, wherein
each respective cluster in the plurality of clusters consists of a unique different subset of the second plurality of entities, and
the clustering loads less than the entirety of the discrete attribute value dataset into the non-persistent memory at any given time during the clustering, thereby allowing the clustering of the discrete attribute value dataset having a size that exceeds storage space in the non-persistent memory allocated to the discrete attribute value dataset;
computing, for each respective first entity in the plurality of first entities for each respective cluster in the plurality of clusters, a difference in the discrete attribute value for the respective first entity across the respective subset of second entities in the respective cluster relative to the discrete attribute value for the respective first entity across the plurality of clusters other than the respective cluster, thereby deriving a differential value for each respective first entity in the plurality of first entities for each respective cluster in the plurality of clusters; and
displaying in a first panel a heat map that comprises a representation of the differential value for each respective first entity in the plurality of first entities for each cluster in the plurality of clusters thereby visualizing the pattern in the discrete attribute value dataset.
28. The visualization system of claim 1 , wherein the clustering the discrete attribute value dataset comprises clustering using the principal components derived from the discrete attribute values for the plurality of first entities, and wherein a number of the principal components is less than a number of the discrete attribute values for the plurality of first entities.