Identifying and clustering call stacks across multiple memory dumps with out-of-memory errors
A method of clustering call stacks from a memory dumps resulting from out-of-memory errors includes accessing a memory dump resulting from an out-of-memory error; identifying call stacks in the memory dump that are associated with the out-of-memory error; accessing call stacks from one or more other memory dumps that were determined to be associated with other out-of-memory errors; generating clusters of call stacks based on a similarity score; and providing a cluster for an analysis of the out-of-memory error.
1. A method of clustering call stacks from a plurality of memory dumps resulting from out-of-memory errors, the method comprising:
accessing a memory dump resulting from an out-of-memory error;
identifying one or more first call stacks in the memory dump that are determined to be associated with the out-of-memory error;
accessing one or more second call stacks from one or more other memory dumps that were determined to be associated with one or more other out-of-memory errors;
generating a plurality of clusters for the one or more first call stacks with the one or more second call stacks; and
providing a cluster from the plurality of clusters for an analysis of the out-of-memory error.
2. The method of claim 1 , further comprising processing the memory dump with a memory analysis tool that indexes the memory dump and generates a statistical analysis of object types found in the memory dump.
3. The method of claim 1 , wherein the memory dump comprises objects allocated on a heap.
4. The method of claim 1 , wherein identifying the one or more first call stacks in the memory dump comprises:
identifying object classes that consume the most memory in the memory dump.
5. The method of claim 4 , wherein identifying the one or more first call stacks in the memory dump further comprises:
identifying the one or more first call stacks as referencing the object classes that consume the most memory in the memory dump.
6. The method of claim 1 , wherein identifying the one or more first call stacks in the memory dump comprises:
calculating an amount of memory used by a process;
determining a maximum amount of memory allowed to be used by the process; and
calculating a ratio of the memory used by the process in comparison to the maximum amount of memory allowed to be used by the process.
7. The method of claim 6 , wherein identifying the one or more first call stacks in the memory dump further comprises:
identifying the one or more first call stacks as belonging to the process if the ratio exceeds a predetermined threshold.
8. A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
accessing a memory dump resulting from an out-of-memory error;
identifying one or more first call stacks in the memory dump that are determined to be associated with the out-of-memory error;
accessing one or more second call stacks from one or more other memory dumps that were determined to be associated with one or more other out-of-memory errors;
generating a plurality of clusters for the one or more first call stacks with the one or more second call stacks; and
providing a cluster from the plurality of clusters for an analysis of the out-of-memory error.
9. The non-transitory computer-readable medium of claim 8 , wherein identifying the one or more first call stacks in the memory dump comprises:
identifying a thread for which a memory request triggered the out-of-memory error; and
identifying the one or more first call stacks as belonging to the thread.
10. The non-transitory computer-readable medium of claim 8 , wherein identifying the one or more first call stacks in the memory dump comprises:
identifying one or more threads operating for more than a predetermined time limit; and
identifying the one or more first call stacks as belonging to the one or more threads.
11. The non-transitory computer-readable medium of claim 8 , wherein identifying the one or more first call stacks in the memory dump comprises:
identifying one or more worker threads that were processing events or requests when the out-of-memory error occurred;
determining whether an amount of memory used by the one or more worker threads exceeds a predetermined threshold; and
identifying the one or more first call stacks as belonging to the one or more worker threads.
12. The non-transitory computer-readable medium of claim 8 , further comprising storing the one or more first call stacks in a document repository with the one or more second call stacks.
13. The non-transitory computer-readable medium of claim 8 , wherein generating the plurality of clusters comprises:
dividing each of the one or more first call stacks and the one or more second call stacks into a plurality of layers.
14. The non-transitory computer-readable medium of claim 13 , wherein generating the plurality of clusters further comprises:
generating a similarity score between each of the plurality of layers individually.
15. The non-transitory computer-readable medium of claim 14 , wherein generating the plurality of clusters further comprises:
generating a full stack similarity score through a weighted combination of the similarity score for each of the plurality of layers.
16. A system comprising:
one or more processors; and
one or more memory devices comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
accessing a memory dump resulting from an out-of-memory error;
identifying one or more first call stacks in the memory dump that are determined to be associated with the out-of-memory error;
accessing one or more second call stacks from one or more other memory dumps that were determined to be associated with one or more other out-of-memory errors;
generating a plurality of clusters for the one or more first call stacks with the one or more second call stacks; and
providing a cluster from the plurality of clusters for an analysis of the out-of-memory error.
17. The system of claim 16 , wherein generating the plurality of clusters comprises:
dividing each of the one or more first call stacks and the one or more second call stacks into a plurality of layers;
generating a similarity score between each of the plurality of layers individually; and
generating a full stack similarity score through a weighted combination of the similarity score for each of the plurality of layers.
18. The system of claim 17 , wherein the similarity score comprises a cosine similarity wherein each unique word in a stack line represents a dimension, and wherein a number of occurrences of each unique word represents a magnitude of a vector.
19. The system of claim 16 , wherein generating the plurality of clusters comprises using a Density-based Spatial Clustering of Application with Noise (DBSCAN) algorithm to form the plurality of clusters from a similarity matrix, wherein the similarity matrix stores a similarity score for each combination of call stacks.
20. The system of claim 16 , further comprising highlighting similarities between call stacks in the cluster from the plurality of clusters.