Efficient caching for large scale distributed computations
Caching is provided to speed up the recomputation of an application, function, or other computation that relies on a very large input dataset, when the input dataset is changed. Previous computation results are stored in storage, for example, in a system-wide, global, persistent cache server. The storage enables the reuse of previous results on the parts of the dataset that are old and unchanged, and to only run the computation on the parts of the dataset that are new or changed. The application then combines the results of the two parts to form the final result.
1 . A computation method on a dataset, comprising:
performing a computation on a dataset to generate a first result;
receiving a change to a portion of the dataset;
performing the computation on the changed portion of the dataset to generate a second result; and
combining the second result and a portion of the first result corresponding to an unchanged portion of the dataset to generate a combined result.
2 . The method of claim 1 , further comprising:
storing the first result prior to receiving the change to the portion of the dataset; and
retrieving the portion of the first result corresponding to the unchanged portion of the dataset prior to combining.
3 . The method of claim 1 , wherein receiving the change to the portion of the dataset comprises appending new data to the dataset.
4 . The method of claim 3 , further comprising removing a second portion of the data from the dataset.
5 . The method of claim 4 , wherein the quantity of data in the second portion is based on the quantity of new data being appended to the dataset.
6 . The method of claim 1 , further comprising providing the combined result as a final computation result.
7 . The method of claim 1 , further comprising receiving a second change to a second portion of the dataset; performing the computation on the second changed portion of the dataset to generate a third result; and combining the third result and a portion of the second result corresponding to a recently unchanged portion of the dataset to generate another combined result.
8 . A computation system for a dataset, comprising:
a computing device that performs a computation on a dataset to generate a first result, receives a change to a portion of the dataset, and performs the computation on the changed portion of the dataset to generate a second result;
a storage device that stores the first result; and
a combiner that combines the second result and a portion of the first result corresponding to an unchanged portion of the dataset to generate a combined result.
9 . The system of claim 8 , wherein the storage device is adapted to store the first result prior to the computing device receiving the change to the portion of the dataset, and the computing device is adapted to retrieve the portion of the first result corresponding to the unchanged portion of the dataset prior to providing the portion to the combiner.
10 . The system of claim 8 , wherein the change to the portion of the dataset comprises new data appended to the dataset.
11 . The system of claim 10 , wherein the computing device is adapted to remove a second portion of the data from the dataset.
12 . The system of claim 11 , wherein the quantity of data in the second portion is based on the quantity of new data appended to the dataset.
13 . The system of claim 8 , wherein the combiner is adapted to provide the combined result as a final computation result.
14 . The system of claim 8 , wherein the computing device is adapted to receive a second change to a second portion of the dataset, and perform the computation on the second changed portion of the dataset to generate a third result, and the combiner is adapted to combine the third result and a portion of the second result corresponding to a recently unchanged portion of the dataset to generate another combined result.
15 . The system of claim 8 , wherein the storage device comprises a cache server
16 . A computation method on a dataset, comprising:
determining whether a fingerprint of data in an input stream matches a fingerprint of data stored in a storage device;
if so, then identifying the largest fingerprint match, and performing a computation on a portion of the data in the input stream to generate a first result, the portion based on the largest fingerprint match; and
if not, then performing the computation on the data in the input stream to generate a second result.
17 . The method of claim 16 , wherein if there is a match, then further comprising combining the first result with a stored result corresponding to a portion of the data in the input stream that had already been computed, to generate a third result.
18 . The method of claim 17 , further comprising storing the second result or the third result in the storage device.
19 . The method of claim 16 , wherein the portion corresponds to the data in the input stream that has not been previously subjected to the computation.
20 . The method of claim 16 , wherein the fingerprint comprises a hash code.