Managing memory and storage space for a data operation

1. A computing system, including:

a memory device providing a working memory space;

a storage device providing an overflow storage space; and

at least one processor configured to process a plurality of data units using a data operation that includes aggregating values from the data units to generate result information that is stored in one or more files, the processing including:

performing the data operation for each data unit of a first subset of data units from the plurality of data units and storing information associated with a result of the data operation in a first set of one or more data structures stored in the working memory space;

after an overflow condition on the working memory space is satisfied, storing information in the overflow storage space and freeing at least some of the working memory space, and performing the data operation for each data unit of a second subset of data units from the plurality of data units and storing information associated with a result of the data operation in a second set of one or more data structures stored in the working memory space; and

combining multiple sets of one or more data structures, including the first and second sets, to generate the result information;

wherein storing the information associated with a result of the data operation in a set of one or more data structures includes, for at least one data unit, performing an operation that changes information in the set of one or more data structures without increasing the amount of the working memory space used for the set of one or more data structures.

2. The computing system of claim 1 , wherein the overflow condition on the working memory space is satisfied if the amount of the working memory space used for the first set of one or more data structures is greater than or equal to a predetermined threshold.

3. The computing system of claim 1 , wherein the processing further includes, after the overflow condition being satisfied and before performing the data operation for each data unit of the second subset of data units, storing the first set of one or more data structures in the overflow storage space, and removing the first set of one or more data structures from the working memory space.

4. The computing system of claim 1 , wherein the memory device includes a volatile memory device.

5. The computing system of claim 1 , wherein the storage device includes a non-volatile storage device.

6. The computing system of claim 1 , wherein combining multiple sets of one or more data structures includes merging at least one data structure from the first set with at least one data structure from the second set.

7. The computing system of claim 6 , wherein merging at least one data structure from the first set with at least one data structure from the second set includes matching a first key in the data structure from the first set of one or more data structures with a second key in the data structure from the second set of one or more data structures, and performing an aggregation operation on a value associated with the first key and a value associated with the second key.

8. The computing system of claim 1 , wherein the processing further includes generating the plurality of data units from a data source, with each data unit including an identifier for a field of the data source and a value appearing in that field within a record of the data source.

9. The computing system of claim 8 , wherein the data operation includes an aggregation of information from multiple data units using the values included in the data units as keys for selecting matching data units from which information is aggregated.

10. The computing system of claim 1 , wherein the processing further includes, after the overflow condition being satisfied and before performing the data operation for each data unit of the second subset of data units, performing the data operation for each data unit of a third subset of data units from the plurality of data units and storing information associated with a result of the data operation in the first set of one or more data structures stored in the working memory space.

11. The computing system of claim 10 , wherein the second subset of data units is a subset of the data units of the third subset of data units.

12. The computing system of claim 10 , wherein the processing further includes, after performing the data operation for a first data unit of the third subset of data units, determining whether to store information associated with a result of the data operation in: (1) the first set of one or more data structures stored in the working memory space, or (2) the overflow storage space.

13. The computing system of claim 12 , wherein the operation that changes information in the set of one or more data structures includes an in-place memory operation that overwrites a value stored in a location within the working memory space with a different value stored in the same location within the working memory space.

14. The computing system of claim 12 , wherein storing information associated with a result of the data operation in the overflow storage space includes storing at least some content of the first data unit in the overflow storage space.

15. The computing system of claim 12 , wherein performing the data operation for the first data unit includes comparing a key in the first data unit to one or more keys in the first set of one or more data structures, and the information associated with a result of the data operation is stored in the first set of one or more data structures stored in the working memory space if the comparison results in a match, and the information associated with a result of the data operation is stored in the overflow storage space if the comparison does not result in a match.

16. A method for processing a plurality of data units using a data operation that includes aggregating values from the data units to generate result information that is stored in one or more files, the method including:

performing the data operation for each data unit of a first subset of data units from the plurality of data units and storing information associated with a result of the data operation in a first set of one or more data structures stored in working memory space of a memory device;

after an overflow condition on the working memory space is satisfied, storing information in overflow storage space of a storage device and freeing at least some of the working memory space, and performing the data operation for each data unit of a second subset of data units from the plurality of data units and storing information associated with a result of the data operation in a second set of one or more data structures stored in the working memory space; and

combining multiple sets of one or more data structures, including the first and second sets, to generate the result information;

wherein storing the information associated with a result of the data operation in a set of one or more data structures includes, for at least one data unit, performing an operation that changes information in the set of one or more data structures without increasing the amount of the working memory space used for the set of one or more data structures.

17. The method of claim 16 , wherein the overflow condition on the working memory space is satisfied if the amount of the working memory space used for the first set of one or more data structures is greater than or equal to a predetermined threshold.

18. The method of claim 16 , wherein the processing further includes, after the overflow condition being satisfied and before performing the data operation for each data unit of the second subset of data units, storing the first set of one or more data structures in the overflow storage space, and removing the first set of one or more data structures from the working memory space.

19. The method of claim 16 , wherein the memory device includes a volatile memory device.

20. The method of claim 16 , wherein the storage device includes a non-volatile storage device.

21. The method of claim 16 , wherein combining multiple sets of one or more data structures includes merging at least one data structure from the first set with at least one data structure from the second set.

22. The method of claim 21 , wherein merging at least one data structure from the first set with at least one data structure from the second set includes matching a first key in the data structure from the first set of one or more data structures with a second key in the data structure from the second set of one or more data structures, and performing an aggregation operation on a value associated with the first key and a value associated with the second key.

23. The method of claim 16 , wherein the processing further includes generating the plurality of data units from a data source, with each data unit including an identifier for a field of the data source and a value appearing in that field within a record of the data source.

24. The method of claim 23 , wherein the data operation includes an aggregation of information from multiple data units using the values included in the data units as keys for selecting matching data units from which information is aggregated.

25. The method of claim 16 , wherein the processing further includes, after the overflow condition being satisfied and before performing the data operation for each data unit of the second subset of data units, performing the data operation for each data unit of a third subset of data units from the plurality of data units and storing information associated with a result of the data operation in the first set of one or more data structures stored in the working memory space.

26. The method of claim 25 , wherein the second subset of data units is a subset of the data units of the third subset of data units.

27. The method of claim 25 , wherein the processing further includes, after performing the data operation for a first data unit of the third subset of data units, determining whether to store information associated with a result of the data operation in: (1) the first set of one or more data structures stored in the working memory space, or (2) the overflow storage space.

28. The method of claim 27 , wherein the operation that changes information in the set of one or more data structures includes an in-place memory operation that overwrites a value stored in a location within the working memory space with a different value stored in the same location within the working memory space.

29. The method of claim 27 , wherein storing information associated with a result of the data operation in the overflow storage space includes storing at least some content of the first data unit in the overflow storage space.

30. The method of claim 27 , wherein performing the data operation for the first data unit includes comparing a key in the first data unit to one or more keys in the first set of one or more data structures, and the information associated with a result of the data operation is stored in the first set of one or more data structures stored in the working memory space if the comparison results in a match, and the information associated with a result of the data operation is stored in the overflow storage space if the comparison does not result in a match.

31. Software stored on a non-transitory computer-readable medium for processing a plurality of data units using a data operation that includes aggregating values from the data units to generate result information that is stored in one or more files, the software including instructions for causing a computing system to:

perform a data operation for each data unit of a first subset of data units from the plurality of data units and storing information associated with a result of the data operation in a first set of one or more data structures stored in the working memory space of a memory device;

after an overflow condition on the working memory space is satisfied, store information in overflow storage space of a storage device and freeing at least some of the working memory space, and perform the data operation for each data unit of a second subset of data units from the plurality of data units and storing information associated with a result of the data operation in a second set of one or more data structures stored in the working memory space; and

combine multiple sets of one or more data structures, including the first and second sets, to generate the result information;

wherein storing the information associated with a result of the data operation in a set of one or more data structures includes, for at least one data unit, performing an operation that changes information in the set of one or more data structures without increasing the amount of the working memory space used for the set of one or more data structures.

32. The software of claim 31 , wherein the overflow condition on the working memory space is satisfied if the amount of the working memory space used for the first set of one or more data structures is greater than or equal to a predetermined threshold.

33. The software of claim 31 , wherein the processing further includes, after the overflow condition being satisfied and before performing the data operation for each data unit of the second subset of data units, storing the first set of one or more data structures in the overflow storage space, and removing the first set of one or more data structures from the working memory space.

34. The software of claim 31 , wherein the memory device includes a volatile memory device.

35. The software of claim 31 , wherein the storage device includes a non-volatile storage device.

36. The software of claim 31 , wherein combining multiple sets of one or more data structures includes merging at least one data structure from the first set with at least one data structure from the second set.

37. The software of claim 36 , wherein merging at least one data structure from the first set with at least one data structure from the second set includes matching a first key in the data structure from the first set of one or more data structures with a second key in the data structure from the second set of one or more data structures, and performing an aggregation operation on a value associated with the first key and a value associated with the second key.

38. The software of claim 31 , wherein the processing further includes generating the plurality of data units from a data source, with each data unit including an identifier for a field of the data source and a value appearing in that field within a record of the data source.

39. The software of claim 38 , wherein the data operation includes an aggregation of information from multiple data units using the values included in the data units as keys for selecting matching data units from which information is aggregated.

40. The software of claim 31 , wherein the processing further includes, after the overflow condition being satisfied and before performing the data operation for each data unit of the second subset of data units, performing the data operation for each data unit of a third subset of data units from the plurality of data units and storing information associated with a result of the data operation in the first set of one or more data structures stored in the working memory space.

41. The software of claim 40 , wherein the second subset of data units is a subset of the data units of the third subset of data units.

42. The software of claim 40 , wherein the processing further includes, after performing the data operation for a first data unit of the third subset of data units, determining whether to store information associated with a result of the data operation in: (1) the first set of one or more data structures stored in the working memory space, or (2) the overflow storage space.

43. The software of claim 42 , wherein the operation that changes information in the set of one or more data structures includes an in-place memory operation that overwrites a value stored in a location within the working memory space with a different value stored in the same location within the working memory space.

44. The software of claim 42 , wherein storing information associated with a result of the data operation in the overflow storage space includes storing at least some content of the first data unit in the overflow storage space.

45. The software of claim 42 , wherein performing the data operation for the first data unit includes comparing a key in the first data unit to one or more keys in the first set of one or more data structures, and the information associated with a result of the data operation is stored in the first set of one or more data structures stored in the working memory space if the comparison results in a match, and the information associated with a result of the data operation is stored in the overflow storage space if the comparison does not result in a match.

46. A computing system, including:

means for providing a working memory space;

means for providing an overflow storage space; and

means for processing a plurality of data units using a data operation that includes aggregating values from the data units to generate result information that is stored in one or more files, the processing including:

performing the data operation for each data unit of a first subset of data units from the plurality of data units and storing information associated with a result of the data operation in a first set of one or more data structures stored in the working memory space;

after an overflow condition on the working memory space is satisfied, storing information in the overflow storage space and freeing at least some of the working memory space, and performing the data operation for each data unit of a second subset of data units from the plurality of data units and storing information associated with a result of the data operation in a second set of one or more data structures stored in the working memory space; and

combining multiple sets of one or more data structures, including the first and second sets, to generate the result information;

wherein storing the information associated with a result of the data operation in a set of one or more data structures includes, for at least one data unit, performing an operation that changes information in the set of one or more data structures without increasing the amount of the working memory space used for the set of one or more data structures.