Systems and methods for memory management in big data applications
The disclosed embodiments describe techniques for allocating memory to functions processing big data. In one embodiment, a method is disclosed comprising allocating a first memory space to a function, the memory space comprising an initial amount of memory for the function; declaring the first memory space as the current memory space; processing data using the function, the processing writing data to the current memory space; determining that the function requires additional memory space; allocating a new memory space based on the current memory space and a growth factor; copying all data in the current memory space to the new memory space; declaring the current memory space as the old memory space; declaring the new memory space as the current memory space; and not deallocating the old memory space.
1. A method comprising:
allocating a first memory space to a function, the memory space comprising an initial amount of memory for the function and the function being one of a plurality of functions processing a big data job;
setting the first memory space as a current memory space;
processing data using the function, the processing comprising writing data to the current memory space;
determining that the function requires additional memory space by monitoring that the function has exhausted its current memory space while processing;
determining that memory usage across the plurality of functions exhibits a power-law distribution wherein a small subset of the functions consume significantly more memory than a majority of the functions;
in response to determining that the memory usage exhibits the power-law distribution:
allocating a new memory space based on the current memory space and a geometric growth factor, the new memory space being larger than the current memory space by a factor determined by the geometric growth factor;
copying data in the current memory space to the new memory space;
setting the current memory space as an old memory space;
setting the new memory space as the current memory space; and
not deallocating the old memory space.
2. The method of claim 1 , wherein allocating a new memory space comprises computing a size of the new memory space by multiplying the current memory space by the growth factor.
3. The method of claim 2 , wherein allocating a new memory space comprises repeatedly allocating memory according to a geometric progression of new memory space sizes.
4. The method of claim 3 , wherein a total space consumed by a function, including non-deallocated memory spaces, comprises a geometric series sum.
5. The method of claim 2 , further comprising determining that the size of the new memory space exceeds a preconfigured maximum and raising one of an exception or a flag to indicate a need for special handling.
6. The method of claim 4 , further comprising executing one of a garbage collection or memory deallocation routine when terminating to reclaim all memory consumed by the function.
7. The method of claim 1 , further comprising initiating a plurality of additional functions and allocating memory spaces to the plurality of additional functions based on the initial amount of memory and the growth factor.
8. A non-transitory computer-readable storage medium for tangibly storing computer program instructions capable of being executed by a computer processor, the computer program instructions defining steps of:
allocating a first memory space to a function, the memory space comprising an initial amount of memory for the function and the function being one of a plurality of functions processing a big data job;
setting the first memory space as a current memory space;
processing data using the function, the processing comprising writing data to the current memory space;
determining that the function requires additional memory space by monitoring that the function has exhausted its current memory space while processing;
determining that memory usage across the plurality of functions exhibits a power-law distribution wherein a small subset of the functions consume significantly more memory than a majority of the functions;
in response to determining that the memory usage exhibits the power-law distribution:
allocating a new memory space based on the current memory space and a geometric growth factor, the new memory space being larger than the current memory space by a factor determined by the geometric growth factor;
copying data in the current memory space to the new memory space;
setting the current memory space as an old memory space;
setting the new memory space as the current memory space; and
not deallocating the old memory space.
9. The non-transitory computer-readable storage medium of claim 8 , wherein allocating a new memory space comprises computing a size of the new memory space by multiplying the current memory space by the growth factor.
10. The non-transitory computer-readable storage medium of claim 9 , wherein allocating a new memory space comprises repeatedly allocating memory according to a geometric progression of new memory space sizes.
11. The non-transitory computer-readable storage medium of claim 10 , wherein a total space consumed by a function, including non-deallocated memory spaces, comprises a geometric series sum.
12. The non-transitory computer-readable storage medium of claim 9 , the steps further comprising determining that the size of the new memory space exceeds a preconfigured maximum and raising one of an exception or a flag to indicate a need for special handling.
13. The non-transitory computer-readable storage medium of claim 11 , the steps further comprising executing one of a garbage collection or memory deallocation routine when terminating to reclaim all memory consumed by the function.
14. The non-transitory computer-readable storage medium of claim 8 , the steps further comprising initiating a plurality of additional functions and allocating memory spaces to the plurality of additional functions based on the initial amount of memory and the growth factor.
15. A device comprising:
a processor configured to:
allocate a first memory space to a function, the memory space comprising an initial amount of memory for the function and the function being one of a plurality of functions processing a big data job;
set the first memory space as a current memory space;
process data using the function, the processing comprising writing data to the current memory space;
determine that the function requires additional memory space by monitoring that the function has exhausted its current memory space while processing;
determining that memory usage across the plurality of functions exhibits a power-law distribution wherein a small subset of the functions consume significantly more memory than a majority of the functions;
in response to determining that the memory usage exhibits the power-law distribution:
allocate a new memory space based on the current memory space and a geometric growth factor, the new memory space being larger than the current memory space by a factor determined by the geometric growth factor;
copy data in the current memory space to the new memory space;
set the current memory space as an old memory space;
set the new memory space as the current memory space; and
not deallocate the old memory space.
16. The device of claim 15 , wherein allocating a new memory space comprises computing a size of the new memory space by multiplying the current memory space by the growth factor.
17. The device of claim 16 , wherein allocating a new memory space comprises repeatedly allocating memory according to a geometric progression of new memory space sizes.
18. The device of claim 17 , wherein a total space consumed by a function, including non-deallocated memory spaces, comprises a geometric series sum.
19. The device of claim 17 , further comprising executing one of a garbage collection or memory deallocation routine when terminating to reclaim all memory consumed by the function.
20. The device of claim 15 , further comprising initiating a plurality of additional functions and allocating memory spaces to the plurality of additional functions based on the initial amount of memory and the growth factor.