Memory allocation in a data analytics system

1. A computer-implemented method of managing memory in a computer comprising:

storing uncompressed memory blocks in a memory;

in response to an amount of the memory occupied by the uncompressed memory blocks exceeding a first threshold,

compressing at least some of the uncompressed memory blocks to form compressed memory blocks,

storing both the at least some of the uncompressed memory blocks and the compressed memory blocks in the memory, the at least some of the uncompressed memory blocks stored as memory blocks in a ready and compressed state, and the compressed memory blocks stored as memory blocks in a compressed state; and

in response to an amount of the memory occupied by the memory blocks in the ready and compressed state exceeding a second threshold, releasing at least some of the memory blocks in the ready and compressed state from the memory.

2. The method of claim 1 , further comprising:

in response to an amount of the memory occupied by the memory blocks in the compressed state exceeding a third threshold, transferring at least some of the memory blocks in the compressed state to a secondary memory.

3. The method of claim 2 , further comprising:

in response to an occupied amount of the memory falling below a fourth threshold, transferring at least some memory blocks in the secondary memory from the secondary memory to the memory.

4. The method of claim 3 , further comprising:

retaining the transferred memory blocks in the secondary memory; and

in response to an occupied amount of the memory exceeding the third threshold, releasing the transferred memory blocks from the memory.

5. The method of claim 1 , wherein storing uncompressed memory blocks in a memory comprises storing the uncompressed memory blocks in the memory in a ready state, further comprising:

receiving a request from a module executing on the computer, the request seeking to access a memory block;

determining whether the requested memory block is in the ready state; and

responsive to a determination that the requested memory block is not in the ready state, changing the requested memory block into the ready state; and

serving the requested memory block in the ready state in response to the access request.

6. The method of claim 1 , further comprising:

receiving a write request from a module executing on the computer, the write request seeking to access a memory block in the ready and compressed state;

changing the memory block into a ready state; and

releasing a compressed memory block in the compressed state that was formed by compressing the memory block.

7. The method of claim 1 , further comprising:

receiving a read request from a module executing on the computer to access a memory block in the ready and compressed state;

serving the read request using the memory block in the ready and compressed state; and

maintaining the memory block in the ready and compressed state.

8. A computer system for managing memory in a computer, the system comprising:

a non-transitory computer-readable storage medium storing executable computer program instructions, the computer program instructions comprising instructions that when executed cause a computer processor to perform steps, comprising:

storing uncompressed memory blocks in a memory;

in response to an amount of the memory occupied by the uncompressed memory blocks exceeding a first threshold,

compressing at least some of the uncompressed memory blocks to form compressed memory blocks,

storing both the at least some of the uncompressed memory blocks and the compressed memory blocks in the memory, the at least some of the uncompressed memory blocks stored as memory blocks in a ready and compressed state, and the compressed memory blocks stored as memory blocks in a compressed state; and

in response to an amount of the memory occupied by the memory blocks in the ready and compressed state exceeding a second threshold, releasing at least some of the memory blocks in the ready and compressed state from the memory.

9. The system of claim 8 , wherein the steps performed by the computer processor further comprise:

in response to an amount of the memory occupied by the memory blocks in the compressed state exceeding a third threshold, transferring at least some of the memory blocks in the compressed state to a secondary memory.

10. The system of claim 9 , wherein the steps performed by the computer processor further comprise:

in response to an occupied amount of the memory falling below a fourth threshold, transferring at least some memory blocks in the secondary memory from the secondary memory to the memory.

11. The system of claim 10 , wherein the steps performed by the computer processor further comprise:

retaining the transferred memory blocks in the secondary memory; and

in response to an occupied amount of the memory exceeding the third threshold, releasing the transferred memory blocks from the memory.

12. The system of claim 8 , wherein storing uncompressed memory blocks in a memory comprises storing the uncompressed memory blocks in the memory in a ready state, wherein the steps performed by the computer processor further comprise:

receiving a request from a module executing on the computer, the request seeking to access a memory block;

determining whether the requested memory block is in the ready state; and

responsive to a determination that the requested memory block is not in the ready state, changing the requested memory block into the ready state; and

serving the requested memory block in the ready state in response to the access request.

13. The system of claim 8 , wherein the steps performed by the computer processor further comprise:

receiving a write request from a module executing on the computer, the write request seeking to access a memory block in the ready and compressed state;

changing the memory block into the ready state; and

releasing a compressed memory block in the compressed state that was formed by compressing the memory block.

14. The system of claim 8 , wherein the steps performed by the computer processor further comprise:

receiving a read request from a module executing on the computer to access a memory block in the ready and compressed state;

serving the read request using the memory block in the ready and compressed state; and

maintaining the memory block in the ready and compressed state.

15. A non-transitory computer readable storage medium storing executable computer program instructions for managing memory in a computer, the computer program instructions comprising instructions that when executed cause a computer processor to:

store uncompressed memory blocks in a memory;

in response to an amount of the memory occupied by the uncompressed memory blocks exceeding a first threshold,

compress at least some of the uncompressed memory blocks to form compressed memory blocks,

store both the at least some of the uncompressed memory blocks and the compressed memory blocks in the memory, the at least some of the uncompressed memory blocks stored as memory blocks in a ready and compressed state, and the compressed memory blocks stored as memory blocks in a compressed state; and

in response to an amount of the memory occupied by the memory blocks in the ready and compressed state exceeding a second threshold, release at least some of the memory blocks in the ready and compressed state from the memory.

16. The computer readable medium of claim 15 , wherein the computer program instructions for managing memory in the computer further comprise instructions that when executed cause the computer processor to:

in response to an amount of the memory occupied by the memory blocks in the compressed state exceeding a third threshold, transfer at least some of the memory blocks in the compressed state to a secondary memory.

17. The computer readable medium of claim 16 , wherein the computer program instructions for managing memory in the computer further comprise instructions that when executed cause the computer processor to:

in response to an occupied amount of the memory falling below a fourth threshold, transfer at least some memory blocks in the secondary memory from the secondary memory to the memory.

18. The computer readable medium of claim 17 , wherein the computer program instructions for managing memory in the computer further comprise instructions that when executed cause the computer processor to:

retain the transferred memory blocks in the secondary memory; and

in response to an occupied amount of the memory exceeding the third threshold, release the transferred memory blocks from the memory.

19. The computer readable medium of claim 15 , wherein the computer program instructions for storing uncompressed memory blocks in a memory comprise instructions that when executed cause the computer processor to store the uncompressed memory blocks in the memory in a ready state, wherein the computer program instructions for managing memory in the computer further comprise instructions that when executed cause the computer processor to:

receive a request from a module executing on the computer, the request seeking to access a memory block;

determine whether the requested memory block is in the ready state; and

responsive to a determination that the requested memory block is not in the ready state, change the requested memory block into the ready state; and

serve the requested memory block in the ready state in response to the access request.

20. The computer readable medium of claim 15 , wherein the computer program instructions for managing memory in the computer further comprise instructions that when executed cause the computer processor to:

receive a write request from a module executing on the computer, the write request seeking to access a memory block in the ready and compressed state;

change the memory block into the ready state; and

release a compressed memory block in the compressed state that was formed by compressing the memory block.