Memory mapping to enhance data cube performance

1. A method performed by one or more computers, the method comprising:

accessing a data set comprising data to be processed into a data cube;

generating a memory-mapped data cube based on the data set, the memory-mapped data cube comprising a plurality of files including different segments of the data cube, wherein generating the memory-mapped data cube comprises:

allocating memory-mapped buffers in non-volatile data storage in response to first memory allocation requests, wherein the memory-mapped buffers each allocate extra capacity such that the buffer exceeds an allocation size indicated in the corresponding first memory allocation request;

after allocating the memory-mapped buffers and writing first components of the data cube in the memory-mapped buffers, responding to second memory allocation requests by providing addresses for locations in the extra capacity of the buffers;

based on the provided addresses, writing second components of the data cube to the memory-mapped buffers; and

storing contents of the respective buffers as files of the data cube; and

loading the memory-mapped data cube by storing the files of the data cube in disk-based storage, mapping the stored files of the data cube to virtual memory addresses, and caching portions of the data cube in random-access memory.

2. The method of claim 1 , wherein the data cube is an online analytical processing (OLAP) data cube or a multi-table data import (MTDI) data cube.

3. The method of claim 1 , comprising adjusting the portions of the data cube that are cached based on accesses to the data cube over a period of time.

4. The method of claim 1 , wherein caching portions of the data cube in random-access memory comprises caching sub-file portions of the files of the data cube.

5. The method of claim 4 , wherein caching portions of the data cube in random-access memory comprises performing page-level caching of the files of the data cube.

6. The method of claim 1 , wherein caching portions of the data cube in random-access memory comprises selectively caching individual files of the data cube based on an access history for the files in the data cube.

7. The method of claim 1 , wherein the memory-mapped buffers have initial sizes that are more than double the allocation sizes requested in the first memory allocations.

8. The method of claim 1 , comprising:

determining that, for a particular request of the second memory allocation requests, a requested allocation size exceeds an amount of available capacity in a memory-mapped buffer identified for handling memory allocation for the particular request;

expanding the identified memory-mapped buffer by a predetermined increment that exceeds the requested allocation size; and

after expanding the identified memory-mapped buffer, providing an address in the expanded buffer in response to the particular request.

9. A system comprising:

one or more computers; and

one or more computer-readable media storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

accessing a data set comprising data to be processed into a data cube;

generating a memory-mapped data cube based on the data set, the memory-mapped data cube comprising a plurality of files including different segments of the data cube, wherein generating the memory-mapped data cube comprises:

allocating memory-mapped buffers in non-volatile data storage in response to first memory allocation requests, wherein the memory-mapped buffers each allocate extra capacity such that the buffer exceeds an allocation size indicated in the corresponding first memory allocation request;

after allocating the memory-mapped buffers and writing first components of the data cube in the memory-mapped buffers, responding to second memory allocation requests by providing addresses for locations in the extra capacity of the buffers;

based on the provided addresses, writing second components of the data cube to the memory-mapped buffers; and

storing contents of the respective buffers as files of the data cube; and

loading the memory-mapped data cube by storing the files of the data cube in disk-based storage, mapping the stored files of the data cube to virtual memory addresses, and caching portions of the data cube in random-access memory.

10. The system of claim 9 , wherein the data cube is an online analytical processing (OLAP) data cube or a multi-table data import (MTDI) data cube.

11. The system of claim 9 , comprising adjusting the portions of the data cube that are cached based on accesses to the data cube over a period of time.

12. The system of claim 9 , wherein caching portions of the data cube in random-access memory comprises caching sub-file portions of the files of the data cube.

13. The system of claim 12 , wherein caching portions of the data cube in random-access memory comprises performing page-level caching of the files of the data cube.

14. The system of claim 9 , wherein caching portions of the data cube in random-access memory comprises selectively caching individual files of the data cube based on an access history for the files in the data cube.

15. The system of claim 9 , wherein the memory-mapped buffers have initial sizes that are more than double the allocation sizes requested in the first memory allocations.

16. The system of claim 9 , comprising:

determining that, for a particular request of the second memory allocation requests, a requested allocation size exceeds an amount of available capacity in a memory-mapped buffer identified for handling memory allocation for the particular request;

expanding the identified memory-mapped buffer by a predetermined increment that exceeds the requested allocation size; and

after expanding the identified memory-mapped buffer, providing an address in the expanded buffer in response to the particular request.

17. One or more non-transitory computer-readable media storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

accessing a data set comprising data to be processed into a data cube;

generating a memory-mapped data cube based on the data set, the memory-mapped data cube comprising a plurality of files including different segments of the data cube, wherein generating the memory-mapped data cube comprises:

allocating memory-mapped buffers in non-volatile data storage in response to first memory allocation requests, wherein the memory-mapped buffers each allocate extra capacity such that the buffer exceeds an allocation size indicated in the corresponding first memory allocation request;

after allocating the memory-mapped buffers and writing first components of the data cube in the memory-mapped buffers, responding to second memory allocation requests by providing addresses for locations in the extra capacity of the buffers;

based on the provided addresses, writing second components of the data cube to the memory-mapped buffers; and

storing contents of the respective buffers as files of the data cube; and

loading the memory-mapped data cube by storing the files of the data cube in disk-based storage, mapping the stored files of the data cube to virtual memory addresses, and caching portions of the data cube in random-access memory.

18. The one or more non-transitory computer-readable media of claim 17 , wherein the data cube is an online analytical processing (OLAP) data cube or a multi-table data import (MTDI) data cube.

19. The one or more non-transitory computer-readable media of claim 17 , comprising adjusting the portions of the data cube that are cached based on accesses to the data cube over a period of time.

20. The one or more non-transitory computer-readable media of claim 17 , wherein caching portions of the data cube in random-access memory comprises caching sub-file portions of the files of the data cube.