System and method for improved performance in a multidimensional database environment
In accordance with an embodiment, described herein is a system and method for improving performance within a multidimensional database computing environment. A multidimensional database, utilizing a block storage option, performs numerous input/output (I/O) operations when executing calculations. To separate I/O operations from calculations, a background task queue is created to identify data blocks requiring I/O. The background task queue is utilized by background writer threads to execute the I/O operations in parallel with calculations.
1. A system for performing input/output operations on data blocks with a multidimensional database, comprising:
a multidimensional database for storage of data; and
a computing device having a processor that executes instructions to:
create a background thread pool for the multidimensional database, the background thread pool being associated with a background queue that indicates data blocks in a data cache stored in memory; and
write data blocks indicated by the background queue to one or more memory-mapped page files mapped to respective regions of a data storage via writer threads in the background thread pool,
wherein writing of the data blocks by the writer threads is performed in parallel with other operations performed on data blocks in the data cache.
2. The system of claim 1 , the processor is further configured to execute calculations on data blocks in the data cache via a plurality of calculation threads.
3. The system of claim 2 , wherein, when a calculation thread performs a calculation on a data block, the calculation thread adds the data block to the background queue.
4. The system of claim 1 , wherein a writer thread selects a data block from the background queue, compresses the data block, and writes the compressed data block to a corresponding memory-mapped page file.
5. The system of claim 4 , wherein the memory-mapped page file is stored in memory,
wherein the processor is further configured, by an operating system, to persist the memory-mapped page file in memory to a disk.
6. The system of claim 1 , wherein the processor is further configured to swap data blocks in the data cache, wherein to effect a swap the processor is configured to:
scan the data cache for a first data block having a clean state;
determine whether the first data block is swappable; and
replace the first data block in the data cache with a new data block from the one or more memory-mapped page files when the first block is swappable.
7. The system of claim 1 , wherein the processor is further configured to;
maintain a lock-free queue to track data blocks written to the one or more memory-mapped page files within a predetermined time period;
identify a data block from the queue, which maintains a clean state and is swappable, as a replacement victim; and
swap the data block identified in the data cache with a new data block from the one or more memory-mapped page files.
8. A method for performing input/output operations on data blocks of a multidimensional database, comprising:
creating a background thread pool for the multidimensional database, the background thread pool being associated with a background queue that indicates data blocks in a data cache stored in memory; and
writing data blocks indicated by the background queue to one or more memory-mapped page files in a file system buffer via writer threads in the background thread pool, the one or more memory-mapped page files are mapped to respective regions of a data storage,
wherein writing of the data blocks by the writer threads is performed in parallel with other operations performed on data blocks in the data cache.
9. The method of claim 8 , further comprising executing calculations on data blocks in the data cache via a plurality of calculation threads.
10. The method of claim 9 , further comprising adding a data block to the background queue when a calculation thread completes a calculation on the data block.
11. The method of claim 8 , further comprising:
selecting, by a writer thread, a data block from the background queue;
compressing the data block by the writer thread; and
writing, by the writer thread, the compressed data block to a corresponding memory-mapped page file.
12. The method of claim 8 , wherein the memory-mapped page file is stored in memory,
wherein the method further comprises persisting the memory-mapped page file in memory to a disk by an operating system.
13. The method of claim 8 , further comprising swapping data blocks in the data cache with data blocks from data storage, wherein swapping comprises:
scanning the data cache for a first data block having a clean state;
determining whether the first data block is swappable; and
replacing the first data block in the data cache with a new data block from the one or more memory-mapped page files when the first lock is swappable.
14. The method of claim 13 , further comprising;
retrieving, by an operating system, a new page file from the data storage when the new data block is not stored in the one or more memory-mapped page files in the file system buffer.
15. A non-transitory computer-readable storage medium having instructions stored thereon, which when read and executed by a computer, cause the computer to:
create a background thread pool for a multidimensional database, the background thread pool being associated with a background queue that indicates data blocks in a data cache stored in memory; and
write data blocks indicated by the background queue to one or more memory-mapped page files mapped to respective regions of a data storage via writer threads in the background thread pool,
wherein writing of data blocks by the writer threads the is performed in parallel with other operations performed on data blocks in the data cache.
16. The non-transitory computer-readable medium of claim 15 , wherein the instructions further configure the computer to execute calculations on data blocks in the data cache via a plurality of calculation threads.
17. The non-transitory computer-readable medium of claim 16 , wherein the instructions further cause the computer to add a data block to the background queue when a calculation thread performs a calculation on the data block.
18. The non-transitory computer-readable medium of claim 15 , wherein the instructions further cause the computer to use a writer thread to select a data block from the background queue, compress the data block, and write the compressed data block to a corresponding memory-mapped page file.
19. The non-transitory computer-readable medium of claim 18 , wherein the memory-mapped page file is stored in memory,
wherein the memory-mapped page file in memory is automatically persisted to a disk by an operating system.
20. The non-transitory computer-readable medium of claim 18 , wherein the instructions further cause the computer to swap data blocks in the data cache, wherein to effect a swap the instructions cause the computer to:
scan the data cache for a first data block having a clean state;
determine whether the first data block is swappable; and
replace the first data block in the data cache with a new data block from the one or more memory-mapped page files when the first block is swappable.