System and method for improved performance in a multidimensional database environment
In accordance with an embodiment, described herein is a system and method for improving performance within a multidimensional database computing environment. A multidimensional database, utilizing a block storage option, performs numerous input/output (I/O) operations when executing calculations. To separate I/O operations from calculations, a background task queue is created to identify data blocks requiring I/O. The background task queue is utilized by background writer threads to execute the I/O operations in parallel with calculations.
1. A system for performing input/output operations on data blocks with a multidimensional database, comprising:
a multidimensional database for storage of data; and
a computing device having a processor that executes instructions to:
create a background thread pool for the multidimensional database;
execute, by a first calculation thread in a calculation thread pool, a first calculation on a first data block in a data cache;
add the first data block to a background queue;
write, by a first background thread, the first data block from the background queue to a buffer, wherein the buffer is in communication with a data store, the buffer comprising at least a partial copy of data in the data store;
wherein after the first data block is written to the buffer, the first data block is written to the data store via a background operation.
2. The system of claim 1 , wherein the processor is further configured to execute, by a second calculation thread in the calculation thread pool, a second calculation on a second data block in the data cache, wherein the second calculation is executed in parallel with the writing of the first data block from the background queue to the buffer.
3. The system of claim 1 , wherein adding the first data block to the background queue includes adding a pointer to the background queue, and wherein the pointer points to the first data block in the data cache.
4. The system of claim 1 , wherein when writing the first data block from the background queue the first background thread compresses the first data block.
5. The system of claim 4 , wherein the buffer comprises a memory mapped page file of the data store.
6. The system of claim 1 , wherein the processor is further configured to swap data blocks in the data cache, wherein to effect a swap the processor is configured to:
scan the data cache for a clean data block having a clean state;
determine whether the clean data block is swappable; and
replace the clean data block in the data cache with a new data block from data storage when the clean data block is swappable.
7. The system of claim 1 , wherein the processor is further configured to;
maintain a lock-free queue to track data blocks written to the buffer within a predetermined time period;
identify a clean data block from the queue, which maintains a clean state and is swappable, as a replacement victim; and
swap the clean data block identified in the data cache with a new data block from the buffer.
8. A method for performing input/output operations on data blocks of a multidimensional database, comprising:
creating a background thread pool for the multidimensional database;
executing, by a first calculation thread in a calculation thread pool, a first calculation on a first data block in a data cache;
adding the first data block to a background queue; and
writing, by a first background thread, the first data block from the background queue to a buffer, wherein the buffer is in communication with a data store, the buffer comprising at least a partial copy of data in the data store;
wherein after the first data block is written to the buffer, the first data block is written to the data store via a background operation.
9. The method of claim 8 , wherein the processor is further configured to execute, by a second calculation thread in the calculation thread pool, a second calculation on a second data block in the data cache, wherein the second calculation is executed in parallel with the writing of the first data block from the background queue to the buffer.
10. The method of claim 8 , wherein adding the first data block to the background queue includes adding a pointer to the background queue, and wherein the pointer points to the first data block in the data cache.
11. The method of claim 8 , wherein when writing the first data block from the background queue the first background thread compresses the first data block.
12. The method of claim 11 , wherein the buffer comprises a memory mapped page file of the data store.
13. The method of claim 8 , further comprising:
swapping data blocks in the data cache, wherein said swapping comprises:
scanning the data cache for a clean data block having a clean state,
determining whether the clean data block is swappable, and
replacing the clean data block in the data cache with a new data block from data storage when the clean data block is swappable.
14. The method of claim 8 , further comprising;
maintaining a lock-free queue to track data blocks written to the buffer within a predetermined time period;
identifying a clean data block from the queue, which maintains a clean state and is swappable, as a replacement victim; and
swapping the clean data block identified in the data cache with a new data block from the buffer.
15. A non-transitory computer-readable storage medium having instructions stored thereon, which when read and executed by a computer, cause the computer to perform steps comprising:
creating a background thread pool for the multidimensional database;
executing, by a first calculation thread in a calculation thread pool, a first calculation on a first data block in a data cache;
adding the first data block to a background queue; and
writing, by a first background thread, the first data block from the background queue to a buffer, wherein the buffer is in communication with a data store, the buffer comprising at least a partial copy of data in the data store;
wherein after the first data block is written to the buffer, the first data block is written to the data store via a background operation.
16. The non-transitory computer-readable medium of claim 15 , wherein the processor is further configured to execute, by a second calculation thread in the calculation thread pool, a second calculation on a second data block in the data cache, wherein the second calculation is executed in parallel with the writing of the first data block from the background queue to the buffer.
17. The non-transitory computer-readable medium of claim 15 , wherein adding the first data block to the background queue includes adding a pointer to the background queue, and wherein the pointer points to the first data block in the data cache.
18. The non-transitory computer-readable medium of claim 15 , wherein when writing the first data block from the background queue the first background thread compresses the first data block; and
wherein the buffer comprises a memory mapped page file of the data store.
19. The non-transitory computer-readable medium of claim 15 , the steps further comprising:
swapping data blocks in the data cache, wherein said swapping comprises:
scanning the data cache for a clean data block having a clean state,
determining whether the clean data block is swappable, and
replacing the clean data block in the data cache with a new data block from data storage when the clean data block is swappable.
20. The non-transitory computer-readable medium of claim 15 , the steps further comprising:
maintaining a lock-free queue to track data blocks written to the buffer within a predetermined time period;
identifying a clean data block from the queue, which maintains a clean state and is swappable, as a replacement victim; and
swapping the clean data block identified in the data cache with a new data block from the buffer.