System and method for supporting elastic data metadata compression in a distributed data grid
A system and method can support metadata compression in a distributed data grid. The system can store a plurality of keys in compact data structure (such as a radix tree) in a heap, wherein each said key is associated with an entry in a journal off-heap. Furthermore, the system can look up the tree for a key, wherein the key is associated with a journal ticket that represents a virtual address of an entry in the journal. Then, the system can use the journal ticket to manage the entry in the journal that is associated with said key.
1. A method for supporting metadata compression in a distributed data grid, comprising:
storing a plurality of keys in a compact data structure in a heap, wherein said compact data structure is a radix tree and each said key is associated with an entry of a journal off-heap;
looking up the radix tree for a key, wherein the key is associated with a journal ticket that represents a virtual address of an entry in the journal; and
using the journal ticket to manage the entry in the journal that is associated with said key.
2. The method according to claim 1 , further comprising:
including a RAM journal in the journal off-heap, and
including a flash journal in the journal off-heap, wherein the flash journal operates to transparently handle data overflow from the RAM journal.
3. The method according to claim 1 , further comprising:
allowing the heap to be a virtual machine (VM) heap.
4. The method according to claim 1 ,
wherein each key in the radix tree corresponds to a value stored in an entry in a journal off-heap.
5. The method according to claim 4 , further comprising:
storing an value corresponding to a key directly in the radix tree if the size of the value is small.
6. The method according to claim 4 , further comprising:
storing one or more journal tickets in the radix tree, wherein each said journal ticket is associated with a key.
7. The method according to claim 4 , further comprising:
storing one or more journal tickets in a long storage array, wherein each said journal ticket is associated with a key.
8. The method according to claim 7 , further comprising:
associating the key stored in the radix tree with a slot number, wherein the slot number indicates a location in the long storage array where said journal ticket stores.
9. The method according to claim 8 , further comprising:
storing additional attributes that are associated with the key in the long storage array.
10. The method according to claim 9 , further comprising:
compacting multiple small attributes together into a single attribute stored in the long storage array.
11. A system for supporting metadata compression in a distributed data grid, comprising:
one or more microprocessors;
the distributed data grid, which runs on the one or more microprocessors, that operates to store a plurality of keys in a radix tree in a compact data structure in a heap, wherein
each said key is associated with an entry of a journal off-heap;
look up the radix tree for a key, wherein the key is associated with a journal ticket that represents a virtual address of an entry in the journal; and
use the journal ticket to manage the entry in the journal that is associated with said key.
12. The system according to claim 11 , wherein:
the journal off-heap includes a RAM journal, and a flash journal that operates to transparently handle data overflow from the RAM journal.
13. The system according to claim 11 , wherein:
the the heap is a virtual machine (VM) heap.
14. The system according to claim 11 , wherein each key in the radix tree corresponds to a value.
15. The system according to claim 14 , wherein:
a value corresponding to a key is stored directly in the radix tree if the size of the value is small.
16. The system according to claim 14 , wherein:
one or more journal tickets are stored in the radix tree, wherein each said journal ticket is associated with a key.
17. The system according to claim 14 , wherein:
one or more journal tickets are stored in a long storage array, wherein each said journal ticket is associated with a key.
18. The system according to claim 17 , wherein:
the key stored in the radix tree is associated with a slot number, wherein the slot number indicates a location in the long storage array where said journal ticket stores, and wherein the long storage array stores additional attributes that are associated with the key.
19. The system according to claim 18 , wherein:
multiple small attributes are compacted together into a single attribute stored in the long storage array.
20. A non-transitory machine readable storage medium having instructions stored thereon for supporting metadata compression in a distributed data grid that when executed cause a system to perform the steps comprising:
storing a plurality of keys in a compact data structure in a heap, wherein said compact data structure is a radix tree and each said key is associated with an entry of a journal off-heap, wherein each said key is associated with an entry in a journal off-heap;
looking up the radix tree for a key, wherein the key is associated with a journal ticket that represents a virtual address of an entry in the journal; and
using the journal ticket to manage the entry in the journal that is associated with said key.