Distributed shared memory paging
Techniques for obtaining metadata may include: receiving, by a director, an I/O operation directed to a target offset of a logical device, wherein the director is located on a board including a local page table used by components on the board; querying the local page table for a global memory address of first metadata for the target offset of the logical device; and responsive to the local page table not having the global memory address of the first metadata for the target offset of the logical device, using at least a first indirection layer to obtain the global memory address of the first metadata. The global memory may be a distributed global memory including memory segments from multiple different boards each including its own local page table. Compare and swap operations may be used to perform atomic operations to ensure synchronized access when updating the distributed global memory.
1. A method of obtaining metadata comprising:
receiving, by a director, an I/O operation directed to a target offset of a logical device, wherein the director is located on a board including a local page table used by components on the board, wherein the board is one of a plurality of boards, wherein the local page table is one of a plurality of local page tables, and wherein each of the plurality of boards includes a different one of the plurality of local page tables used by components of said each board;
querying the local page table for a global memory address of first metadata for the target offset of the logical device; and
responsive to the local page table not having the global memory address of the first metadata for the target offset of the logical device, using at least a first indirection layer to obtain the global memory address of the first metadata, wherein the first indirection layer is used to obtain a first page including the first metadata for the logical device, wherein the first page is stored at a first global memory address and wherein the first page has a corresponding page descriptor that includes a node mask identifying which of the plurality of boards includes one of the plurality of local page tables with a reference to the first global memory address stored in the one local page table.
2. The method of claim 1 , wherein the global memory address is an address in a distributed global memory comprising a plurality of global memory segments stored on the plurality of boards, wherein a first of the plurality of global memory segments is included on the board with the director.
3. The method of claim 2 , wherein the first indirection layer includes a plurality of pointers to pages of metadata for the logical device, and wherein the first indirection layer is stored in the distributed global memory.
4. The method of claim 3 , wherein the pages of metadata for the logical device include the first page and wherein the pages of metadata for the logical device are configured to be paged in and out of the distributed global memory at various points in time.
5. The method of claim 4 , wherein each of the plurality of boards includes at least one director that uses a particular one of the plurality of local pages tables, included on said each board, to obtain metadata for at least one logical device.
6. The method of claim 1 , wherein the first metadata is used in connection with servicing the I/O operation, and wherein the first metadata includes any of:
location information identifying a physical storage area mapped to the target offset of the logical device;
cache location information denoting if user data of the target offset of the logical device is stored in cache and if so, identify a location in the cache where the user data is stored; and
information used to verify or validate the user data.
7. The method of claim 5 , wherein the first page including the first metadata has the corresponding page descriptor stored in the distributed global memory, wherein the page descriptor includes the node mask and further includes:
a lock used to control and synchronize access to the corresponding page descriptor;
an indicator denoting whether data of the first page stored in the distributed global memory is clean or dirty, wherein dirty indicates that the data of the first page stored in the distributed global memory is a more recent version than another copy of data of the first page stored on non-volatile storage; and
a key identifying data stored in the first page, or otherwise denoting that a page of global memory associated with the page descriptor is free.
8. The method of claim 7 , wherein the node mask is used in connection with first processing performed to remove the first page from the distributed global memory, wherein the first processing includes selectively sending a request to each of the plurality of boards identified in the node mask, the request requesting that said each board remove the reference to the first global memory address from one of the plurality of local page tables included on said each board.
9. The method of claim 8 , wherein a compare and swap operation is performed to write to the distributed global memory in connection with modifying at least one of: the page descriptor, the first indirection layer, and the first metadata of the first page stored at the first global memory address.
10. The method of claim 1 , wherein a plurality of indirection layers include the first indirection layer, and wherein the plurality of indirection layers are used to obtain the global memory address of the first metadata.
11. The method of claim 10 , wherein the global memory is a distributed global memory, the plurality of indirection layers are stored in the distributed global memory, and wherein at least a second of the plurality of indirection layers includes pages configured to be paged in and out of the distributed global memory.
12. The method of claim 11 , wherein the plurality of indirection layers includes global memory addresses of locations in the distributed global memory.
13. The method of claim 11 , wherein one of the indirection layers includes global memory addresses storing metadata for the logical device.
14. A system comprising:
one or more processors; and
a memory comprising code stored thereon that, when executed, performs a method of obtaining metadata comprising:
receiving, by a director, an I/O operation directed to a target offset of a logical device, wherein the director is located on a board including a local page table used by components on the board, wherein the board is one of a plurality of boards, wherein the local page table is one of a plurality of local page tables, and wherein each of the plurality of boards includes a different one of the plurality of local page tables used by components of said each board;
querying the local page table for a global memory address of first metadata for the target offset of the logical device; and
responsive to the local page table not having the global memory address of the first metadata for the target offset of the logical device, using at least a first indirection layer to obtain the global memory address of the first metadata, wherein the first indirection layer is used to obtain a first page including the first metadata for the logical device, wherein the first page is stored at a first global memory address and wherein the first page has a corresponding page descriptor that includes a node mask identifying which of the plurality of boards includes one of the plurality of local page tables with a reference to the first global memory address stored in the one local page table.
15. A non-transitory computer readable medium comprising code stored thereon that, when executed, performs a method of obtaining metadata comprising:
receiving, by a director, an I/O operation directed to a target offset of a logical device, wherein the director is located on a board including a local page table used by components on the board, wherein the board is one of a plurality of boards, wherein the local page table is one of a plurality of local page tables, and wherein each of the plurality of boards includes a different one of the plurality of local page tables used by components of said each board;
querying the local page table for a global memory address of first metadata for the target offset of the logical device; and
responsive to the local page table not having the global memory address of the first metadata for the target offset of the logical device, using at least a first indirection layer to obtain the global memory address of the first metadata, wherein the first indirection layer is used to obtain a first page including the first metadata for the logical device, wherein the first page is stored at a first global memory address and wherein the first page has a corresponding page descriptor that includes a node mask identifying which of the plurality of boards includes one of the plurality of local page tables with a reference to the first global memory address stored in the one local page table.
16. The non-transitory computer readable medium of claim 15 , wherein the global memory address is an address in a distributed global memory comprising a plurality of global memory segments stored on the plurality of boards, and wherein a first of the plurality of global memory segments is included on the board with the director.
17. The non-transitory computer readable medium of claim 16 , wherein the first indirection layer includes a plurality of pointers to pages of metadata for the logical device, and wherein the first indirection layer is stored in the distributed global memory.
18. The non-transitory computer readable medium of claim 17 , wherein the pages of metadata for the logical include the first page and wherein the pages of metadata for the logical device are configured to be paged in and out of the distributed global memory at various points in time.