Storage system with self describing data

1. A method for storing data, comprising:

receiving a request to write a first datum to persistent storage, wherein the first datum is defined using a first logical address;

determining a first physical address in the persistent storage, wherein the first physical address comprises a first block ID and first sub block ID;

writing the first datum to a first location in the persistent storage identified by the first physical address;

generating a first table of contents entry (TE) comprising the first logical address, and the first sub block ID, wherein the first TE is not stored in a sub block identified by the first sub block ID and wherein the first logical address and the first sub block ID are not stored in the sub block identified by the first sub block ID; and

writing the first TE to a second physical address in the persistent storage, wherein the second physical address comprises the first block ID and a second sub block ID, wherein a second sub block corresponds to the second sub block ID, and wherein the second sub block is located within a first block corresponding to the first block ID and wherein a sub block identified by the second sub block ID does not include the first datum.

2. The method of claim 1 , further comprising:

receiving a request to write a second datum to the persistent storage, wherein the second datum is defined using a second logical address;

determining a third physical address in the persistent storage, wherein the third physical address comprises the first block ID and third sub block ID;

writing the second datum to the third physical address;

generating a second TE comprising the second logical address and the third sub block ID, wherein the second TE is not stored in a sub block identified by the third sub block ID;

writing the second TE to a fourth physical address in the persistent storage, wherein the fourth physical address comprises the first block ID and a fourth sub block ID, wherein a fourth sub block corresponds to the fourth sub block ID, and wherein the fourth sub block is located within the first block, and wherein a sub block identified by the fourth sub block ID does not include the first datum.

3. The method of claim 1 , wherein the persistent storage is solid state memory, and wherein the first block is a NAND block, and the first sub block is a NAND page.

4. The method of claim 1 , wherein the fourth sub block comprises a reference to the second sub block.

5. A method for storing data, comprising:

receiving a request to write a first datum to persistent storage, wherein the first datum is defined using a first logical address;

determining a first physical address in persistent storage, wherein the first physical address comprises a first block ID and a first page ID;

writing a first frag comprising a copy of the first datum to a first location in the persistent storage identified by the first physical address;

generating a first table of contents entry (TE) comprising the first logical address and the first page ID, wherein the first TE is not stored in a page identified by the first page ID and wherein the first logical address and the first page ID are not stored in the page identified by the first page ID;

receiving a request to write a second datum to the persistent storage, wherein the second datum is defined using a second logical address;

determining a second physical address in the persistent storage, wherein the second physical address comprises the first block ID and a second page ID;

writing a second frag comprising a copy of the second datum to a second location in the persistent storage by identified by the second physical address;

generating a second TE comprising the second logical address and the second page ID, wherein the second TE is not stored in a page identified by the second page ID and wherein the second logical address and the second page ID are not stored in the page identified by the second page ID;

generating a table of contents (TOC) page, wherein the TOC page comprises the first TE and the second TE, and wherein the TOC page does not include the first datum and the second datum; and

writing the TOC page to a third physical address in the persistent storage, wherein the third physical address comprises the first block ID and a third page ID.

6. The method of claim 5 , further comprising:

receiving a request to write a third datum to the persistent storage, wherein the third datum is defined using a third logical address;

determining a fourth physical address in the persistent storage, wherein the fourth physical address comprises the first block ID and a fourth page ID;

writing a third frag comprising a copy of the third datum to the fourth physical address;

generating a third TE comprising the third logical address, and the fourth page ID, wherein the third TE is not stored in a page identified by the third page ID;

generating a second TOC page comprising the third TE;

writing the second TOC page to a fifth physical address in the persistent storage, wherein the fifth physical address comprises the first block ID and a fifth page ID and, wherein the second TOC page does not include the third datum.

7. The method of claim 6 , wherein a fifth page corresponding to the fifth page ID is the last page in the first block.

8. The method of claim 7 , wherein the second TOC page comprises a reference to the first TOC page.

9. The method of claim 8 , wherein the reference is located at the end of the second TOC page.

10. The method of claim 5 , wherein prior to writing the first frag to the first physical address in the persistent storage, a first copy of the first frag is stored in a first memory and a second copy of the first frag is stored in a second memory, wherein a third TE is generated for the second copy of the first frag, wherein the third TE is erased once the TOC page is written to the third physical memory address.

11. The method of claim 10 , wherein the first memory is located in a first storage module and the second memory is located in a second storage module, and wherein the persistent storage is located in the first memory module.

12. The method of claim 5 , wherein the first TE further comprises at least one selected from a group consisting of a birth time field, a type field, a logical length field, and a kind field.

13. The method of claim 5 , wherein the first logical address comprises an object ID and an offset ID.

14. A method for populating an in-memory datum structure, comprising:

(a) selecting a first block in a persistent storage;

(b) extracting a last page in the first block, wherein the first block is associated with a first block ID;

(c) extracting a first table of contents entry (TE) from the last page in the first block, wherein the first TE comprises a first logical address for a first datum, and a first page ID identifying a page in the first block in which the first datum is located, wherein the page is not the last page and wherein the first logical address does not include the page ID and wherein the first logical address and the first page ID are not stored in the page;

(d) generating a first physical address for the first datum using the first block ID and the first page ID extracted from the first TE, wherein the first physical address comprises the first page ID;

(e) hashing the first logical address to obtain a first hash value; and

(f) populating the in-memory data structure with a first mapping between the first hash value and the first physical address.

15. The method of claim 14 , further comprising:

(g) selecting a second block in the persistent storage;

(h) extracting a last page in the second block, wherein the second block is associated with a second block ID;

(i) extracting second TE from the last page in the second block, wherein the second TE comprises a second logical address for a second datum, and a second page ID corresponding to a page in the second block in which the second datum is located;

(j) generating a second physical address for the second datum using the second block ID and the second page ID extracted from the second TE;

(k) hashing the second logical address to obtain a second hash value; and

(l) populating the in-memory data structure with a second mapping between the second hash value and the second physical address.

16. The method of claim 15 , wherein (a)-(c) and (g)-(i) occur in parallel.

17. The method of claim 15 , wherein the first block is located in a first storage module, the second block is located in the second storage module, and wherein the in-memory data structure is located in memory external to the first storage module and the second storage module.

18. The method of claim 14 , further comprising:

making a determination that the last page in the first block comprises a reference to the second page in the first block, wherein the second page comprises a second TE, wherein the second TE comprises a second logical address for a second datum, and a second page ID corresponding to a page in the first block in which the second datum is located;

generating a second physical address for the second datum using the first block ID, and the second page ID;

hashing the second logical address to obtain a second hash value; and

populating the in-memory data structure with a second mapping between the second hash value and the second physical address.

19. The method of claim 14 , wherein the persistent storage is solid state memory.

20. The method of claim 14 , wherein the in-memory data structure is populated with all TEs in the persistent storage prior to any operations in a group consisting of a read operation, a write operation, and an erase operation being performed on any datum stored in the persistent storage.

21. The method of claim 14 , wherein the first logical address comprises an object ID and an offset ID.

22. A method for storing data, comprising:

receiving a request to write a first datum to persistent storage, wherein the first datum is defined using a first logical address;

determining a first physical address in persistent storage, wherein the first physical address comprises a first block ID and a first page ID;

prior to writing a first frag to a first physical address in the persistent storage:

storing a first copy of the first frag is in a first memory and a second copy of the first frag in a second memory;

writing the first frag comprising a copy of the first datum to the first physical address;

generating a first table of contents entry (TE) comprising the first logical address and the first page ID;

receiving a request to write a second datum to the persistent storage, wherein the second datum is defined using a second logical address;

determining a second physical address in the persistent storage, wherein the second physical address comprises the first block ID and a second page ID;

writing a second frag comprising a copy of the second datum to the second physical address;

generating a second TE comprising the second logical address and the second page ID;

generating a table of contents (TOC) page, wherein the TOC page comprises the first TE and the second TE; and

writing the TOC page to a third physical address in the persistent storage, wherein the third physical address comprises the first block ID and a third page ID, wherein a third TE is generated for the second copy of the first frag, wherein the third TE is erased once the TOC page is written to the third physical memory address.