Methods, systems, and computer readable media for automatically deriving hints from accesses to a storage device and from file system metadata and for optimizing utilization of the storage device based on the hints

1. A storage device comprising:

non-volatile memory; and

a hint derivation module configured to analyze at least one of:

input/output (I/O) command sequences for I/O operations involving the non-volatile memory; and

file system metadata comprising a master file table (MFT);

the hint derivation module configured to automatically derive hints regarding expected access patterns to the non-volatile memory based on the analyzing and to use the hints to optimize utilization of the non-volatile memory, wherein the hint derivation module is further configured to:

detect an I/O operation pertaining to the MFT;

locate an entry in the MFT corresponding to the detected I/O operation, the entry corresponding to a particular file;

parse the entry of the MFT to determine whether the entry comprises resident file data, the resident file data comprising file data of the particular file stored within the MFT;

record a first hint in response to determining that the entry comprises resident file data stored within the MFT, the first hint including an indication that file data of the particular file are stored within the entry of the MFT; and

record a second hint that does not include the indication in response to determining that the entry does not comprise resident file data-stored within the MFT, the second hint indicating that file data of the particular file are stored outside of the MFT.

2. The memory device of claim 1 , wherein the hint derivation module determines that the entry comprises resident file data stored within the MFT in response to determining that the entry comprises a resident data stream.

3. The memory device of claim 1 , wherein the non-volatile memory comprises flash memory having one or more of a two-dimensional configuration and a three-dimensional configuration, the flash memory comprising one or more of NAND flash memory and NOR flash memory.

4. The memory device of claim 1 , wherein the non-volatile memory comprises a solid state memory and at least one disk storage device.

5. The memory device of claim 1 , wherein the hint derivation module is configured to automatically derive hints from the I/O operations involving the non-volatile memory.

6. The memory device of claim 5 , wherein the hint derivation module is configured to parse payload portions of the I/O command sequences to identify patterns that are indicative of respective hints.

7. The memory device of claim 6 , wherein the hint derivation module is configured to identify a matching file name pattern and a corresponding file type associated with the file name pattern in response to parsing a payload portion of an I/O command sequence.

8. The memory device of claim 7 , wherein the hint derivation module is configured to mark logical block addresses associated with the I/O command sequence with a hint that corresponds to the identified file type.

9. The memory device of claim 5 , wherein the hint derivation module is configured to determine an access frequency associated with an address range specified in one or more of the I/O operations and to determine whether the determined access frequency is consistent with a current hint stored for the address range.

10. The memory device of claim 5 , wherein the hint derivation module is configured to derive the hints in-line with the I/O operations.

11. The memory device of claim 5 , wherein the hint derivation module is configured to derive hints asynchronously with respect to the I/O operations.

12. The memory device of claim 1 , wherein the hint derivation module is configured to automatically derive hints from the file system metadata in response to one or more of: the file system metadata being stored in the non-volatile memory, file system metadata being written to the non-volatile memory, and file system metadata being read from the non-volatile memory.

13. The memory device of claim 12 , wherein the hint derivation module is configured to derive hints by parsing the file system metadata.

14. The memory device of claim 12 , wherein the hint derivation module is configured to parse at least one of file timestamp attributes, file access frequency attributes, file name attributes, and file extent maps.

15. A method, comprising:

analyzing one or more of:

input/output (I/O) command sequences for I/O operations involving a non-volatile memory; and

file system metadata comprising a master file table (MFT);

automatically deriving hints regarding expected access patterns to the non-volatile memory based on the analyzing, wherein automatically deriving the hints includes identifying an I/O command sequence for an I/O operation that pertains to the MFT and, wherein, in response to the identifying, the method further comprises:

parsing the MFT to locate an entry in the MFT corresponding to the I/O operation;

analyzing the entry of the MFT to determine whether the entry comprises one of: a resident entry configured to store file data of a file within the MFT and a non-resident entry configured to reference file data stored outside of the MFT; and

in response to determining that the entry comprises a resident entry, marking the entry of the MFT with a first hint, the first hint comprising an indicator designating the entry as a resident entry configured to store the file data of the file within the MFT,

in response to determining that the entry comprises a non-resident entry, marking the entry of the MFT with a second hint that does not comprise the indicator, the second hint indicating that the entry is configured to reference file data stored outside of the MFT; and

using the hints to optimize utilization of the non-volatile memory.

16. The method of claim 15 , wherein determining whether the entry of the MFT is configured to store file data within the MFT comprises determining whether the entry comprises a resident data stream.

17. The method of claim 15 wherein the non-volatile memory comprises flash memory having one or more of a two-dimensional configuration and a three-dimensional configuration, the flash memory comprising one or more of NAND flash memory and NOR flash memory.

18. The method of claim 15 , wherein the non-volatile memory comprises solid state memory and at least one disk storage device.

19. The method of claim 15 , further comprising receiving the I/O command sequences at a storage device.

20. The method of claim 15 , wherein identifying the I/O command sequence comprises matching a payload portion of the I/O command sequence to a pattern.

21. The method of claim 15 , wherein automatically deriving the hints comprises parsing the I/O command sequences to determine file types for the I/O command sequences based on file name patterns identified within data of the I/O command sequences.

22. The method of claim 21 , wherein using the hints comprises marking logical block addresses corresponding to the I/O operations of the I/O command sequences with the file types determined for the I/O command sequences.

23. The method of claim 15 , further comprising:

storing the hints on the non-volatile memory, each stored hint associated with a respective address range;

determining access frequencies associated with address ranges in response to analyzing the I/O command sequences; and

determining whether an access frequency determined for a particular address range is consistent with a stored hint associated with the particular address range.

24. The method of claim 15 , wherein automatically deriving the hints comprises automatically deriving a hint in-line with an I/O operation of an I/O command sequence.

25. The method of claim 15 , wherein automatically deriving the hints comprises automatically deriving a hint asynchronously with respect to an I/O operation of an I/O command sequence.

26. The method of claim 15 , wherein automatically deriving the hints comprises analyzing the file system metadata in response to one or more of: file system metadata being stored in the non-volatile memory, file system metadata being written to the non-volatile memory, and file system metadata being read from the non-volatile memory.

27. The method of claim 26 , wherein automatically deriving the hints comprises parsing the file system metadata to determine file attributes corresponding to respective address ranges.

28. The method of claim 26 , wherein automatically deriving the hints comprises parsing the file system metadata to determine one or more of file name attributes, file timestamp attributes, file access frequency attributes, and file extent maps in the file system metadata.

29. A non-transitory computer readable medium having stored thereon executable instructions that, when executed by a processor, control the processor to perform operations in a non-volatile storage device including non-volatile memory, the operations comprising:

automatically deriving hints regarding expected access patterns to the non-volatile memory by parsing input/output operations and file system metadata, the file system metadata comprising a master file table (MFT); and

utilizing the hints to optimize utilization of the non-volatile memory, wherein automatically deriving the hints further comprises:

parsing an input/output operation that matches an MFT pattern;

identifying an entry in the MFT that corresponds to the parsed input/output operation;

determining whether a file associated with the entry comprises one of a resident file and a non-resident file, comprising determining that the file comprises a resident file in response to the entry comprising file data of the file stored within the MFT, and determining that the file comprises a non-resident file in response to the entry referencing file data stored external to the MFT;

marking the entry with a first hint indicating that the file associated with the entry comprises a resident file and that the file data of the file are stored within the MFT in response to determining that the file comprises a resident file; and

marking the entry with a second hint indicating that the file comprises a non-resident file and that the entry references file data stored external to the MFT in response to determining that the file comprises a non-resident file.