Data storage testing
Methods, systems, and apparatus, including computer program products, for creating test directories and files in a logical storage in a data storage system. The directories include a root directory subdirectories in the root directory. Each directory has a directory key name that has a prefix value; a subdirectory value, the subdirectory value defining a number of immediate subdirectories in the directory; a subdirectory prefix value, the subdirectory prefix value defining a prefix value of each directory key name of each immediate subdirectory in the directory; and a file value, the file value defining a number of files stored in the directory. The contents of any directory can be checked against the directory key name to determine if a data storage error has occurred during testing.
1. A system, comprising:
a data processing apparatus;
a data store in data communication with the data processing apparatus and storing instructions encoded on a computer readable medium, the instructions executable by the data processing apparatus and operable to cause the data processing apparatus to perform operations comprising:
accessing a data storage system in data communication with the data processing apparatus;
creating a plurality of directories in a logical storage in the data storage system, the directories including a root directory and a plurality of subdirectories in the root directory, each directory having a directory key name comprising:
a prefix value;
a subdirectory value, the subdirectory value being a number defining a total number of immediate subdirectories in the directory;
a subdirectory prefix value, the subdirectory prefix value defining a prefix value of each directory key name of each immediate subdirectory in the directory; and
a file value, the file value being a number defining a total number of files stored in the directory.
2. The system of claim 1 , wherein the prefix value, subdirectory value, subdirectory prefix value and file values are each based on respective outputs of a pseudo-random number generator.
3. The system of claim 1 , wherein:
the subdirectories include leaf directories that have no subdirectories; and
the subdirectory value for the directory key name for each leaf directory is a null value that defines zero immediate subdirectories in the leaf directory.
4. The system of claim 1 , wherein creating the plurality of directories comprises the operations of:
creating a prefix queue;
generating a prefix value for a new directory;
generating a subdirectory value, a subdirectory prefix value, and a file value for the new directory;
creating the new directory with a directory key name comprising the prefix value, the subdirectory value, the subdirectory prefix value, and the file value;
selecting the new directory as a current directory;
creating a plurality of subdirectory prefix values based on the subdirectory value and the subdirectory prefix value directory key name of the current directory;
storing the plurality of subdirectory prefix values in the prefix queue; and
writing a number of files in the current directory, the number of files defined by the file value in the directory key name of the current directory.
5. The system of claim 4 , wherein creating the plurality of directories further comprises the operations of:
iteratively selecting subdirectory prefix values that are next in the prefix queue as prefix values for new directories, generating respective subdirectory values, subdirectory prefix values, and file value for the new directories, storing the plurality of subdirectory prefix values in the prefix queue, and creating the new directories with respective directory key names comprising the prefix values, the subdirectory values, the subdirectory prefix values, and the file values; and
determining whether an end condition has occurred after each iteration;
if an end condition has not occurred, then writing the number of files in a respective current directory defined by the file value in the directory key name of the current directory;
if an end condition has occurred for an iteration, then creating directories corresponding to only the subdirectory prefix values that are stored in the prefix queue when the end condition is determined to have occurred.
6. The system of claim 5 , wherein creating directories corresponding to only the subdirectory prefix values that are stored in the prefix queue when the end condition is determined to have occurred comprises the operations of:
generating null values for subdirectory values in each iteration subsequent to the determination that the end condition occurred.
7. The system of claim 6 , wherein determining whether an end condition has occurred after each iteration comprises determining if a current directory corresponds to a maximum directory path depth.
8. The system of claim 6 , wherein each file name is based on:
a prefix value of the directory in which the file is stored; and
a size value that defines a size of the file.
9. The system of claim 7 , wherein the instructions are further operable to cause the data processing apparatus to perform operations comprising:
accessing each of the directories, and for each directory:
determining that a storage error has occurred if the number of files stored in the directory does not match the number of files defined by the file value in the directory key name of the directory; and
determining that a storage error has occurred if the number of immediate subdirectories in the directory does not match the number of immediate subdirectories defined by the subdirectory value in the directory key name of the directory.
10. The system of claim 9 , wherein the instructions are further operable to cause the data processing apparatus to perform operations comprising:
determining that a storage error has occurred if a portion of each file name of files stored in the directory does not match the prefix value of the directory in which the file is stored.
11. The system of claim 10 , wherein the instructions are further operable to cause the data processing apparatus to perform operations comprising:
determining that a storage error has occurred if the size of each file does not match the size defined by the size value of the file name.
12. The system of claim 6 , wherein each directory key name further comprises a content value that is used to determine the content of files stored in the directory, and wherein each file name stored in a directory is based on:
a content value of the directory in which the file is stored; and
a size value that defines a size of the file.
13. The system of claim 4 , wherein each file name is based on:
a prefix value of the directory in which the file is stored; and
a size value that defines a size of the file.
14. The system of claim 13 , wherein:
the size value defines the size of the file in bytes; and
each byte is random based on the file name.
15. A computer-implemented method, comprising:
creating a prefix queue in a computer memory;
generating in the computer memory a prefix value for a new directory;
generating in the computer memory a subdirectory value that is a number defining a total number of immediate subdirectories in the new directory, a subdirectory prefix value defining a prefix value of each directory key name of each immediate subdirectory to be created in the new directory, and a file value for the new directory that is a number defining a total number of files to be stored in the new directory;
creating in a logical storage the new directory with a directory key name comprising the prefix value, the subdirectory value, the subdirectory prefix value, and the file value;
selecting the new directory as a current directory;
creating in the computer memory a plurality of subdirectory prefix values based on the subdirectory value and the subdirectory prefix value of the directory key name of the current directory;
storing the plurality of subdirectory prefix values in the prefix queue in the computer memory; and
writing the number of files in the current directory defined by the file value in the directory key name of the current directory.
16. The method of claim 15 , comprising:
iteratively selecting subdirectory prefix values that are next in the prefix queue as prefix values for new directories, generating respective subdirectory values, subdirectory prefix values, and a file value for the new directories, storing the plurality of subdirectory prefix values in the prefix queue, and creating the new directories with respective directory key names comprising the prefix values, the subdirectory values, the subdirectory prefix values, and the file values;
determining whether an end condition has occurred after each iteration;
if an end condition has not occurred, then writing the number of files in a respective current directory defined by the file value in the directory key name of the current directory; and
if an end condition has occurred for an iteration, then creating directories corresponding to only the subdirectory prefix values that are stored in the prefix queue when the end condition is determined to have occurred.
17. The method of claim 16 , wherein creating directories corresponding to only the subdirectory prefix values that are stored in the prefix queue when the end condition is determined to have occurred comprises the operations of:
generating null values for subdirectory values in each iteration subsequent to the determination that the end condition occurred.
18. The method of claim 16 , wherein determining whether an end condition has occurred after each iteration comprises determining if a current directory corresponds to a maximum directory path depth.
19. The method of claim 16 , wherein each file name is based on:
a prefix value of the directory in which the file is stored; and
a size value that defines the size of the file.
20. The method of claim 19 , further comprising:
accessing each of the directories, and for each directory:
determining that a storage error has occurred if the number of files stored in the directory does not match the number of files defined by the file value in the directory key name of the directory; and
determining that a storage error has occurred if the number of immediate subdirectories in the directory does not match the number of immediate subdirectories defined by the subdirectory value in the directory key name of the directory.
21. The method of claim 20 , further comprising:
determining that a storage error has occurred if the portion of each file name of files stored in the directory does not match the prefix value of the directory in which the file is stored.
22. The method of claim 21 , further comprising:
determining that a storage error has occurred if the size of each file does not match the size defined by the size value of the file name.
23. The method of claim 22 , wherein:
the size value defines the size of the file in bytes; and
each byte is a random byte based on the file name.
24. The method of claim 16 , further comprising:
determining a set of file attributes for a file from a file name of the file; and
setting the attributes of the file to the determined set of file attributes.
25. A system, comprising:
a data processing apparatus;
a data store in data communication with the data processing apparatus and storing instructions encoded on a computer readable medium, the instructions executable by the data processing apparatus and operable to cause the data processing apparatus to perform operations comprising:
accessing a data storage system in data communication with the data processing apparatus;
creating a plurality of directories in a logical storage in the data storage system, the directories including a root directory and a plurality of subdirectories in the root directory, each directory having a directory key name comprising a plurality of values that are deterministic of:
a number of immediate subdirectories stored in the directory;
at least a portion of the directory key name of each subdirectory;
a number of files stored in the directory; and
at least a portion of the file name of each file stored in the subdirectory; wherein each directory key name comprises:
a prefix value;
a subdirectory value, the subdirectory value defining a number of immediate subdirectories in the directory;
a subdirectory prefix value, the subdirectory prefix value defining a prefix value of each directory key name of each immediate subdirectory in the directory; and
a file value, the file value defining a number of files stored in the directory.