The present invention can provide an OS independent system for storing data within a storage system. The invention can provide a directory system which is designed dynamically to adapt based upon the caching memory available for searching directories. Received files can be stored in a current directory until a predetermined limit is reached. In parallel, a database can be created to record which files are stored in which directory. This database can be designed to be kept in physical memory to minimize file access latency. This arrangement provides that a data storage system can store data in a simple order of receipt manner while also managing the storage structure to limit the number of data objects in any given container, thus preventing a search function analysing any given container from needing to access an excessive number of data objects and thus slow down the search to an unacceptable level.
1. A data storage system comprising:
one or more computer systems configured to execute program instructions, wherein the program instructions are executable to:
receive a data object for storage from a remote storage agent;
assign a first identifier to the data object, wherein the first identifier specifies an endpoint container in which to store the data object;
assign a second identifier to the data object, wherein the second identifier specifies a location of the data object within the endpoint container;
determine whether the second identifier exceeds a predetermined value, wherein said second identifier exceeds the predetermined value at least once during execution of said instructions;
in response to determining the second identifier does not exceed the predetermined value:
store the data object in the endpoint container at a location identified by the second identifier;
store a storage location record including the second identifier and the first identifier indicating where the data object is stored; and
increment the second identifier, thereby creating a new second identifier;
in response to determining the second identifier does exceed the predetermined value:
reset the second identifier to an initial value, thereby creating a reset second identifier;
assign the second identifier with the reset value to the data object;
increment the first identifier and create a new endpoint container which is identified by the incremented first identifier;
store the data object in the new endpoint container at a location identified by the reset second identifier;
store a storage location record including the reset second identifier and the incremented first identifier indicating where the data object is stored; and
increment the reset second identifier.
2. The system of claim 1 , wherein the program instructions are executable to store the storage location record for the data object in a database of storage location records.
3. The system of claim 2 , wherein a maximum number of data objects which can be stored by the data storage system is limited to enable the database of storage location records to be maintained in physical memory of a host computer of the data storage system.
4. The system of claim 1 , wherein the storage location record includes a pointer to the particular storage container in which the data object was stored and a unique identifier for that data object.
5. The system of claim 1 , wherein the array structure is hierarchical and comprises containers configured to contain other containers and containers configured to contain data objects.
6. The system of claim 5 , wherein a plurality of containers for data objects reside within a single container for containers.
7. The system of claim 6 , comprising a plurality of tiers of container containers.
8. The system of claim 5 , wherein the hierarchical structure is defined using a pair of container size parameters, wherein the pair of container size parameters comprises one parameter to define a maximum number of containers that can exist in a container and one parameter to define a maximum number of data objects that can exist in an endpoint container.
9. The system of claim 1 , wherein new data objects received for storage in a container are assigned identifiers in sequential order.
10. The system of claim 8 , wherein values of the parameters are set such that directory information for each container fits into a single block on the physical disk which hosts the container array.
11. The system of claim 1 , wherein said first identifier is used in a transformation comprising a modulo operation to identify a physical storage location.
12. A method of storing data received at a data storage system, the method comprising:
receiving a data object for storage from a remote storage agent;
assigning a first identifier to the data object, wherein the first identifier specifies an endpoint container in which to store the data object;
assigning a second identifier to the data object, wherein the second identifier specifies a location of the data object within the endpoint container;
determining whether the second identifier exceeds a predetermined value, wherein said second identifier exceeds the predetermined value at least once during operation;
in response to determining the second identifier does not exceed the predetermined value:
storing the data object in the endpoint container at a location identified by the second identifier;
storing a storage location record including the second identifier and the first identifier indicating where the data object is stored; and
incrementing the second identifier, thereby creating a new second identifier;
in response to determining the second identifier does exceed the predetermined value:
resetting the second identifier to an initial value, thereby creating a reset second identifier;
assigning the second identifier with the reset value to the data object;
incrementing the first identifier and creating a new endpoint container which is identified by the incremented first identifier;
storing the data object in the new endpoint container at a location identified by the reset second identifier;
storing a storage location record including the reset second identifier and the incremented first identifier indicating where the data object is stored; and
incrementing the reset second identifier.
13. The method of claim 12 , further comprising assigning identifier to new data objects received for storage in sequential order.
14. The method of claim 12 , wherein said first identifier is used in a transformation comprising a modulo operation to identify a physical storage location.
15. A computer-readable storage medium storing program instructions executable to:
receive a data object for storage from a remote storage agent;
assign a first identifier to the data object, wherein the first identifier specifies an endpoint container in which to store the data object;
assign a second identifier to the data object, wherein the second identifier specifies a location of the data object within the endpoint container;
determine whether the second identifier exceeds a predetermined value, wherein said second identifier exceeds the predetermined value at least once during execution of said instructions;
in response to determining the second identifier does not exceed the predetermined value:
store the data object in the endpoint container at a location identified by the second identifier;
store a storage location record including the second identifier and the first identifier indicating where the data object is stored; and
increment the second identifier, thereby creating a new second identifier;
in response to determining the second identifier does exceed the predetermined value:
reset the second identifier to an initial value, thereby creating a reset second identifier;
assign the second identifier with the reset value to the data object;
increment the first identifier and create a new endpoint container which is identified by the incremented first identifier;
store the data object in the new endpoint container at a location identified by the reset second identifier;
store a storage location record including the reset second identifier and the incremented first identifier indicating where the data object is stored; and
increment the reset second identifier.
16. The storage medium of claim 15 , wherein identifiers assigned to new data objects received for storage are assigned in sequential order.
17. The storage medium of claim 15 , wherein said first identifier is used in a transformation comprising a modulo operation to identify a physical storage location.