System and method for content addressable storage
View Patent ↗Information, such as files received from a client, etc., is stored in a storage system, such as a content addressable storage system. A file server receives data from a client and chunks the data into blocks of data. The file server also generates metadata for use in forming a data structure. The blocks of data are stored in a block store and a copy of the data blocks and the metadata are locally cached at the file server. A commit server retrieves the metadata. In at least one embodiment, the metadata is retrieved from an update log shared between the file server and the commit server. Based on the retrieved metadata, the commit server generates a version of a data structure. The data structure is then stored at the block store.
1. A method of storing information in a storage system comprising:
chunking data into one or more blocks of data with a file server;
generating metadata operations of a file system with the file server;
storing the one or more blocks of data in a block store;
retrieving the metadata operations with a commit server;
generating a data structure based on the metadata operations and the one or more blocks of data with the commit server;
storing the data structure in the block store,
storing the metadata operations in an update log; and
wherein retrieving the metadata operations with the commit server comprises
retrieving the metadata operations from the update log;
wherein chunking data into one or more blocks of data
and generating metadata operations with the file server further comprises associating a timestamp with the generated metadata operations and comprising:
storing the generated metadata operations at the file server; and
cleaning the metadata stored at the file server with the metadata stored as a version of the data structure based at least in part on the one or more timestamps associated with the retrieved metadata operations.
2. The method of claim 1 wherein storing the one or more blocks of data in the block store and storing the data structure in the block store occur substantially simultaneously.
3. The method of claim 1 further comprising:
receiving data from a client at the file server; and
wherein chunking data into one or more blocks of data with the file server comprises separating the data received from the client into the one or more blocks of data.
4. A system for storing data comprising:
a file server configured to chunk data into blocks of data and generate metadata operations;
a memory configured to store the blocks of data;
a commit server configured for retrieving the metadata operations, generating a data structure indicative of the metadata operations, storing store the data structure in the memory, storing the data structure in the block store, and storing the metadata operations in an update log; and
wherein retrieving the metadata operations with the commit server comprises retrieving the metadata operations from the update log;
wherein chunking data into one or more blocks of data
and generating metadata operations with the file server further comprises associating a timestamp with the generated metadata operations and comprising:
storing the generated metadata operations at the file server; and
cleaning the metadata stored at the file server with the metadata stored as a version of the data structure based at least in part on the one or more timestamps associated with the retrieved metadata operations.
5. The system of claim 4 further comprising:
an update log configured to receive the metadata operations from the file server and pass the metadata operations to the commit server.
6. A method of storing information in a storage system comprising:
chunking data into one or more blocks of data with a file server;
generating metadata operations of a file system with the file server;
storing the one or more blocks of data in a block store;
retrieving the metadata operations with a commit server;
generating a data structure based on the metadata operations and the one or more blocks of data with the commit server;
storing the data structure in the block store, and
storing the metadata operations in an update log; and
wherein retrieving the metadata operations with the commit server comprises
retrieving the metadata operations from the update log;
wherein generating the data structure comprising the metadata with the commit server comprises:
retrieving one or more timestamps associated with the retrieved metadata operations;
modifying metadata in a metadata cache based on the retrieved metadata operations;
batching the metadata operations; and
storing the metadata as a version of the data structure based at least in part on the one or more timestamps associated with the retrieved metadata.
7. A system for storing data comprising:
a file server configured to chunk data into blocks of data and generate metadata operations;
a memory configured to store the blocks of data;
a commit server configured for retrieving the metadata operations, generating a data structure indicative of the metadata operations, storing the data structure in the memory, storing the data structure in the block store, and storing the metadata operations in an update log; and
wherein generating the data structure comprising the metadata with the commit server comprises:
retrieving one or more timestamps associated with the retrieved metadata operations;
modifying metadata in a metadata cache based on the retrieved metadata operations;
batching the metadata operations; and
storing the metadata as a version of the data structure based at least in part on the one or more timestamps associated with the retrieved metadata.