More elegant exastore apparatus and method of operation
An apparatus to scale for multiple petabyte backup in redundant locations. Workload is automatically shared among many servers by a characteristic derived from the content itself. Duplicate storage is eliminated by checking for the existence or absence of simple files and appending client identification to files shared among multiple subscribers. Replication depends on simple atomic file operations rather than use of tables or databases. An efficient storage method for much larger quantities of data than conventional services.
1. A computer-implemented method for control of an apparatus comprising a processor configured by instructions stored on a non-transitory computer readable medium, the method comprising:
receiving from a backup client of a plurality of backup clients distinguishable by a client identifier, a substantially unique characteristic derived from a shard;
determining when the shard is a previously stored shard or a new shard to be stored by determining if a flat file already exists which has a name comprising the substantially unique characteristic derived from the shard,
when the shard is a previously stored shard,
appending the client identifier of the backup client to an end of a previously stored uniquely named flat file,
when the shard is a new shard to be stored,
requesting and storing the shard into an assigned server, and
creating and storing a new uniquely named flat file which contains an encryption key and the client identifier of the backup client, wherein the name of the new uniquely named flat file comprises the substantially unique characteristic derived from the shard,
creating a token file to request replication of the shard and the uniquely named flat file to at least one non-transitory computer-readable medium on another apparatus.
2. The method of claim 1 wherein the substantially unique characteristic derived from the shard is a plurality of cryptographic hashes concatenated into a delimited string.
3. The method of claim 1 wherein the shard is a portion of a data file.
4. The method of claim 1 further comprising
deleting the token file when a shard of a uniquely named flat file has been successfully replicated.
5. The method of claim 4 further comprising
assigning the shard and the substantially named flat file to a server according to an excerpt from the substantially unique characteristic derived from the shard.
6. The method of claim 5 wherein the excerpt is a range of characters in the substantially unique characteristic.
7. The method of claim 6 further comprising
assigning the shard to a group of servers based on the client identifier of the backup client.
8. A method comprising,
receiving a shard and meta-data of a shard from a backup client, wherein the meta-data comprises a key for the shard;
assigning the shard to a server based on the first n bytes of the key,
creating a file containing an encryption key and a client id of the backup client, wherein the file containing the encryption key and the client id of the backup client referencing the shard is named with the metadata of the shard; and
storing the file and the shard on the server.
9. The method of claim 8 wherein storing the file and storing the shard comprises
adding an action request file to the queue of the selected server and its replication server(s).
10. The method of claim 9 further comprising
assigning to a group of servers based on the backup client ID.
11. The method of claim 8 wherein the key is a hash computed on the shard.
12. An apparatus comprising
a file management circuit to determine if a uniquely named flat file exists,
when the uniquely named flat file exists, to append a client identifier to the end of the file,
when the uniquely named flat file does not exist, to create and store a flat file comprising an encryption key and a client identifier and to request and store a shard with the substantially unique characteristic as the name of the flat file,
a receiver circuit configured to receive from a backup client a substantially unique characteristic derived from a shard and a client identifier, to assign the shard to a server according to an excerpt from the substantially unique characteristic, to assign the shard to a group of servers according to the client identifier, and request and receive a shard when the shard has been determined not to have been previously stored,
a file write and delete circuit configured to
append an additional client identifier to an end of a stored uniquely named flat file,
create and store a new uniquely named flat file containing an encryption key and a client identifier when the flat file does not already exist,
create at least one token file to request replication of a shard or a uniquely named flat file, and
delete the token file when the requested replication has succeeded.