Data archiving system
An encrypted file storage solution consists of a cluster of processing nodes, external data storage, and a software agent (the “File System Watcher”), which is installed on the application servers. Cluster sizes of one node up to many hundreds of nodes are possible. There are also remote “Key Servers” which provide various services to one or more clusters. The preceding describes a preferred embodiment, though in some cases it may be desirable to “collapse” some of the functionality into a smaller number of hardware devices, typically trading off cost versus security and fault-tolerance.
1. A computer-implemented method of archiving data files, comprising:
receiving the files from a client computer system, each of the files having associated therewith a unique file identifier;
serializing the files by assigning a consecutive serial number to each of the files;
storing each file and its assigned consecutive serial number to a first storage;
verifying the presence of each stored file serially using the stored consecutive serial numbers, wherein each consecutive serial number is accounted for and each file matches its associated unique file identifier;
reporting an error if one of the files is missing from the first storage; and
attaching a server identification field to each consecutive serial number to indicate a server that assigned the consecutive serial number;
creating an associative cryptographic hash of each file;
storing the associative cryptographic hash of each file to the first storage;
re-computing the associative cryptographic hash of a given one of the files;
verifying the authenticity of the given file by comparing its respective stored associative cryptographic hash to the re-computed cryptographic hash.
2. The computer-implemented method of claim 1 , further comprising:
providing the assigned consecutive serial number for retrieval of a corresponding stored file; and
retrieving the corresponding stored file, based on the assigned consecutive serial number.
3. The computer-implemented method of claim 2 , wherein the assigned consecutive serial number is provided to a user of the client computer system for retrieving the corresponding file by means of a HTTP symbolic link.
4. The computer-implemented method of claim 1 , further comprising: reporting an error when the comparison indicates a mismatch between the stored associative cryptographic hash and the re-computed hash for the given file.
5. The computer-implemented method of claim 4 , further comprising:
periodically replicating the first storage to a second storage;
if an error is reported indicating a file is missing from the first storage, verifying a presence of the missing file in the second storage; and
if an error is reported indicating a mismatch between the stored associative cryptographic hash and the re-computed hash for the given file in the first storage, verifying an authenticity of the given file in the second storage.
6. The computer-implemented method of claim 1 , wherein the server reviews a retention policy to determine whether each file is to be encrypted.
7. The computer-implemented method of claim 6 , further comprising storing each file to be encrypted as an asset on a removable storage device.
8. The computer-implemented method of claim 7 , wherein the asset is redundantly stored on at least two removable storage devices.
9. The computer-implemented method of claim 7 , further comprising calculating a unique file identifier from the contents of the asset, wherein the unique file identifier is used to identify the asset.
10. The computer-implemented method of claim 7 , further comprising associating the consecutive serial numbers and unique file identifiers of each asset to create an audit trail.
11. The computer-implemented method of claim 10 , further comprising examining the manifest to ensure that each consecutive serial number is accounted for and to ensure that each file in the manifest matches its unique file identifier.
12. The computer-implemented method of claim 11 , further comprising listing the manifest as an asset in a next manifest, along with a unique file identifier including a hash of data in the manifest.
13. A computer-implemented method of auditing stored data in an auditable data storage system, comprising:
recording, in a manifest, data associated with files received from a client computer system for storage in the auditable data storage system, each of the files received for storage having associated therewith a unique file identifier;
assigning consecutive serial numbers to each of the files received for storage;
closing the manifest;
storing the manifest as a file in the auditable data storage system, wherein removal of a serialized entry in the manifest is noted by the gap in the consecutive serial numbers;
creating a new manifest associated with files received for storage in the auditable data storage system, the new manifest recording, as a file entry, data including a unique file identifier associated with the closed manifest;
examining the new manifest to verify the existence of the associated unique file identifier for each file recorded in the new manifest; and
determining that each file having associated data stored in the new manifest is present in the auditable storage system, wherein each consecutive serial number is accounted for and each file matches its associated unique file identifier.
14. The computer-implemented method of claim 13 , further comprising recording, in the manifest, the unique file identifier, a file size and the consecutive serial number assigned to each of the files received for storage.
15. The computer-implemented method of claim 14 , further comprising storing retention information in an asset record associated with each of the files received for storage, wherein the retention information relates to a retention period based on a customer profile.
16. The computer-implemented method of claim 14 wherein the unique file identifier is determined in accordance with at least two cryptographic hashes of the associated file received for storage.
17. The computer-implemented method of claim 13 wherein at least one of the files received for storage is encrypted, and further comprising recording, in the manifest, an encryption key container serial number associated with an encryption key used to encrypt the at least one file.
18. The computer-implemented method of claim 13 , wherein two of the files received for storage have identical content and are associated with a same unique file identifier.
19. The computer-implemented method of claim 18 , wherein the two files associated with the same unique file identifier have a different creator, creation date or file name.