Data archiving system
An encrypted file storage solution consists of a cluster of processing nodes, external data storage, and a software agent (the “File System Watcher”), which is installed on the application servers. Cluster sizes of one node up to many hundreds of nodes are possible. There are also remote “Key Servers” which provide various services to one or more clusters. The preceding describes a preferred embodiment, though in some cases it may be desirable to “collapse” some of the functionality into a smaller number of hardware devices, typically trading off cost versus security and fault-tolerance.
1. A computer-implemented method of storing data files, comprising:
receiving data files for storage;
serializing the files by assigning an associative serial number to each file;
creating an associative cryptographic hash of each file;
generating a first list of the serialized associative cryptographic hashes of the received files;
closing the first list after a predetermined time interval;
determining an associative cryptographic hash of the closed first list;
storing the closed list in a first storage;
iteratively generating, closing, and storing subsequent lists of serialized associative cryptographic hashes of received files, wherein the first element of each subsequent list is the associative cryptographic hash of the last closed list;
re-computing the associative cryptographic hash of a closed list;
verifying the authenticity of a closed list by comparing the recorded associative cryptographic hash of the closed list in a subsequent list to the re-computed cryptographic hash of the closed list; and
reporting an error when the comparison indicates a mismatch.
2. The computer-implemented method of claim 1 , wherein the lists are closed after a predetermined number of associative cryptographic hashes have been recorded.
3. The computer-implemented method of claim 1 , further comprising: periodically storing the closed lists in a second storage for verification.
4. The computer-implemented method of claim 3 , further comprising:
re-computing the associative cryptographic hash of a list stored in the second storage;
verifying the authenticity of the list in the second storage, by comparing the recorded associative cryptographic hash of the list in a subsequent list to the re-computed associative cryptographic hash of the list; and
reporting an error when the comparison indicates a mismatch.
5. The computer-implemented method of claim 1 , further comprising:
encrypting at least one of the files received for storage with an encryption key,
recording a record of the serial number of the encryption key used and the associated cryptographic hash of the at least one file in a key manager.
6. The computer-implemented method of claim 1 further comprising: generating unique encryption keys in a key server; obtaining the unique encryption key from the key server for encrypting the at least one of the files received for storage; and recording the record of the serial number of the encryption key used and the associated cryptographic hash of the at least one file in the key server.
7. The computer-implemented method of claim 1 further comprising: assigning a minimum retention date and expiry date for each file received for storage; generating a list of stored files ready for deletion according to the minimum retention date and expiry date; and deleting the stored files and their associative parts in the list upon approval.