FLATTENED DOCUMENT DATABASE WITH COMPRESSION AND CONCURRENCY
A system for a flattened document database with compression and concurrency, comprising a fragmentation module that divides a structured data file into fragments, computes hash values for the fragments, and produces an index file from the resulting hash values. The fragments are produced according to a key-value pair structure, wherein each key corresponds to a unique fragment of data in the structured data file, each value corresponds to the data within the fragment; and the index file identifies each fragment by a corresponding hash value, and each hash value corresponds to a fragment of the structured data file.
1 . A system for a flattened document database with compression and concurrency, comprising:
a data normalization module comprising at least a plurality of programming instructions stored in a memory and operating on a processor of a network-connected computing device and configured to divide at least an unstructured data file into a plurality of fragments, wherein the fragments are produced according to a key-value pair structure, wherein each key corresponds to a unique fragment of data in the unstructured data file and wherein each value corresponds to at least a portion of the data within the unique fragment;
a data hashing module comprising at least a plurality of programming instructions stored in a memory and operating on a processor of a network-connected computing device and configured to compute a plurality of hash values for each of at least a portion of the plurality of fragments wherein each hash value uniquely corresponds to a particular fragment of the unstructured data file; and
a data indexing module comprising at least a plurality of programming instructions stored in a memory and operating on a processor of a network-connected computing device, and configured to produce an index file from at least a portion of the resulting hash values, wherein the index file uniquely identifies each fragment by each corresponding hash value.
2 . A method for a flattened document database with compression and concurrency, comprising the steps of:
receiving at least an unstructured data file;
dividing, using a data normalization module comprising at least a plurality of programming instructions stored in a memory and operating on a processor of a network-connected computing device, and configured to divide at least an unstructured data file into a plurality of structured key-value based fragments wherein the key is derived from a programmatically selected unique keyword within each fragment;
computing a unique hash value for each of at least a portion of the plurality of fragments using a data hashing module comprising at least a plurality of programming instructions stored in a memory and operating on a processor of a network-connected computing device; and
producing an index file from at least a portion of the resulting unique hash values, using a data indexing module comprising at least a plurality of programming instructions stored in a memory and operating on a processor of a network-connected computing device to allow the directed retrieval of each key-value based fragment.