Single instance storage of metadata and extracted text
Embodiments of the present disclosure are directed to a method for processing computer readable electronic files in an investigation in a computer system including a processor coupled to a display and an electronic storage device coupled to the processor. The method includes the processor accessing the electronic files and related data from a data source. The accessed files and related data are culled by the processor based on predetermined filter criteria. The processor stores the remaining files and related data in a third-party data repository and maps a set of electronic files and related data stored in the third-party data repository into a predetermined database schema. The mapped files and related data are analyzed by the processor, which applies a status decision on them. The analyzed electronic files and related data are submitted to a third-party e-discovery processing application based on the applied status decision.
1. A method of storing a collection of electronic files in at least one storage device of a computer system also including at least one processor, the method comprising:
for a new electronic file received to be added to the collection:
determining, by the at least one processor, a hash value of the new electronic file, wherein the new electronic file is assigned a unique inventory identification number different from the determined hash value and the new electronic file is listed as an entry in a first table stored in the at least one storage device based on the assigned unique inventory identification number, and
wherein the entry in the first table includes a link to an entry in a second table in the at least one storage device which includes metadata for the new electronic file;
determining, by the at least one processor, whether the hash value of the new electronic file matches a hash value of an electronic file already in the collection;
when the hash value of the new electronic file does not match the hash value of the electronic file already in the collection:
adding, by the at least one processor, the new electronic file and the hash value of the new electronic file to the collection,
adding, by the at least one processor, content of the new electronic file as an entry to a third table in the at least one storage device,
assigning, by the at least one processor, a new metadata identifier, associated with the hash value of the new electronic file, to the entry corresponding to the new electronic file in the second table, wherein the new metadata identifier is different from the assigned unique inventory identification number and the determined hash value, and
adding, by the at least one processor, the assigned unique inventory identification number and the new metadata identifier to the entry of the new electronic file in the first table, wherein the new metadata identifier is a link to the entry corresponding to the new electronic file in the third table;
when the hash value of the new electronic file matches the hash value of the electronic file already in the collection:
adding, by the at least one processor, the assigned unique inventory identification number and a metadata identifier, present in the second table, associated with the hash value of the electronic file already in the collection to the entry of the new electronic file in the first table without storing the new electronic file in the collection, wherein the metadata identifier of the electronic file in the first table is a link to an entry in the third table which includes content of the electronic file, and wherein a plurality of the assigned unique inventory identification numbers correspond to the same metadata identifier present in the second table; and
generating, by the at least one processor, a report based on at least one of the first table, the second table, and the third table.
2. The method of claim 1 , wherein the hash value of the new electronic file is stored in the second table in the at least one storage device.
3. The method of claim 1 , wherein when the hash value of the new electronic file matches the hash value of the electronic file already in the collection:
determining, by the at least one processor, the metadata identifier of the electronic file already in the collection with the hash value; and
referring, by the at least one processor, the contents of the new electronic file using the determined metadata identifier of the electronic file already in the collection.
4. The method of claim 1 , further comprising parsing, by the at least one processor, the new electronic file and mapping the corresponding metadata and the corresponding content, from the parsed new electronic file, to a predetermined schema.
5. The method of claim 1 , further comprising labeling the new electronic file, by the at least one processor, for determining relevancy of the new electronic file based on the determined hash value of the new electronic file.
6. The method of claim 1 , wherein the generated report at least indicates the electronic files not stored in the collection.
7. A system to store a collection of electronic files, the system comprising:
at least one processor; and
at least one electronic storage device coupled to the at least one processor, wherein:
the at least one processor determines a hash value of a new electronic file received to be added to the collection, the new electronic file being assigned a unique inventory identification number different from the determined hash value and the new electronic file being listed as an entry in a first table stored in the at least one electronic storage device based on the assigned unique inventory identification number, and
the entry in the first table includes a link to an entry in a second table in the at least one electronic storage device which includes metadata for the new electronic file;
the at least one processor determines whether the hash value of the new electronic file matches a hash value of an electronic file already in the collection;
when the hash value of the new electronic file does not match the hash value of the electronic file already in the collection:
the at least one processor adds the new electronic file and the hash value of the new electronic file to the collection,
the at least one processor adds content of the new electronic file as an entry to a third table in the at least one storage device,
the at least one processor assigns a new metadata identifier, associated with the hash value of the new electronic file, to the entry corresponding to the new electronic file in the second table, wherein the new metadata identifier is different from the assigned unique inventory identification number and the determined hash value, and
the at least one processor adds the assigned unique inventory identification number and the new metadata identifier to the entry of the new electronic file in the first table, wherein the new metadata identifier is a link to the entry corresponding to the new electronic file in the third table;
when the hash value of the new electronic file matches the hash value of the electronic file already in the collection:
the at least one processor adds the assigned unique inventory identification number and a metadata identifier, present in the second table, associated with the hash value of the electronic file already in the collection to the entry of the new electronic file in the first table without storing the new electronic file in the collection, wherein the metadata identifier of the electronic file in the first table is a link to an entry in the third table which includes content of the electronic file, and wherein a plurality of the assigned unique inventory identification numbers correspond to the same metadata identifier present in the second table; and
the at least one processor generates a report based on at least one of the first table, the second table, and the third table.
8. The system of claim 7 , wherein the at least one processor stores the hash value of the new electronic file in the second table in the at least one electronic storage device.
9. The system of claim 7 , wherein when the hash value of the new electronic file matches the hash value of the electronic file already in the collection:
the at least one processor determines the metadata identifier of the electronic file already in the collection with the hash value; and
the at least one processor refers the contents of the new electronic file using the determined metadata identifier of the electronic file already in the collection.
10. The system of claim 7 , wherein the at least one processor further parses the new electronic file and maps the corresponding metadata and the corresponding content, from the parsed new electronic file, to a predetermined schema.
11. The system of claim 7 , wherein the at least one processor further labels the new electronic file, based on the determined hash value of the new electronic file, for determining relevancy of the new electronic file.
12. The system of claim 7 , wherein the generated report at least indicates the electronic files not stored in the collection.
13. A non-transitory computer-readable medium including instructions for storing a collection of electronic files in at least one storage device of a computer system, the non-transitory computer-readable medium including instructions executable by a processing resource to:
determine a hash value of a new electronic file, wherein the new electronic file is assigned a unique inventory identification number different from the determined hash value and the new electronic file is listed as an entry in a first table stored in the at least one storage device based on the assigned unique inventory identification number,
wherein an entry in the first table includes a link to an entry in a second table in the at least one storage device which includes metadata for the new electronic file;
determine whether the hash value of the new electronic file matches a hash value of an electronic file already in the collection;
when the hash value of the new electronic file does not match the hash value of the electronic file already in the collection:
add the new electronic file and the hash value of the new electronic file to the collection,
add content of the new electronic file as an entry to a third table in the at least one storage device,
assign a new metadata identifier, associated with the hash value of the new electronic file, to the entry corresponding to the new electronic file in the second table, wherein the new metadata identifier is different from the assigned unique inventory identification number and the determined hash value, and
add the assigned unique inventory identification number and the new metadata identifier to the entry of the new electronic file in the first table, wherein the metadata identifier is a link to the entry of the new electronic file in the third table; and when the hash value of the new electronic file matches the hash value of the electronic file already in the collection: add the assigned unique inventory identification number and a metadata identifier, present in the second table, corresponding to the hash value of the electronic file already in the collection to the entry of the new electronic file in the first table without storing the new electronic file in the collection, wherein the metadata identifier associated with the hash value of the electronic file in the first table is a link to an entry in the third table which includes content of the electronic file, and wherein a plurality of the assigned unique inventory identification numbers correspond to the same metadata identifier present in the second table; and
generate a report based on at least one of the first table, the second table, and the third table.
14. The non-transitory computer-readable medium of claim 13 , wherein the non-transitory computer-readable medium further includes instructions executable by the processing resource to store the hash value of the new electronic file in the second table in the at least one storage device.
15. The non-transitory computer-readable medium of claim 13 , wherein when the hash value of the new electronic file matches the hash value of the electronic file already in the collection, the non-transitory computer-readable medium further includes instructions executable by the processing resource to:
determine the metadata identifier of the electronic file already in the collection with the hash value; and
refer the contents of the new electronic file using the determined metadata identifier of the electronic file already in the collection.
16. The non-transitory computer-readable medium of claim 13 , wherein the non-transitory computer-readable medium further includes instructions executable by the processing resource to parse the new electronic file and map the corresponding metadata and the corresponding content, from the parsed new electronic file, to a predetermined schema.
17. The non-transitory computer-readable medium of claim 13 , wherein the non-transitory computer-readable medium further includes instructions executable by the processing resource to label the new electronic file, based on the determined hash value of the new electronic file, for determining relevancy of the new electronic file.
18. The non-transitory computer-readable medium of claim 13 , wherein the generated report at least indicates the electronic files not stored in the collection.