Searchable archive
View Patent ↗A searchable archiving system. A searchable archiving system includes one or more compacted files of archive data loosely coupled to a search process. To create a compacted file, an archiving process tokenizes the archive data, optimizes the tokenized archive data, and extracts archive metadata from the tokenized data. The tokenized data may then be compressed in a variety of ways into compressed segments that may be individually accessed and decompressed by the search agents. Before compression, segment metadata is extracted from the segments. The compressed segments and segment metadata are then combined to create a compacted file. The search process accesses the compacted files by consulting locally stored archive metadata extracted from the files during the compaction process. The search process then invokes one or more search agents that actively search the compacted files. The search agents do so by using the segment metadata to identify segments to decompress and search.
1. A method of creating a searchable archive accessible by a data processing system, comprising:
generating a domain structure and tokenized data from an archive data set, the domain structure including tokens corresponding to unique values in the archive data set and the tokenized data including token columns corresponding to value columns in the archive data set;
determining archive metadata from the domain structure and the tokenized data;
dividing the tokenized data into one or more token column segments;
determining token column segment metadata from the one or more token column segments;
creating one or more compressed token column segments from the token column segments;
creating one or more compacted files from the one or more compressed token column segments and the token column segment metadata; and
storing the one or more compacted files in a file system coupled to the data processing system.
2. The method of claim 1 , wherein determining metadata further comprises determining a maximum value and a minimum value for each of the token columns.
3. The method of claim 1 , wherein determining metadata further comprises determining a maximum tupleid and a minimum tupled for each of the one or more token column segments.
4. The method of claim 1 , further comprising:
dividing the domain structure into one or more domain structure segments;
determining metadata from the domain structure segments;
compressing the one or more domain structure segments; and
creating one or more compacted files further includes storing the compressed domain structure segments in the compacted file.
5. A method of creating a searchable archive accessible by a data processing system, comprising:
generating a domain structure and tokenized data from archive data;
determining metadata from the tokenized data;
generating a set of bit vectors from the tokenized data;
creating one or more compacted files from the set of bit vectors; and
storing the one or more compacted files in a file system coupled to the data processing system.
6. The method of claim 5 , wherein the tokenized data set includes one or more columns of tokens and extracting archive metadata further comprises determining a maximum token value and a minimum token value for each of the one or more columns of tokens.
7. A data processing system for creating a searchable archive, comprising:
a processor; and
a memory coupled to the processor, the memory having program instructions executable by the processor stored therein, the program instructions including:
generating a domain structure and tokenized data from an archive data set, the domain structure including tokens corresponding to unique values in the archive data set and the tokenized data including token columns corresponding to value columns in the archive data set;
determining archive metadata from the domain structure and the tokenized data;
dividing the tokenized data into one or more token column segments;
determining token column segment metadata from the one or more token column segments;
creating one or more compressed token column segments from the token column segments;
creating one or more compacted files from the one or more compressed token column segments and the token column segment metadata; and
storing the one or more compacted files in a file system coupled to the data processing system.
8. The data processing system of claim 7 , the program instructions for determining metadata further including determining a maximum value and a minimum value for each of the token columns.
9. The data processing system of claim 7 , the program instructions for determining metadata further including determining a maximum tupleid and a minimum tupled for each of the one or more token column segments.
10. The data processing system of claim 7 , the program instructions further including:
dividing the domain structure into one or more domain structure segments; determining metadata from the domain structure segments;
compressing the one or more domain structure segments; and
creating one or more compacted files further includes storing the compressed domain structure segments in the compacted file.
11. A data processing system for creating a searchable archive, comprising:
a processor; and
a memory coupled to the processor, the memory having program instructions executable by the processor stored therein, the program instructions including:
generating a domain structure and tokenized data from archive data;
determining metadata from the tokenized data;
generating a set of bit vectors from the tokenized data;
creating one or more compacted files from the set of bit vectors; and
storing the one or more compacted files in a file system coupled to the data processing system.
12. The data processing system of claim 11 , wherein the tokenized data set includes one or more columns of tokens, the program instructions for extracting archive metadata further including determining a maximum token value and a minimum token value for each of the one or more columns of tokens.
13. A method of utilizing a searchable archive by a data processing system, comprising:
generating a domain structure and tokenized data from archive data;
determining archive metadata from the tokenized data;
dividing the tokenized data into one or more segments;
determining segment metadata from the one or more segments;
creating one or more compressed segments from the segments;
creating one or more compacted files from the one or more compressed segments and the segment metadata; and
storing the one or more compacted files in a file system coupled to the data processing system.
14. The method of claim 13 , further comprising:
selecting a selected compacted file from the one or more compacted files that may include a datum using the archive metadata;
accessing the selected compacted file;
selecting a selected compressed segment from the one or more compressed segments in the selected compacted file using the segment metadata;
generating a decompressed segment from the selected compressed segment; and
searching the decompressed segment to determine if the decompressed segment includes the datum.
15. The method of claim 14 wherein:
selecting a selected compacted file is performed by a search process; and
accessing the selected compacted file, selecting a selected compressed segment, generating a decompressed segment, and searching the decompressed segment are performed by one or more search agents invoked by the search process.
16. A method of utilizing a searchable archive by a data processing system, comprising:
generating a domain structure and tokenized data from archive data;
determining archive metadata from the tokenized data;
generating a set of bit vectors from the tokenized data;
creating one or more compacted files from the set of bit vectors; and
storing the one or more compacted files in a file system coupled to the data processing system.
17. The method of claim 16 , further comprising:
selecting a selected compacted file from the one or more compacted files that may include a datum using the archive metadata;
accessing the selected compacted file;
selecting one or more bit vectors from the selected compacted file; and
performing a Boolean operation on the bit vectors included in the to determine if the datum is stored in the compacted file.
18. The method of claim 17 , wherein:
selecting a selected compacted file is performed by a search process; and
accessing the selected compacted file and
performing a Boolean operation is performed by one or more search agents invoked by the search process.
19. A data processing system for utilizing a searchable archive, comprising:
a processor; and
a memory coupled to the processor, the memory having program instructions executable by the processor stored therein, the program instructions including:
generating a domain structure and tokenized data from archive data;
determining archive metadata from the tokenized data;
dividing the tokenized data into one or more segments;
determining segment metadata from the one or more segments;
creating one or more compressed segments from the segments;
creating one or more compacted files from the one or more compressed segments and the segment metadata; and
storing the one or more compacted files in a file system coupled to the data processing system.
20. The data processing system of claim 19 , the program instructions further including:
selecting a selected compacted file from the one or more compacted files that may include a datum using the archive metadata;
accessing the selected compacted file;
selecting a selected compressed segment from the one or more compressed segments in the selected compacted file using the segment metadata;
generating a decompressed segment from the selected compressed segment; and
searching the decompressed segment to determine if the decompressed segment includes the datum.
21. The data processing system of claim 20 , wherein
selecting a selected compacted file is performed by a search process; and
accessing the selected compacted file, selecting a selected compressed segment, generating a decompressed segment, and searching the decompressed segment are performed by one or more search agents invoked by the search process.
22. A data processing system for utilizing a searchable archive, comprising: a processor; and
a memory coupled to the processor, the memory having program instructions
executable by the processor stored therein, the program instructions including:
generating a domain structure and tokenized data from archive data;
determining archive metadata from the tokenized data;
generating a set of bit vectors from the tokenized data;
creating one or more compacted files from the set of bit vectors; and
storing the one or more compacted files in a file system coupled to the data processing system.
23. The data processing system of claim 22 , the program instructions further including:
selecting a selected compacted file from the one or more compacted files that may include a datum using the archive metadata;
accessing the selected compacted file;
selecting one or more bit vectors from the selected compacted file; and
performing a Boolean operation on the bit vectors included in the to determine if the datum is stored in the compacted file.
24. The data processing system of claim 23 , wherein:
selecting a selected compacted file is performed by a search process; and
accessing the selected compacted file and
performing a Boolean operation is performed by one or more search agents invoked by the search process.