Data collection cataloguing and searching method and system
A method of cataloguing a data structure employs the formation of a catalogue data structure which is used to associate data items transformed from data elements present within the data collection with storage addresses of the original transformed data elements. This catalogue data structure may be sorted to facilitate searching through same to detect the presence of search patterns. Such searches may be completed through the formation of a plurality of search queries from a received search pattern sequence where the results of running these search queries may then be subsequently considered in conjunction with a search pattern sequence detection process.
1. A method of cataloguing a data collection composed of a plurality of data symbols, said data symbols having a defined order with respect to one another within the data collection, said method of cataloguing comprising:
(i) forming a first data element from an initial sequencing of symbols present within the data collection, said first data element being stored at a storage location within the data collection having a storage address,
(ii) transforming the first data element into a first data item, said first data item being capable of being ranked with respect to other data items,
(iii) storing the first data item using an ordered catalogue data structure which defines a plurality of sequentially arranged storage positions,
(iv) associating with the first data item the storage address from which the first data element was retrieved,
(v) consecutively repeating steps (i) through (iv) for each adjacent data element from the data collection, where each subsequent adjacent data element is formed from the same number of symbols as that used to form the first data element, and
(vi) sorting the ordered catalogue data structure by ranking the data items stored within said catalogue data structure.
2. The method of cataloguing a data collection as claimed in claim 1 , wherein the size of the catalogue data structure is equivalent to the size of the data collection.
3. The method of cataloguing a data collection as claimed in claim 1 , wherein the catalogue data structure is formed from an array.
4. The method of cataloguing a data collection as claimed in claim 1 , wherein data items are associated with storage addresses through the provision of a parallel address data structure.
5. The method of cataloguing a data collection as claimed in claim 4 wherein the parallel address data structure is formed from an array.
6. The method of cataloguing a data collection as claimed in claim 1 wherein data items are associated with storage addresses through the provision of a duplicate based data structure.
7. The method of cataloguing a data collection as claimed in claim 1 wherein the transform process applied to data elements provides numeric format data elements.
8. The method of cataloguing a data collection as claimed in claim 1 wherein the transform process applied to data elements is implemented by a direct hash function.
9. The method of cataloguing a data collection as claimed in claim 1 wherein the transform process applied to data elements is implemented by a lossy hash function.
10. The method of cataloguing a data collection as claimed in claim 9 wherein a retrieval validation process is executed within a search process which uses the catalogue data structure.
11. The method of cataloguing a data collection as claimed in claim 1 wherein a most significant symbol of the data items is not stored within the sorted catalogue data structure.
12. The method of cataloguing a data collection as claimed in claim 1 wherein a class data structure is provided.
13. The method of cataloguing a data collection as claimed in claim 12 wherein a number of unique most significant signals are present within the data items and the class data structure is provided by an array which defines a number of storage positions equal to the number of unique most significant symbols present within data items.
14. The method of cataloguing a data collection as claimed in claim 1 , further comprising;
(vii) forming at least one ordered content data structure which defines a number of storage positions equal to a maximum number of unique data elements which can exist,
(viii) associating with each storage position at least one data element, and
(ix) storing within each storage position of the content data structure a positive or negative indicator depending on whether said at least one data element associated with the storage position is present within the data collection.
15. The method of cataloguing a data collection as claimed in claim 14 wherein the content data structure is formed from an array.
16. The method of cataloguing a data collection as claimed in claim 1 wherein the catalogue data structure is used to find specific data element symbol sequences by running a plurality of search queries generated from a data element symbol sequence.
17. The method of cataloguing a data collection as claimed in claim 16 wherein the plurality of search queries run are generated using a sliding window process.
18. The method of cataloguing a data collection as claimed in claim 17 wherein the results generated by the plurality of search queries run are processed by a search pattern sequence detection process.
19. The method of cataloguing a data collection as claimed in claim 18 wherein the search pattern sequence detection process detects the presence of a pattern on inspection of the spatial relationship between storage addresses associated with data elements found by said search queries.
20. The method of cataloguing a data collection as claimed in claim 19 wherein an error tolerance threshold is used to test whether matching pattern sequences have been detected.
21. The method of claim 1 further comprising:
receiving a search pattern sequence,
forming a plurality of search queries by running a sliding window process over the received search pattern sequence,
retrieving a plurality of storage addresses from the catalogue data structure, said storage addresses being associated with data elements which match the search queries and having a spatial relationship, and
detecting a presence of the search pattern sequence received by inspecting the spatial relationship between the storage addresses received.
22. The method as claimed in claim 21 wherein an error tolerant threshold is used to test whether matching pattern sequences have been detected.
23. The method of cataloguing a data collection as claimed in claim 13 wherein data items are associated with storage addresses throughout the storage of storage addresses within the content data structure.
24. Computer executable instructions stored on a computer readable storage medium, said computer executable instructions being adapted to execute the method of cataloguing a data collection composed of a plurality of data symbols, said data symbols having a defined order with respect to one another within the data collection, said method of cataloguing comprising:
(i) forming a first data element from an initial sequencing of symbols present within the data collection, said first data element being stored at a storage location within the data collection having a storage address, and
(ii) transforming the first data element into a first data item, said first data item being capable of being ranked with respect to other data items, and
(iii) storing the first data item using an ordered catalogue data structure which defines a plurality of sequentially arranged storage positions, and
(iv) associating with the first data item the storage address from which the first data element was retrieved, and
(v) consecutively repeating steps (i) through (iv) for each adjacent data element from the data collection, where each subsequent adjacent data element is formed from the same number of symbols as that used to form the first data element, and
(vi) sorting the ordered catalogue data structure by ranking the data items stored within said catalogue data structure.
25. A computer readable storage medium with computer executable instructions stored therein, said computer executable instructions being adapted to execute the method of cataloguing a data collection composed of a plurality of data symbols, said data symbols having a defined order with respect to one another within the data collection, said method of cataloguing comprising:
(i) forming a first data element from an initial sequencing of symbols present within the data collection, said first data element being stored at a storage location within the data collection having a storage address, and
(ii) transforming the first data element into a first data item, said first data item being capable of being ranked with respect to other data items, and
(iii) storing the first data item using an ordered catalogue data structure which defines a plurality of sequentially arranged storage positions, and
(iv) associating with the first data item the storage address from which the first data element was retrieved, and
(v) consecutively repeating steps (i) through (iv) for each adjacent data element from the data collection, where each subsequent adjacent data element is formed from the same number of symbols as that used to form the first data element, and
(vi) sorting the ordered catalogue data structure by ranking the data items stored within said catalogue data structure.