Method and system for facilitating access to recorded data
The present invention relates to a method and system for facilitating access to recorded data. The system comprises an interface and a processing device. The interface is arranged to receive data and the processing device is arranged to separate the received data in data subsets, compress each data subset and assign an identifier to each compressed data subset, thereby creating data units each comprising a compressed data subset and an associated identifier, the processing device further being arranged to establish an index on the basis of the assigned identifiers.
1. A system configured to access recorded data, the system comprising:
a processor;
a communication interface; and
a memory configured to store computer readable instructions that, when executed, cause the system to:
receive data via the communication interface;
separate the received data into data subsets;
compress each data subset, by applying a compression technique, such that each data subset includes compressed payload data;
create data units from each compressed data subset;
establish an index associated with the data units, wherein the index includes a plurality of index entries, and each index entry comprises:
an identifier;
timing parameters; and
memory address parameters of memory locations containing the data units, wherein the memory address parameters include at least a first memory address and a second memory address;
search the memory, for the data units, using the established index;
decompress the data units within a timing window using the timing parameters; and
generate a view that includes the decompressed data units within the timing window.
2. The system of claim 1 , wherein
the first memory address is a starting memory address,
the second memory address is an ending memory address, and
the timing parameters include a start time parameter and an end time parameter.
3. The system of claim 1 , wherein the view is generated in near real time.
4. The system of claim 1 , wherein the index indicates a physical memory location, in the memory, of each data unit.
5. The system of claim 1 , wherein
the memory includes a first memory and a second memory, and
the system is further caused to search in both the first memory, for real-time analysis, and the second memory, for historical analysis, using the established index.
6. The system of claim 1 , wherein the compression technique utilizes a software library associated with any of zlib, bzip2, or snappy.
7. The system of claim 1 , wherein the data subsets are configurable in size such that larger subsets translate to faster scan and smaller index while smaller subsets translate to slower scan and larger index.
8. A non-transitory computer readable storage medium configured to store computer readable instructions that, when executed by a processor of an information processing apparatus, cause the information processing apparatus to provide execution comprising:
receiving data and separating the received data into data subsets;
compressing each data subset, by applying a compression technique, such that each data subset includes compressed payload data;
creating data units from each compressed data subset;
establishing an index associated with the data units, wherein the index includes a plurality of index entries, and each index entry comprises:
an identifier;
timing parameters; and
memory address parameters of memory locations containing the data units;
searching the memory, for the data units, using the established index;
decompressing the data units within a timing window using the timing parameters; and
generating a view that includes the decompressed data units within the timing window.
9. The non-transitory computer readable storage medium of claim 8 , wherein the memory address parameters include at least a first memory address and a second memory address.
10. The non-transitory computer readable storage medium of claim 8 , wherein
the first memory address is a starting memory address,
the second memory address is an ending memory address, and
the timing parameters include a start time parameter and an end time parameter.
11. The non-transitory computer readable storage medium of claim 8 , wherein
the memory includes a first memory and a second memory, and
the information processing apparatus is further caused to provide execution comprising searching in both the first memory, for real-time analysis, and the second memory, for historical analysis, using the established index.
12. A method for accessing recorded data, the method comprising:
at an information processing system having a processor and a memory:
receiving data and separating the received data into data subsets;
compressing each data subset, by applying a compression technique, such that each data subset includes compressed payload data;
creating data units from each compressed data subset;
establishing an index associated with the data units, wherein the index includes a plurality of index entries, and each index entry comprises:
an identifier;
timing parameters; and
memory address parameters of memory locations containing the data units;
searching the memory, for the data units, using the established index; and
decompressing the data units within a timing window using the timing parameters.
13. The method of claim 12 , further comprising:
generating a view that includes the decompressed data units within the timing window.
14. The method of claim 12 , wherein the memory address parameters include at least a first memory address and a second memory address.
15. The method of claim 14 , wherein
the first memory address is a starting memory address,
the second memory address is an ending memory address, and
the timing parameters include a start time parameter and an end time parameter.
16. The method of claim 12 , wherein
the memory includes a first memory and a second memory, and
the method further comprises searching in both the first memory, for real-time analysis, and the second memory, for historical analysis, using the established index.
17. The method of claim 16 , wherein
the first memory is configured to store the data units, and
the stored data units are transferred from the first memory to the second memory in association with a memory utilization degree reaching a threshold value.
18. The non-transitory computer readable storage medium of claim 11 , wherein
the first memory is configured to store the data units, and
the stored data units are transferred from the first memory to the second memory when a memory utilization degree reaches a threshold value.
19. The system of claim 5 , wherein
the first memory is configured to store the data units, and
the stored data units are transferred from the first memory to the second memory when a memory utilization degree reaches a threshold value.
20. The system of claim 5 , wherein
the first memory is configured to store the data units, and
the stored data units are transferred from the first memory to the second memory in association with a memory flush instruction.