Systems and methods for data compression
View Patent ↗Event data comprising an unordered string set may be received. String set dictionary indexes may be assigned for strings of the unordered string set in a string set dictionary. The unordered string set may be sorted to provide a sorted series based on the string set dictionary indexes for the unordered string set. A differential series may be computed from the sorted series. The differential series may be encoded into binary code words. In an embodiment, the event data also may comprise strings. A schema version associated with the strings in a row may be determined. Computing resources may be allocated based on the schema version.
1. A system comprising:
at least one processor; and
a memory storing instructions configured to instruct the at least one processor to perform:
receiving event data comprising an unordered string set;
assigning string set dictionary indexes for strings of the unordered string set in a string set dictionary;
sorting the unordered string set to provide a sorted series based on the string set dictionary indexes for the unordered string set;
computing a differential series from the sorted series; and
encoding the differential series into binary code words.
2. The system of claim 1 , wherein the instructions are configured to instruct the at least one processor to perform storing the binary code words to a leaf node of an in-memory data storage module.
3. The system of claim 2 , wherein the instructions are configured to instruct the at least one processor to perform storing the string set dictionary to the leaf node.
4. The system of claim 2 , wherein the instructions are configured to instruct the at least one processor to perform decompressing the binary code words.
5. The system of claim 1 , wherein the sorted series comprises an ascending series.
6. The system of claim 1 , wherein the binary code words comprise Fibonacci code words representing the differential series.
7. The system of claim 1 , wherein the unordered string set is associated with user experiment group information or error report information for a social networking system.
8. The system of claim 1 , wherein the unordered string set is part of a row being ingested into a portion of an in-memory data storage module.
9. The system of claim 8 , wherein the portion comprises a leaf node or an aggregator of the in-memory data storage module.
10. The system of claim 1 , wherein the event data comprises strings, and the instructions are configured to instruct the at least one processor to perform:
determining a schema version for a row index associated with the strings; and
allocating resources based on the schema version.
11. The system of claim 10 , wherein the strings are organized by columns.
12. The system of claim 11 , wherein a string dictionary is associated with each column.
13. The system of claim 12 , wherein a string dictionary index is assigned to a unique string in a column.
14. The system of claim 13 , wherein the instructions are configured to instruct the at least one processor to perform writing the string dictionary to a leaf node of an in-memory data storage module.
15. The system of claim 10 , wherein the determining the schema version comprises:
creating the row index based on string dictionary indexes for the strings in a row; and
assigning the schema version based on the row index.
16. The system of claim 15 , wherein the creating the row index comprises:
creating a string dictionary index for each unique string; and
assigning the row index based on the string dictionary index for each unique string.
17. The system of claim 1 , wherein the event data corresponds to one or more of a performance event on a computing system, user interaction with the computing system, or a behavioral event on the computing system.
18. The system of claim 1 , wherein the system is incorporated into a social networking system.
19. A computer implemented method comprising:
receiving, by a computer system, event data containing an unordered string set;
assigning, by the computer system, string set dictionary indexes for strings of the unordered string set in a string set dictionary;
sorting, by the computer system, the unordered string set to provide a sorted series based on the string set dictionary indexes for the unordered string set;
computing, by the computer system, a differential series from the sorted series; and
encoding, by the computer system, the differential series into binary code words.
20. A non-transitory computer storage medium storing computer-executable instructions that, when executed by a processor, cause a computer system to perform a computer-implemented method comprising:
receiving event data comprising an unordered string set;
assigning string set dictionary indexes for strings of the unordered string set in a string set dictionary;
sorting the unordered string set to provide a sorted series based on the string set dictionary indexes for the unordered string set;
computing a differential series from the sorted series; and
encoding the differential series into binary code words.