Index compression and decompression
A machine can compress an entry of an index, starting with accessing first data that corresponds to a segment of content and is expressed in a first number of bits. The machine classifies the content into a first class or a second class and generates second data that is expressed in a fewer, second number of bits. To generate an upper portion of the second data, the machine selects and executes one of two algorithms based on the class of the content. The machine then assigns the generated second data to the segment. To decompress the second data and thereby recover the first data, the machine parses upper and lower portions of the second data, determines the class of the content, and recovers upper and lower portions of the first data. The upper portion is recovered via one of two algorithms based on the class of the content.
1. A method comprising:
accessing, by one or more processors, first data that corresponds to a segment of media content and is expressed in a first number of bits, an upper portion of the first number of bits having a first data type, a lower portion of the first number of bits having a second data type;
classifying, by the one or more processors, the media content into a binary class among a first binary class and a second binary class;
compressing, by the one or more processors, the first data by generating second data expressed in a second number of bits that are fewer than the first number of bits, the generating of the second data including:
if the media content has the first binary class, generating a first upper portion of the second number of bits by subtracting the upper portion of the first number of bits from a maximum value representable by the second number of bits, the first upper portion having a predetermined first size;
otherwise, if the media content has the second binary class, generating a second upper portion of the second number of bits by adding the upper portion of the first number of bits to a minimum value representable by the second number of bits, the second upper portion having a predetermined second size; and
assigning, by the one or more processors, the generated second data expressed in the second number of bits to the segment of the media content, the first or second upper portion of the second number of bits having the first data type, a lower portion of the second number of bits having the second data type and having a predetermined third size.
2. The method of claim 1 , wherein:
the first data identifies the segment of the media content;
the upper portion of the first number of bits encodes a media identifier that identifies the media content; and
the lower portion of the first number of bits encodes a location identifier that identifies a location of the segment within the media content.
3. The method of claim 1 , wherein:
the predetermined first size of the first upper portion, the predetermined second size of the second upper portion, and the predetermined third size of the lower portion are equal;
the upper portion of the first number of bits is an upper half of the first number of bits and expresses a first numerical value that corresponds to the first data type; and
the lower portion of the first number of bits is a lower half of the first number of bits and expresses a second numerical value that corresponds to the second data type.
4. The method of claim 1 , wherein:
the first data is a 64-bit entry within a pre-compression hash table;
the upper portion of the first number of bits is a first 32-bit number that identifies the media content;
the lower portion of the first number of bits is a second 32-bit number that identifies a location of the segment within the media content; and
the assigning of the generated second data to the segment of the media content includes storing the second data in a post-compression hash table.
5. The method of claim 4 , wherein:
the first or second upper portion of the second number of bits identifies the media content;
the lower portion of the second number of bits identifies the segment of the media content; and
the second data is a 32-bit entry within the post-compression hash table.
6. The method of claim 1 , wherein:
the first predetermined size of the first upper portion of the second number of bits is larger than the second predetermined size of the second upper portion of the second number of bits; and
the compressing of the first data further includes:
if the media content has the first binary class, generating a first lower portion of the second number of bits based on the lower portion of the first number of bits;
otherwise, if the media content has the second binary class, generating a second lower portion of the second number of bits based on the lower portion of the first number of bits; wherein
the first lower portion of the second number of bits has more bits than the second lower portion of the second number of bits.
7. The method of claim 1 , wherein:
the classifying of the media content includes comparing a duration of the media content to a threshold duration, the first binary class indicating that the duration of the media content meets or exceeds the threshold duration, the second binary class indicating that the duration of the media content fails to exceed the threshold duration.
8. The method of claim 7 , wherein:
the first predetermined size of the first upper portion of the second number of bits is larger than the second predetermined size of the second upper portion of the second number of bits; and
the compressing of the first data further includes:
if the duration of the media content meets or exceeds the threshold duration, generating the first upper portion of the second number of bits based on the upper portion of the first number of bits;
otherwise, if the duration of the media content fails to exceed the threshold duration, generating the second upper portion of the second number of bits based on the upper portion of the first number of bits; wherein
the first upper portion of the second number of bits has fewer bits than the second upper portion of the second number bits.
9. The method of claim 7 , wherein:
the compressing of the first data further includes:
if the duration of the media content meets or exceeds the threshold duration, generating a first lower portion of the second number of bits based on the lower portion of the first number of bits;
otherwise, if the duration of the media content fails to exceed the threshold duration, generating a second lower portion of the second number of bits based on the lower portion of the first number of bits; wherein
the first lower portion of the second number of bits has more bits than the second lower portion of the second number of bits.
10. The method of claim 7 , further comprising:
determining the threshold duration prior to the classifying of the media content, the determining of the threshold duration including:
generating a histogram of durations of media content stored in a database of media content; and
calculating a predetermined percentile of the durations of media content from the generated histogram, the determined threshold duration representing the predetermined percentile of the durations.
11. The method of claim 10 , wherein:
the predetermined percentile is 95%;
the first binary class indicates that the duration of the media content meets or exceeds at least 95% of the durations represented by the histogram; and
the second binary class indicates that the duration of the media content fails to exceed 95% of the durations represented by the histogram.
12. The method of claim 1 , wherein:
the second data is a 32-bit number within which the first upper portion of the second number of bits is an 18-bit number that identifies the media content and within which the second upper portion of the second number of bits is a 15-bit number that identifies the media content.
13. The method of claim 1 , wherein:
the compressing of the first data further includes:
if the media content has the first binary class, generating a first lower portion of the second number of bits based on the lower portion of the first number of bits;
otherwise, if the media content has the second binary class, generating a second lower portion of the second number of bits based on the lower portion of the first number of bits; and
the second data is a 32-bit number within which the first lower portion of the second number of bits is a 14-bit number that identifies a location of the segment within the media content and within which the second lower portion of the second number of bits is a 17-bit number that identifies the location of the segment within the media content.
14. The method of claim 1 , further comprising:
indexing a sub-fingerprint of the media content by indexing a fingerprint of the segment of the media content, the indexing of the fingerprint of the segment including assigning the fingerprint of the segment to the generated second data within a hash table that correlates the generated second data to the segment of the media content.
15. The method of claim 1 , wherein:
the minimum value representable by the second number of bits is a string of hexadecimal zero (0x0) values; and
the maximum value representable by the second number of bits is a string of hexadecimal fifteen (0xF) values.
16. A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:
accessing first data that corresponds to a segment of media content and is expressed in a first number of bits, an upper portion of the first number of bits having a first data type, a lower portion of the first number of bits having a second data type;
classifying the media content into a binary class among a first binary class and a second binary class;
compressing the first data by generating second data expressed in a second number of bits that are fewer than the first number of bits, the generating of the second data including:
if the media content has the first binary class, generating a first upper portion of the second number of bits by subtracting the upper portion of the first number of bits from a maximum value representable by the second number of bits, the first upper portion having a predetermined first size;
otherwise, if the media content has the second binary class, generating a second upper portion of the second number of bits by adding the upper portion of the first number of bits to a minimum value representable by the second number of bits, the second upper portion having a predetermined second size; and
assigning the generated second data expressed in the second number of bits to the segment of the media content, the first or second upper portion of the second number of bits having the first data type, a lower portion of the second number of bits having the second data type and having a predetermined third size.
17. The non-transitory machine-readable storage medium of claim 16 , wherein:
the first predetermined size of the first upper portion of the second number of bits is larger than the second predetermined size of the second upper portion of the second number of bits; and
the compressing of the first data further includes:
if the media content has the first binary class, generating a first lower portion of the second number of bits based on the lower portion of the first number of bits;
otherwise, if the media content has the second binary class, generating a second lower portion of the second number of bits based on the lower portion of the first number of bits; wherein
the first lower portion of the second number of bits has more bits than the second lower portion of the second number of bits.
18. A system comprising:
one or more processors; and
a memory storing instructions that, when executed by at least one processor among the one or more processors, cause the system to perform operations comprising:
accessing first data that corresponds to a segment of media content and is expressed in a first number of bits, an upper portion of the first number of bits having a first data type, a lower portion of the first number of bits having a second data type;
classifying the media content into a binary class among a first binary class and a second binary class;
compressing the first data by generating second data expressed in a second number of bits that are fewer than the first number of bits, the generating of the second data including:
if the media content has the first binary class, generating a first upper portion of the second number of bits by subtracting the upper portion of the first number of bits from a maximum value representable by the second number of bits, the first upper portion having a predetermined first size;
otherwise, if the media content has the second binary class, generating a second upper portion of the second number of bits by adding the upper portion of the first number of bits to a minimum value representable by the second number of bits, the second upper portion having a predetermined second size; and
assigning the generated second data expressed in the second number of bits to the segment of the media content, the first or second upper portion of the second number of bits having the first data type, a lower portion of the second number of bits having the second data type and having a predetermined third size.
19. The system of claim 18 , wherein:
the classifying of the media content includes comparing a duration of the media content to a threshold duration, the first binary class indicating that the duration of the media content exceeds the threshold duration, the second binary class indicating that the duration of the media content fails to exceed the threshold duration.
20. The system of claim 19 , wherein:
the first predetermined size of the first upper portion of the second number of bits is larger than the second predetermined size of the second upper portion of the second number of bits; and
the compressing of the first data further includes:
if the duration of the media content exceeds the threshold duration, generating the first upper portion of the second number of bits based on the upper portion of the first number of bits;
otherwise, if the duration of the media content fails to exceed the threshold duration, generating the second upper portion of the second number of bits based on the upper portion of the first number of bits; wherein
the first upper portion of the second number of bits has fewer bits than the second upper portion of the second number bits.