Parallel decompression of compressed data streams
In various examples, metadata may be generated corresponding to compressed data streams that are compressed according to serial compression algorithms—such as arithmetic encoding, entropy encoding, etc.—in order to allow for parallel decompression of the compressed data. As a result, modification to the compressed data stream itself may not be required, and bandwidth and storage requirements of the system may be minimally impacted. In addition, by parallelizing the decompression, the system may benefit from faster decompression times while also reducing or entirely removing the adoption cycle for systems using the metadata for parallel decompression.
1 . A method comprising:
determining first metadata associated with segments of compressed data;
determining second metadata associated with locations of segments of a dictionary, the dictionary being associated with the compressed data; and
decompressing, using the dictionary and based at least on the first metadata and the second metadata, two or more of the segments of the compressed data in parallel to generate a decompressed output.
2 . The method of claim 1 , wherein the first metadata is indicative of at least a first input segment location and a first output segment location associated with an individual segment of the segments of the compressed data.
3 . The method of claim 1 , wherein the decompressing the segments of the compressed data to generate the decompressed output comprises:
decompressing, based at least on the second metadata, one or more of the segments of the dictionary to generate the dictionary; and
decompressing, based at least on the dictionary and the first metadata, the segments of the compressed data to generate the decompressed output.
4 . The method of claim 1 , wherein an individual segment of the segments of the dictionary is used to decompress at least an individual segment of the segments of the compressed data.
5 . The method of claim 1 , further comprising generating third metadata indicative of at least one of:
a first index associated with the segments of the compressed data; or
a second index associated with the segments of the dictionary.
6 . The method of claim 1 , further comprising:
receiving input data;
identifying segments of the input data that correspond to segments of the dictionary; and
generating the compressed data by at least compressing the segments of the input data using the dictionary.
7 . A system comprising:
one or more processing units to:
generate first metadata associated with segments of compressed data;
generate second metadata associated with segments of a dictionary, the dictionary being associated with the compressed data;
associate the first metadata with at least the compressed data and the second metadata with at least the dictionary;
transmit the first metadata, the second metadata, and the compressed data to cause a decompression of two or more of the segments of the compressed data in parallel using the dictionary and based on the first metadata and the second metadata.
8 . The system of claim 7 , wherein:
the first metadata is indicative of at least a first input segment location and a first output segment location associated with an individual segment of the segments of the compressed data; and
the second metadata is indicative of at least a second input segment location and a second output segment location associated with an individual segment of the segments of the dictionary.
9 . The system of claim 7 , wherein the one or more processing units are further to decompress, based at least on the first metadata and the second metadata, segments of the segments of the compressed data to generate output data.
10 . The system of claim 9 , wherein a decompression of the segments of the compressed data to generate the output data is achieved by:
decompressing, based at least on the second metadata, one or more segments of the segments of the dictionary to generate the dictionary; and
decompressing, based at least on the dictionary and the first metadata, the segments of the compressed data to generate the output data.
11 . The system of claim 7 , wherein an individual segment of the dictionary is used to decompress at least an individual segment of the compressed data.
12 . The system of claim 7 , wherein the one or more processing units are further to generate third metadata indicative of at least one of:
a first index associated with the segments of the compressed data; or
a second index associated with the segments of the dictionary.
13 . The system of claim 7 , wherein the one or more processing units are further to:
receive input data;
identify segments of the input data that correspond to segments of the dictionary; and
generate the compressed data by at least compressing the segments of the input data using the dictionary.
14 . The system of claim 7 , wherein the system is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing deep learning operations;
a system for performing real-time streaming broadcasts;
a system for performing video monitoring services;
a system for performing intelligent video analysis;
a system implemented using an edge device;
a system for generating ray-traced graphical output;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
15 . A processor comprising:
one or more processing units to generate output data by decompressing at least two segments of compressed data in parallel based at least on first metadata associated with the at least two segments of the compressed data and second metadata associated with segments of a dictionary, the first metadata indicating one or more first locations associated with the at least two segments of the compressed data and the second metadata indicating one or more second locations associated with the segments of the dictionary.
16 . The processor of claim 15 , wherein the first metadata is indicative of at least a first input segment location and a first output segment location associated with an individual segment of the at least two segments of the compressed data.
17 . The processor of claim 15 , wherein the decompressing the at least two segments of the compressed data comprises:
decompressing, based at least on the second metadata, one or more of the segments of the dictionary to generate the dictionary; and
decompressing, based at least on the dictionary and the first metadata, the at least two segments of the compressed data to generate the output data.
18 . The processor of claim 15 , wherein an individual segment of the segments of the dictionary is used to decompress at least an individual segment of the at least two segments of the compressed data.
19 . The processor of claim 15 , wherein the generation of the output data is further by using third metadata indicative of at least one of:
a first index associated with the at least two segments of the compressed data; or
a second index associated with the segments of the dictionary.
20 . The processor of claim 15 , wherein the processor is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing deep learning operations;
a system for performing real-time streaming broadcasts;
a system for performing video monitoring services;
a system for performing intelligent video analysis;
a system implemented using an edge device;
a system for generating ray-traced graphical output;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.