Data transformation of Cassandra files for improved deduplication during backup
Cassandra SSTable data is transformed to provide data rows that are a consistent size such that data in each row has a length that is contained within a selected fixed sized kilobyte segment for deduplication. Tables of a Cassandra cluster node are translated in parallel to JSON format using Cassandra SSTableDump and the table rows are parsed to provide data rows corresponding to the data in each table row. Each row of data is padded with a predictable pattern of bits such that the data row has a length corresponding to the selected fixed segment size and has boundary locations that correspond to multiple of the selected segment size. Since each row of data starts on a segment boundary, duplicate rows of data will be identified wherever they move within a table.
1. A method of improving deduplication and backup of data files that store changing data in tables as a plurality of rows of variable length data that are reordered and merged with rows as data changes, comprising:
selecting a fixed segment size of data as a data unit for deduplication;
translating each row of data in a selected table into a format that enables access to the data in each said row in said selected table;
parsing the data in said translated rows of said selected table to provide row lengths that are sized to fit within multiples of said fixed segment size;
adjusting the row length of each parsed row in said selected table to have a length that corresponds to said fixed segment size by padding each parsed row with a predetermined predictable pattern of bytes to produce a transformed table having rows of the same size; and
deduplicating individual rows of data in said transformed table to identify data that is a duplicate of previously stored data.
2. The method of claim 1 , wherein each row of data has a unique key value, and data rows are ordered within said selected table by said key values.
3. The method of claim 1 , wherein said translating comprises translating the data in the selected table into a format such that the data in each row is contained within a length corresponding to said fixed segment size.
4. The method of claim 1 , wherein said adjusting comprises padding the length of each row of said selected table such that the boundaries of each such row are on locations in said selected table corresponding to multiples of said selected segment size.
5. The method of claim 1 , wherein said tables are Cassandra tables and wherein said translating, said parsing and said adjusting are performed in parallel on a plurality of said Cassandra tables.
6. The method of claim 5 , wherein the Cassandra tables are located on one or more nodes of a Cassandra cluster of nodes, and said method comprises transferring said transformed table over a network to another cluster for said deduplicating.
7. The method of claim 6 further comprising generating a fingerprint for each row of data, and transferring said fingerprints over said network for comparison with stored fingerprints of previous data to identify a row of duplicate data before transferring said row of duplicate data over said network.
8. Non-transitory computer readable storage medium storing executable instructions for controlling a computer to perform a method of deduplication and backup of data files that store changing data in tables as a plurality of rows of variable length data that are reordered and merged with rows as data changes, comprising:
selecting a fixed segment size of data as a data unit for deduplication;
translating of each row of data in a selected table into a format that enables access to the data in each row in said selected table;
parsing the data in said translated rows of said selected table to provide row lengths that are sized to fit within multiples of said fixed segment size;
adjusting the row length of each parsed row in said selected table to have a length that corresponds to said fixed segment size by padding each parsed row with a predetermined predictable pattern of bytes to produce a transformed table having rows of data of the same size; and
deduplicating individual rows of data in said transformed table to identify data that is a duplicate of previously stored data.
9. The non-transitory computer readable storage medium of claim 8 , wherein each row of data has a unique key value, and data rows are ordered within said selected table by said key values.
10. The non-transitory computer readable storage medium of claim 8 , wherein said translating comprises translating the data in the selected table into a format such that the data in each row is contained within a length corresponding to said fixed segment size.
11. The non-transitory computer readable storage medium of claim 8 , wherein said adjusting comprises padding the length of each row of said selected table such that the boundaries of each such row are on locations in said selected table corresponding to multiples of said selected segment size.
12. The non-transitory computer readable storage medium of claim 8 , wherein said tables are Cassandra tables and wherein said translating, said parsing and said adjusting are performed in parallel on a plurality of said Cassandra tables.
13. The non-transitory computer readable storage medium of claim 12 , wherein the Cassandra tables are located on one or more nodes of a Cassandra cluster of nodes, and said method comprises transferring said transformed table over a network to another cluster for said deduplicating.
14. The non-transitory computer readable storage medium of claim 13 , further comprising generating a fingerprint for each row of data, and transferring said fingerprints over said network for comparison with stored fingerprints of previous data to identify a row of duplicate data before transferring said row of duplicate data over said network.