Systems and methods for sequence-based data chunking for deduplication
Disclosed herein are systems and method for data chunking. In one aspect, a method includes consecutively scanning each respective byte in a byte stream of data. The method includes in response to detecting a first amount of adjacent bytes with values arranged in a decreasing order, marking a cut-off point of a data chunk comprising at least the adjacent bytes. The method includes in response to detecting a second amount of bytes with values arranged in an increasing order, executing a jump mechanism that skips scanning of a fixed amount of bytes after the bytes with values arranged in the increasing order. The method includes subsequent to scanning the byte stream in entirety, outputting a plurality of cut-off points of identified data chunks.
1 . A method for data chunking, the method comprising:
identifying, by a hardware processor, a desired throughput value for data chunking a byte stream of data by:
determining, by the hardware processor, a size of the byte stream;
receiving, by the hardware processor, a threshold period of time to complete the data chunking; and
calculating, by the hardware processor, the desired throughput value based on the size of the byte stream and the threshold period of time;
setting, by the hardware processor, a first amount of adjacent bytes, a second amount of bytes, and a fixed amount of bytes of a jump mechanism that achieve the desired throughput value;
consecutively scanning, by the hardware processor, each respective byte in the byte stream of data;
in response to detecting the first amount of adjacent bytes with values arranged in a decreasing order, marking, by the hardware processor, a cut-off point of a data chunk comprising at least the adjacent bytes;
in response to detecting the second amount of bytes with values arranged in an increasing order, executing, by the hardware processor, the jump mechanism that skips scanning of the fixed amount of bytes after the bytes with values arranged in the increasing order; and
subsequent to scanning the byte stream in entirety, outputting, by the hardware processor, a plurality of cut-off points of identified data chunks.
2 . The method of claim 1 , wherein setting the first amount of adjacent bytes, the second amount of bytes, and the fixed amount of bytes of the jump mechanism that achieve the desired throughput value comprises executing a machine learning model trained to calculate for an input throughput value: an output amount of adjacent bytes, an output amount of bytes, and an output amount of bytes for the jump mechanism.
3 . The method of claim 2 , wherein the machine learning model is trained on a training dataset comprising output byte amounts for various input throughput values.
4 . A method for data chunking, the method comprising:
identifying, by a hardware processor, a desired throughput value for data chunking a byte stream of data by:
determining, by the hardware processor, a size of the byte stream;
receiving, by the hardware processor, a threshold period of time to complete the data chunking; and
calculating, by the hardware processor, the desired throughput value based on the size of the byte stream and the threshold period of time;
setting, by the hardware processor, a first amount of adjacent bytes, a second amount of bytes, and a fixed amount of bytes of a jump mechanism based on the desired throughput value;
consecutively scanning, by the hardware processor, each respective byte in the byte stream of data;
in response to detecting the first amount of adjacent bytes with values arranged in an increasing order, marking, by the hardware processor, a cut-off point of a data chunk comprising at least the adjacent bytes;
in response to detecting the second amount of bytes with values arranged in a decreasing order, executing, by the hardware processor, the jump mechanism that skips scanning of the fixed amount of bytes after the bytes with values arranged in the decreasing order; and
subsequent to scanning the byte stream in entirety, outputting, by the hardware processor, a plurality of cut-off points of identified data chunks.
5 . The method of claim 4 , wherein the bytes with values arranged in the decreasing order are not all adjacent to one another.
6 . The method of claim 5 , wherein a majority of the second amount of bytes are arranged in the decreasing order.
7 . The method of claim 4 , wherein one or both of the first amount of adjacent bytes and the second amount of bytes is incremented when a respective cut-off point is marked.
8 . The method of claim 4 , wherein one or both of the first amount of adjacent bytes and the second amount of bytes is reset to a default value when the jump mechanism is executed.
9 . The method of claim 4 , further comprising:
subsequent to identifying a data chunk by marking a respective cut-off point, determining an amount of bytes remaining in the byte stream; and
in response to determining that the amount of bytes is less than the first amount of adjacent bytes, identifying the bytes remaining in the byte stream as a last data chunk in the byte stream.
10 . A system for data chunking, comprising:
at least one memory; and
at least one hardware processor coupled with the at least one memory and configured, individually or in combination, to:
identify a desired throughput value for data chunking a byte stream of data by:
determining a size of the byte stream;
receiving a threshold period of time to complete the data chunking; and
calculating the desired throughput value based on the size of the byte stream and the threshold period of time;
set a first amount of adjacent bytes, a second amount of bytes, and a fixed amount of bytes of a jump mechanism that achieve the desired throughput value;
consecutively scan each respective byte in the byte stream of data;
in response to detecting the first amount of adjacent bytes with values arranged in an increasing order, mark a cut-off point of a data chunk comprising at least the adjacent bytes;
in response to detecting the second amount of bytes with values arranged in a decreasing order, execute the jump mechanism that skips scanning of the fixed amount of bytes after the bytes with values arranged in the decreasing order; and
subsequent to scanning the byte stream in entirety, output a plurality of cut-off points of identified data chunks.
11 . The system of claim 10 , wherein the bytes with values arranged in the decreasing order are not all adjacent to one another.
12 . The system of claim 11 , wherein a majority of the second amount of bytes are arranged in the decreasing order.
13 . The system of claim 10 , wherein the second amount of bytes is incremented when a respective cut-off point is marked.
14 . The system of claim 10 , wherein the second amount of bytes is reset to a default value when the jump mechanism is executed.
15 . The system of claim 10 , wherein the at least one hardware processor is further configured to:
subsequent to identifying a data chunk by marking a respective cut-off point, determine an amount of bytes remaining in the byte stream; and
in response to determining that the amount of bytes is less than the first amount of adjacent bytes, identify the bytes remaining in the byte stream as a last data chunk in the byte stream.