Encoding / decoding system and method
A computer-implemented method, computer program product and computing system for: processing an unencoded data file to identify a plurality of file segments, wherein the unencoded data file is a dataset for use with an ML process; mapping each of the plurality of file segments to a portion of a dictionary file to generate a plurality of mappings that each include a starting location and a length, thus generating a related encoded data file based, at least in part, upon the plurality of mappings; receiving a request to manipulate the unencoded data file from the ML process; and processing the related encoded data file based, at least in part, upon the plurality of mappings and the dictionary file to generate a modified encoded data file that represents the requested manipulations of the unencoded data file.
1 . A computer-implemented method, executed on a computing device comprising:
processing an unencoded data file to identify a plurality of file segments, wherein the unencoded data file is a dataset for use with a machine learning (ML) process;
mapping each of the plurality of file segments to a portion of a dictionary file, wherein the dictionary file includes a binary sequence of all possible combinations of a number having a defined bit length;
generating a plurality of mappings that each include a starting location as a bit-wise offset within the dictionary file and a length as a number of bits within the dictionary file beginning from the starting location within the dictionary file, thus generating a related encoded data file based, at least in part, upon the plurality of mappings;
receiving a request to manipulate the unencoded data file from the ML process; and
processing the related encoded data file based, at least in part, upon the plurality of mappings and the dictionary file to generate a modified encoded data file that represents the requested manipulations of the unencoded data file.
2 . The computer-implemented method of claim 1 wherein processing the related encoded data file based, at least in part, upon the plurality of mappings and the dictionary file to generate the modified encoded data file that represents the requested manipulations of the unencoded data file includes one or more of:
performing a homomorphic encoding operation; and
performing a heteromorphic encoding operation.
3 . The computer-implemented method of claim 1 wherein the related encoded data file includes one or more of:
a related compressed data file; and
a related encrypted data file.
4 . The computer-implemented method of claim 1 wherein the dictionary file includes a plurality of discrete entries.
5 . The computer-implemented method of claim 4 wherein processing the related encoded data file based, at least in part, upon the plurality of mappings and the dictionary file to generate the modified encoded data file that represents the requested manipulations of the unencoded data file includes:
processing the related encoded data file, in a byte-wise fashion, based upon the plurality of mappings and the dictionary file to generate the modified encoded data file that represents the requested manipulations of the unencoded data file.
6 . The computer-implemented method of claim 1 wherein the dictionary file includes a plurality of concatenated entries.
7 . The computer-implemented method of claim 6 wherein processing the related encoded data file based, at least in part, upon the plurality of mappings and the dictionary file to generate the modified encoded data file that represents the requested manipulations of the unencoded data file includes:
processing the related encoded data file, in a bit-wise fashion, based upon the plurality of mappings and the dictionary file to generate a modified encoded data file that represents the requested manipulations of the unencoded data file.
8 . The computer-implemented method of claim 1 wherein the requested manipulations comprise one or more of:
a computation operation;
a search operation;
an append operation;
a splitting operation;
a joining operation; and
a concatenating operation.
9 . The computer-implemented method of claim 1 wherein the related encoded data file includes a massive data set.
10 . The computer-implemented method of claim 1 wherein the related encoded data file requires less computational overhead than the unencoded data file.
11 . A non-transitory computer readable medium having a plurality of instructions stored thereon which, when executed by a processor, cause the processor to perform operations comprising:
processing an unencoded data file to identify a plurality of file segments, wherein the unencoded data file is a dataset for use with a machine learning (ML) process;
mapping each of the plurality of file segments to a portion of a dictionary file, wherein the dictionary file includes a binary sequence of all possible combinations of a number having a defined bit length;
generating a plurality of mappings that each include a starting location as a bit-wise offset within the dictionary file and a length as a number of bits within the dictionary file beginning from the starting location within the dictionary file, thus generating a related encoded data file based, at least in part, upon the plurality of mappings;
receiving a request to manipulate the unencoded data file from the ML process; and
processing the related encoded data file based, at least in part, upon the plurality of mappings and the dictionary file to generate a modified encoded data file that represents the requested manipulations of the unencoded data file.
12 . The non-transitory computer readable medium of claim 11 wherein processing the related encoded data file based, at least in part, upon the plurality of mappings and the dictionary file to generate the modified encoded data file that represents the requested manipulations of the unencoded data file includes one or more of:
performing a homomorphic encoding operation, and
performing a heteromorphic encoding operation.
13 . The non-transitory computer readable medium of claim 11 wherein the related encoded data file includes one or more of:
a related compressed data file; and
a related encrypted data file.
14 . The non-transitory computer readable medium of claim 11 wherein the dictionary file includes a plurality of discrete entries.
15 . The non-transitory computer readable medium of claim 14 wherein processing the related encoded data file based, at least in part, upon the plurality of mappings and the dictionary file to generate the modified encoded data file that represents the requested manipulations of the unencoded data file includes:
processing the related encoded data file, in a byte-wise fashion, based upon the plurality of mappings and the dictionary file to generate a modified encoded data file that represents the requested manipulations of the unencoded data file.
16 . The non-transitory computer readable medium of claim 11 wherein the dictionary file includes a plurality of concatenated entries.
17 . The non-transitory computer readable medium of claim 16 wherein processing the related encoded data file based, at least in part, upon the plurality of mappings and the dictionary file to generate the modified encoded data file that represents the requested manipulations of the unencoded data file includes:
processing the related encoded data file, in a bit-wise fashion, based upon the plurality of mappings and the dictionary file to generate the modified encoded data file that represents the requested manipulations of the unencoded data file.
18 . The non-transitory computer readable medium of claim 11 wherein the requested manipulations comprise one or more of:
a computation operation;
a search operation;
an append operation;
a splitting operation;
a joining operation; and
a concatenating operation.
19 . The non-transitory computer readable medium of claim 11 wherein the related encoded data file includes a data set.
20 . The non-transitory computer readable medium of claim 11 wherein the related encoded data file requires less computational overhead than the unencoded data file.
21 . A computing system comprising a memory storing instructions and a processor configured to execute the instructions to cause the system to perform operations comprising:
processing an unencoded data file to identify a plurality of file segments, wherein the unencoded data file is a dataset for use with a machine learning (ML) process;
mapping each of the plurality of file segments to a portion of a dictionary file, wherein the dictionary file includes a binary sequence of all possible combinations of a number having a defined bit length;
generating a plurality of mappings that each include a starting location as a bit-wise offset within the dictionary file and a length as a number of bits within the dictionary file beginning from the starting location within the dictionary file, thus generating a related encoded data file based, at least in part, upon the plurality of mappings;
receiving a request to manipulate the unencoded data file from the ML process; and
processing the related encoded data file based, at least in part, upon the plurality of mappings and the dictionary file to generate a modified encoded data file that represents the requested manipulations of the unencoded data file.
22 . The computing system of claim 21 wherein processing the related encoded data file based, at least in part, upon the plurality of mappings and the dictionary file to generate the modified encoded data file that represents the requested manipulations of the unencoded data file includes one or more of:
performing a homomorphic encoding operation; and
performing a heteromorphic encoding operation.
23 . The computing system of claim 21 wherein the related encoded data file includes one or more of:
a related compressed data file; and
a related encrypted data file.
24 . The computing system of claim 21 wherein the dictionary file includes a plurality of discrete entries.
25 . The computing system of claim 24 wherein processing the related encoded data file based, at least in part, upon the plurality of mappings and the dictionary file to generate a modified encoded data file that represents the requested manipulations of the unencoded data file includes:
processing the related encoded data file, in a byte-wise fashion, based upon the plurality of mappings and the dictionary file to generate the modified encoded data file that represents the requested manipulations of the unencoded data file.
26 . The computing system of claim 21 wherein the dictionary file includes a plurality of concatenated entries.
27 . The computing system of claim 26 wherein processing the related encoded data file based, at least in part, upon the plurality of mappings and the dictionary file to generate a modified encoded data file that represents the requested manipulations of the unencoded data file includes:
processing the related encoded data file, in a bit-wise fashion, based upon the plurality of mappings and the dictionary file to generate a modified encoded data file that represents the requested manipulations of the unencoded data file.
28 . The computing system of claim 21 wherein the requested manipulations comprise one or more of:
a computation operation;
a search operation;
an append operation;
a splitting operation;
a joining operation; and
a concatenating operation.
29 . The computing system of claim 21 wherein the related encoded data file includes a massive data set.
30 . The computing system of claim 21 wherein the related encoded data file requires less computational overhead than the unencoded data file.