Structural data matching using neural network encoders
Implementations of the present disclosure include methods, systems, and computer-readable storage mediums for receiving first and second data sets, both the first and second data sets including structured data in a plurality of columns, for each of the first data set and the second data set, inputting each column into an encoder specific to a column type of a respective column, the encoder providing encoded data for the first data set, and the second data set, respectively, providing a first multi-dimensional vector based on encoded data of the first data set, providing a second multi-dimensional vector based on encoded data of the second data set, and outputting the first multi-dimensional vector and the second multi-dimensional vector to a loss-function, the loss-function processing the first multi-dimensional vector and the second multi-dimensional vector to provide an output, the output representing matched data points between the first and second data sets.
1 . A computer-implemented method executed by one or more processors, the method comprising:
receiving a first data set and a second data set, both the first data set and the second data set comprising structured data in a plurality of columns;
pre-processing data values of each column of each of the first data set and the second data set, such that data values within individual columns are of a same length in terms of number of characters;
for each of the first data set and the second data set, inputting each column into an encoder specific to a column type of a respective column, the encoder providing encoded data for the first data set, and the second data set, respectively;
providing a first multi-dimensional vector based on encoded data of the first data set by mapping a first output of first fully connected layers to a latent space independently of a second output of second fully connected layers;
providing a second multi-dimensional vector based on encoded data of the second data set by mapping the second output of the second fully connected layers to the latent space independently of the first output of the first fully connected layers; and
outputting the first multi-dimensional vector and the second multi-dimensional vector to a loss-function, the loss function being computed during a supervised training process using triplet mining comprising anchor points, positive points that match respective anchor points, and negative points that are non-matching to respective anchor points, the loss-function processing the first multi-dimensional vector and the second multi-dimensional vector to provide an output, the output representing an exact match between a data point of the first data set and a data point of the second data set.
2 . The method of claim 1 , wherein a same encoder is used to provide the encoded data of the first data set, and the encoded data of the second data set.
3 . The method of claim 1 , wherein, prior to the encoder providing encoded data, data values of one or more of the first data set, and the second data set are pre-processed to provide revised data values.
4 . The method of claim 3 , wherein pre-processing comprises pre-appending one or more zeros to a numerical data value.
5 . The method of claim 3 , wherein pre-processing comprises pre-appending one or more spaces to a string data value.
6 . The method of claim 1 , further comprising filtering at least one column from each of the first data set, and the second data set prior to providing encoded data.
7 . The method of claim 1 , further comprising determining a column type for each column of the plurality of columns.
8 . A non-transitory computer-readable storage medium coupled to one or more processors and having instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to perform operations, the operations comprising:
receiving a first data set and a second data set, both the first data set and the second data set comprising structured data in a plurality of columns;
pre-processing data values of each column of each of the first data set and the second data set, such that data values within individual columns are of a same length in terms of number of characters;
for each of the first data set and the second data set, inputting each column into an encoder specific to a column type of a respective column, the encoder providing encoded data for the first data set, and the second data set, respectively;
providing a first multi-dimensional vector based on encoded data of the first data set by mapping a first output of first fully connected layers to a latent space independently of a second output of second fully connected layers;
providing a second multi-dimensional vector based on encoded data of the second data set by mapping the second output of the second fully connected layers to the latent space independently of the first output of the first fully connected layers; and
outputting the first multi-dimensional vector and the second multi-dimensional vector to a loss-function, the loss function being computed during a supervised training process using triplet mining comprising anchor points, positive points that match respective anchor points, and negative points that are non-matching to respective anchor points, the loss-function processing the first multi-dimensional vector and the second multi-dimensional vector to provide an output, the output representing an exact match between a data point of the first data set and a data point of the second data set.
9 . The computer-readable storage medium of claim 8 , wherein a same encoder is used to provide the encoded data of the first data set, and the encoded data of the second data set.
10 . The computer-readable storage medium of claim 8 , wherein, prior to the encoder providing encoded data, data values of one or more of the first data set, and the second data set are pre-processed to provide revised data values.
11 . The computer-readable storage medium of claim 10 , wherein pre-processing comprises pre-appending one or more zeros to a numerical data value.
12 . The computer-readable storage medium of claim 10 , wherein pre-processing comprises pre-appending one or more spaces to a string data value.
13 . The computer-readable storage medium of claim 8 , wherein operations further comprise filtering at least one column from each of the first data set, and the second data set prior to providing encoded data.
14 . The computer-readable storage medium of claim 8 , wherein operations further comprise determining a column type for each column of the plurality of columns.
15 . A system, comprising:
a computing device; and
a computer-readable storage device coupled to the computing device and having instructions stored thereon which, when executed by the computing device, cause the computing device to perform operations, the operations comprising:
receiving a first data set and a second data set, both the first data set and the second data set comprising structured data in a plurality of columns;
pre-processing data values of each column of each of the first data set and the second data set, such that data values within individual columns are of a same length in terms of number of characters;
for each of the first data set and the second data set, inputting each column into an encoder specific to a column type of a respective column, the encoder providing encoded data for the first data set, and the second data set, respectively;
providing a first multi-dimensional vector based on encoded data of the first data set by mapping a first output of first fully connected layers to a latent space independently of a second output of second fully connected layers;
providing a second multi-dimensional vector based on encoded data of the second data set by mapping the second output of the second fully connected layers to the latent space independently of the first output of the first fully connected layers; and
outputting the first multi-dimensional vector and the second multi-dimensional vector to a loss-function, the loss function being computed during a supervised training process using triplet mining comprising anchor points, positive points that match respective anchor points, and negative points that are non-matching to respective anchor points, the loss-function processing the first multi-dimensional vector and the second multi-dimensional vector to provide an output, the output representing an exact match between a data point of the first data set and a data point of the second data set.
16 . The system of claim 15 , wherein a same encoder is used to provide the encoded data of the first data set, and the encoded data of the second data set.
17 . The system of claim 15 , wherein, prior to the encoder providing encoded data, data values of one or more of the first data set, and the second data set are pre-processed to provide revised data values.
18 . The system of claim 17 , wherein pre-processing comprises pre-appending one or more zeros to a numerical data value.
19 . The system of claim 17 , wherein pre-processing comprises pre-appending one or more spaces to a string data value.
20 . The system of claim 15 , wherein operations further comprise filtering at least one column from each of the first data set, and the second data set prior to providing encoded data.