Hierarchical delimiter identification for parsing of raw data
An example system for parsing and transforming input data that includes processing circuitry and memory, the memory configured to store the input data. The processing circuitry is configured to determine a first delimiter in the input data. The processing circuitry is configured to determine a plurality of second delimiter hypotheses and parse the input data according to the first delimiter and the plurality of second delimiter hypotheses to generate a plurality of tables that are each associated with a respective one of the plurality of second delimiter hypotheses. The processing circuitry is configured to determine a respective consistency score for each of the plurality of tables and select a table from among the plurality of tables based on the respective consistency score associated with the table. The processing circuitry is configured to format the input data based on the selected table to generate formatted data and output the formatted data.
1 . A computing system comprising:
one or more processors; and
one or more memories storing processor-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
obtaining input data;
determining a first delimiter within the input data;
generating, from the input data and using the first delimiter, an arranged string;
determining an N most frequent characters in the arranged string, wherein Nis greater than one;
determining a plurality of second delimiter hypotheses, the plurality of second delimiter hypotheses comprising the N most frequent characters in the arranged string, each of the plurality of second delimiter hypotheses comprising a respective potential delimiter to use with the first delimiter;
parsing the arranged string using each of the plurality of second delimiter hypotheses;
generating, based on parsing the arranged string using each of the plurality of second delimiter hypotheses, a plurality of tables that are each associated with a respective one of the plurality of second delimiter hypotheses;
determining a respective consistency score associated with each respective table of the plurality of tables, wherein the respective consistency score is based on at least one of a total number of patterns in the respective table, a total number of tuples in the respective table, a total number of delimiters per pattern in the respective table, a total number of columns or rows in the respective table, or a total number cells defined by the rows and the columns of the respective table filled by values;
selecting a table from among the plurality of tables based on the respective consistency score associated with the table;
formatting the input data based on the selected table to generate formatted data; and
outputting the formatted data.
2 . The computing system of claim 1 , wherein formatting the input data comprises formatting the input data into payload data labeled according to a column and a row of the selected table.
3 . The computing system of claim 1 , wherein selecting the table from among the plurality of tables comprises selecting the table having a highest consistency score among the plurality of tables.
4 . The computing system of claim 1 , wherein the operations further comprise:
determining a plurality of child delimiter hypotheses within the input data for a column or a row of the selected table, the column including column data or the row including row data, each of the plurality of child delimiter hypotheses comprising a respective potential child delimiter;
parsing the column data or the row data according to each of the plurality of child delimiter hypotheses;
generating, based on parsing the column data or the row data, a plurality of child tables, each of the plurality of child tables associated with a respective one of the plurality of child delimiter hypotheses;
determining a respective child consistency score for each of the plurality of child delimiter hypotheses; and
selecting a child delimiter hypothesis from among the plurality of child delimiter hypotheses or no child delimiter hypothesis for the column or the row based on the respective child consistency score for each of the plurality of child delimiter hypotheses.
5 . The computing system of claim 4 , wherein selecting the child delimiter hypothesis from among the plurality of child delimiter hypotheses or no child delimiter hypothesis for the column or the row comprises selecting no child delimiter hypothesis based on the respective child consistency score for each of the plurality of child delimiter hypotheses being equal to zero.
6 . The computing system of claim 4 , wherein selecting the child delimiter hypothesis from among the plurality of child delimiter hypotheses or no child delimiter hypothesis for the column or the row comprises selecting the child delimiter hypothesis based on the child delimiter hypothesis having a highest consistency score among the respective child consistency score of each of the plurality of child delimiter hypotheses for the column or the row.
7 . The computing system of claim 1 , wherein the first delimiter comprises a row delimiter, the plurality of second delimiter hypotheses comprises a plurality of column delimiter hypotheses, and wherein determining the respective consistency score comprises determining:
P
(
x
,
θ
)
=
1
k
∑
k
=
1
k
N
k
(
M
k
(
M
k
+
1
)
)
*
(
M
col
R
C
filled
)
,
where P is a function yielding the respective consistency score, x is a block of input text, θ is a hypothetical delimiter applied to the input text, k is a total number of unique patterns found while processing for θ, N k is a total number of tuples, M k is a total number of delimiters per pattern, M col is a total number of columns created and RC filled is a total number of rows and columns filled by values.
8 . The computing system of claim 1 , wherein the operations further comprise:
determining a second delimiter hypothesis as a respective one of the plurality of second delimiter hypotheses associated with the selected table; and
outputting at least two of the first delimiter, the second delimiter hypothesis, or a child delimiter hypothesis.
9 . The computing system of claim 1 , wherein determining the first delimiter comprises determining that the first delimiter comprises an only potential first delimiter from a plurality of potential first delimiters to appear in input text.
10 . The computing system of claim 1 , wherein determining the first delimiter comprises determining the first delimiter to be a potential first delimiter with a greatest regularity in a number of characters separating the first delimiter from among a plurality of potential first delimiters or a potential first delimiter having a highest consistency score among the plurality of potential first delimiters.
11 . A computer-implemented method comprising:
determining, by one or more processors, a first delimiter within input data;
generating, by the one or more processors, from the input data and using the first delimiter, an arranged string;
determining, by the one or more processors, an N most frequent characters in the arranged string, wherein N is greater than one;
determining, by the one or more processors, a plurality of second delimiter hypotheses, the plurality of second delimiter hypotheses comprising the N most frequent characters in the arranged string, each of the plurality of second delimiter hypotheses comprising a respective potential delimiter to use with the first delimiter;
parsing, by the one or more processors, the arranged string using each of the plurality of second delimiter hypotheses;
generating, by the one or more processors, based on parsing the arranged string using each of the plurality of second delimiter hypotheses, a plurality of tables that are each associated with a respective one of the plurality of second delimiter hypotheses;
determining, by the one or more processors, a respective consistency score associated with each respective table of the plurality of tables, wherein the respective consistency score is based on at least one of a total number of patterns in the respective table, a total number of tuples in the respective table, a total number of delimiters per pattern in the respective table, a total number of columns or rows in the respective table, or a total number cells defined by the rows and the columns of the respective table filled by values;
selecting, by the one or more processors, a table from among the plurality of tables based on the respective consistency score associated with the table;
formatting, by the one or more processors, the input data based on the selected table to generate formatted data; and
outputting, by the one or more processors, the formatted data.
12 . The computer-implemented method of claim 11 , wherein formatting the input data comprises formatting the input data into payload data labeled according to a column and a row of the selected table.
13 . The computer-implemented method of claim 11 , wherein selecting the table from among the plurality of tables comprises selecting the table having a highest consistency score among the plurality of tables.
14 . The computer-implemented method of claim 11 , further comprising:
determining, by the one or more processors, a plurality of child delimiter hypotheses within the input data for a column or a row of the selected table, the column including column data or the row including row data, each of the plurality of child delimiter hypotheses comprising a respective potential child delimiter;
parsing, by the one or more processors, the column data or the row data according to each of the plurality of child delimiter hypotheses;
generating, by the one or more processors, based on parsing the column data or the row data, a plurality of child tables, each of the plurality of child tables associated with a respective one of the plurality of child delimiter hypotheses;
determining, by the one or more processors, a respective child consistency score for each of the plurality of child delimiter hypotheses; and
selecting, by the one or more processors, a child delimiter hypothesis from among the plurality of child delimiter hypotheses or no child delimiter hypothesis for the column or the row based on the respective child consistency score for each of the plurality of child delimiter hypotheses.
15 . The computer-implemented method of claim 14 , wherein selecting the child delimiter hypothesis from among the plurality of child delimiter hypotheses or no child delimiter hypothesis for the column or the row comprises selecting no child delimiter hypothesis based on the respective child consistency score for each of the plurality of child delimiter hypotheses being equal to zero.
16 . The computer-implemented method of claim 14 , wherein selecting the child delimiter hypothesis from among the plurality of child delimiter hypotheses or no child delimiter hypothesis for the column or the row comprises selecting the child delimiter hypothesis based on the child delimiter hypothesis having a highest consistency score among the repsective child consistency score for each of the plurality of child delimiter hypotheses for the column or the row.
17 . The computer-implemented method of claim 11 , wherein the first delimiter comprises a row delimiter, the plurality of second delimiter hypotheses comprises a plurality of column delimiter hypotheses, and wherein determining the respective consistency score comprises determining
P
(
x
,
θ
)
=
1
k
∑
k
=
1
k
N
k
(
M
k
(
M
k
+
1
)
)
*
(
M
col
R
C
filled
)
,
where P is a function yielding the respective consistency score, x is a block of input text, θ is a hypothetical delimiter applied to the input text, k is a total number of unique patterns found while processing for θ, N k is a total number of tuples, M k is a total number of delimiters per pattern, M col is a total number of columns created and RC filled is a total number of rows and columns filled by values.
18 . The computer-implemented method of claim 11 , wherein determining the first delimiter comprises determining that the first delimiter comprises an only potential first delimiter from a plurality of potential first delimiters to appear in a block of input text.
19 . The computer-implemented method of claim 11 , wherein determining the first delimiter comprises determining the first delimiter to be a potential first delimiter with a greatest regularity in a number of characters separating the first delimiter from among a plurality of potential first delimiters or a potential first delimiter having a highest consistency score among the plurality of potential first delimiters.
20 . One or more non-transitory, computer-readable media storing processor-executable instructions for parsing and transforming input data that, when executed by one or more processors, cause the one or more processors to:
obtain the input data;
determine a first delimiter within the input data;
generate, from the input data and using the first delimiter, an arranged string;
determine an N most frequent characters in the arranged string, wherein N is greater than one;
determine a plurality of second delimiter hypotheses, the plurality of second delimiter hypotheses comprising the N most frequent characters in the arranged string, each of the plurality of second delimiter hypotheses comprising a respective potential delimiter to use with the first delimiter;
parse the arranged string using each of the plurality of second delimiter hypotheses;
generate, based on parsing the arranged string using each of the plurality of second delimiter hypotheses, a plurality of tables that are each associated with a respective one of the plurality of second delimiter hypotheses;
determine a respective consistency score associated with each respective table of the plurality of tables, wherein the respective consistency score is based on at least one of a total number of patterns in the respective table, a total number of tuples in the respective table, a total number of delimiters per pattern in the respective table, a total number of columns or rows in the respective table, or a total number cells defined by the rows and the columns of the respective table filled by values;
select a table from among the plurality of tables based on the respective consistency score associated with the table;
format the input data based on the selected table to generate formatted data; and
output the formatted data.