System and method for identifying delimiters in a computer file
A system and method automatically identifies any or all of potential row, column and string delimiters in a file in which such delimiters are unknown to the program making such identification.
1. A computer-implemented method of parsing a file, comprising:
receiving a plurality of tokenizing schemas, each tokenizing schema specifying a plurality of delimiters;
for each of the plurality of tokenizing schemas:
generating a tokenized version of the file based on the tokenizing schema, the tokenized version of the file obtained by replacing characters of the file with tokens representing delimiters according to the tokenized schema;
for each of a plurality of sizes, identifying in the tokenized version of the file, a pattern of delimiters that repeats in the tokenized version of file, the pattern having the size; and
for each of the plurality of patterns of delimiters identified, determining a score based on a frequency with which the pattern of delimiters appears in the tokenized version of the file;
selecting a tokenizing schema from the plurality of tokenized schemas based on the scores of the plurality of patterns of delimiters associated with each of the plurality of tokenized schemas; and
parsing the file using the selected tokenizing schema.
2. The computer-implemented method of claim 1 , further comprising:
selecting one or more characters corresponding to a terminator delimiter of the tokenizing schema as a row delimiter for the file, wherein the terminator delimiter is a sequence of one or more tokens at the end of a pattern.
3. The computer-implemented method of claim 2 , method further comprising:
selecting one or more characters corresponding to a token other than the terminator token as a column delimiter for the file responsive to the other token being a most common token within the pattern of the selected tokenizing schema.
4. The computer-implemented method of claim 1 , wherein the plurality of delimiters for each tokenizing schema comprises a potential string delimiter pair including an opening delimiter and a closing delimiter, wherein generating the tokenized version of the file skips characters between the opening delimiter and the closing delimiter.
5. The computer-implemented method of claim 4 , wherein the plurality of delimiters for each tokenizing schema further comprises a potential row delimiter and a potential column delimiter, wherein generating the tokenized version of the file skips characters between the opening delimiter and the closing delimiter including potential row delimiters and potential column delimiters.
6. The computer-implemented method of claim 1 , wherein the score for a particular pattern of a particular size is further based on a measure of a distance between the particular pattern of a particular size and tokens of the tokenized file in a sliding window having the particular size.
7. A computer system for parsing a file, comprising:
a computer processor; and
a non-transitory computer readable storage medium storing instructions for:
receiving a plurality of tokenizing schemas, each tokenizing schema specifying a plurality of delimiters;
for each of the plurality of tokenizing schemas:
generating a tokenized version of the file based on the tokenizing schema, the tokenized version of the file obtained by replacing characters of the file with tokens representing delimiters according to the tokenized schema;
for each of a plurality of sizes, identifying in the tokenized version of the file a pattern of delimiters that repeats in the tokenized version of file, the pattern having the size; and
for each of the plurality of patterns of delimiters identified, determining a score based on a frequency with which the pattern of delimiters appears in the tokenized version of the file;
selecting a tokenizing schema from the plurality of tokenized schemas based on the scores of the plurality of patterns of delimiters associated with each of the plurality of tokenized schemas; and
parsing the file using the selected tokenizing schema.
8. The computer system of claim 7 , wherein the instructions stored in the non-transitory computer readable storage medium are further for:
selecting one or more characters corresponding to a terminator delimiter of the tokenizing schema as a row delimiter for the file, wherein the terminator delimiter is a sequence of one or more tokens at the end of a pattern.
9. The computer system of claim 8 , wherein the instructions stored in the non-transitory computer readable storage medium are further for:
selecting one or more characters corresponding to a token other than the terminator token as a column delimiter for the file responsive to the other token being a most common token within the pattern of the selected tokenizing schema.
10. The computer system of claim 7 , wherein the plurality of delimiters for each tokenizing schema comprises a potential string delimiter pair including an opening delimiter and a closing delimiter, wherein generating the tokenized version of the file skips characters between the opening delimiter and the closing delimiter.
11. The computer system of claim 10 , wherein the plurality of delimiters for each tokenizing schema further comprises a potential row delimiter and a potential column delimiter, wherein generating the tokenized version of the file skips characters between the opening delimiter and the closing delimiter including potential row delimiters and potential column delimiters.
12. The computer system of claim 7 , wherein the score for a particular pattern of a particular size is further based on a measure of a distance between the particular pattern of a particular size and tokens of the tokenized file in a sliding window having the particular size.
13. A non-transitory computer readable storage medium storing instructions for parsing a file, wherein the instructions are for:
receiving a plurality of tokenizing schemas, each tokenizing schema specifying a plurality of delimiters;
for each of the plurality of tokenizing schemas:
generating a tokenized version of the file based on the tokenizing schema, the tokenized version of the file obtained by replacing characters of the file with tokens representing delimiters according to the tokenized schema;
for each of a plurality of sizes, identifying in the tokenized version of the file, a pattern of delimiters that repeats in the tokenized version of file, the pattern having the size; and
for each of the plurality of patterns of delimiters identified, determining a score based on a frequency with which the pattern of delimiters appears in the tokenized version of the file;
selecting a tokenizing schema from the plurality of tokenized schemas based on the scores of the plurality of patterns of delimiters associated with each of the plurality of tokenized schemas; and
parsing the file using the selected tokenizing schema.
14. The non-transitory computer readable storage medium of claim 13 , wherein the instructions stored in the non-transitory computer readable storage medium are further for:
selecting one or more characters corresponding to a terminator delimiter of the tokenizing schema as a row delimiter for the file, wherein the terminator delimiter is a sequence of one or more tokens at the end of a pattern.
15. The non-transitory computer readable storage medium of claim 14 , wherein the instructions stored in the non-transitory computer readable storage medium are further for:
selecting one or more characters corresponding to a token other than the terminator token as a column delimiter for the file responsive to the other token being a most common token within the pattern of the selected tokenizing schema.
16. The non-transitory computer readable storage medium of claim 13 , wherein the plurality of delimiters for each tokenizing schema comprises a potential string delimiter pair including an opening delimiter and a closing delimiter, wherein generating the tokenized version of the file skips characters between the opening delimiter and the closing delimiter.
17. The non-transitory computer readable storage medium of claim 16 , wherein the plurality of delimiters for each tokenizing schema further comprises a potential row delimiter and a potential column delimiter, wherein generating the tokenized version of the file skips characters between the opening delimiter and the closing delimiter including potential row delimiters and potential column delimiters.
18. The non-transitory computer readable storage medium of claim 13 , wherein the score for a particular pattern of a particular size is further based on a measure of a distance between the particular pattern of a particular size and tokens of the tokenized file in a sliding window having the particular size.