System and method for identifying fields in a file using examples in the file received from a user
A system and method parses one or more fields from a file by receiving example locations of the field in the file, fashioning rules that describe the field from the locations, and then scoring the rules against some or all of the file.
1. A method of parsing a file, comprising:
retrieving from a file storage, a subset of a file stored in the file storage;
sending for display, via a user interface of a client device, the subset of the file;
receiving, via the user interface of the client device, a selection of a set of contiguous characters of the subset of the file;
determining a plurality of pattern sets, each pattern set comprising one or more patterns, each pattern describing one or more of a left set of contiguous characters occurring to the left of the selected set of contiguous characters, a right set of contiguous characters occurring to the right of the selected set of contiguous characters, and the selected set of contiguous characters;
for each of the plurality of pattern sets, assigning a plurality of scores comprising:
a first score indicative of regularity of the pattern set in rows of the file,
a second score indicative of a number of times the pattern set occurs in each row of the file, and
a third score indicative of uniformity of matches of the pattern set per row based on a deviation from an average number of matches per row of the file;
selecting a subset of the plurality of pattern sets based on the scores;
providing for display, via the user interface of the client device, results of an application of each of the selected subset of the plurality of pattern sets to each of at least one portion of the file;
receiving, via the user interface of the client device, a selection of one of the subset of the plurality of pattern sets for which results were provided for display; and
parsing the file using the selected pattern set.
2. The method of claim 1 , further comprising:
generating a tokenized version of at least a portion of the file, the tokenized version including a set of before tokens based on the left set of contiguous characters, a set of after tokens based on the right set of contiguous characters, and a set of selected tokens based on the selected set of contiguous characters.
3. The method of claim 1 , wherein the first score is based on a number of non-matching rows between rows that matched the selected pattern set for each pair of rows nearest to one another.
4. The method of claim 1 , further comprising:
providing to the user interface of the client device, information describing the selected subset of the plurality of pattern sets, the user interface configured to receive a selection of a pattern set.
5. The method of claim 1 , further comprising:
generating a new file based on data obtained by parsing the file; and
providing the new file as input to a program configured to analyze data obtained from the file.
6. A non-transitory computer readable storage medium storing instructions that when executed by a computer processor cause the computer processor to perform:
retrieving from a file storage, a subset of a file stored in the file storage;
sending for display, via a user interface of a client device, the subset of the file;
receiving, via the user interface of the client device, a selection of a set of contiguous characters of the subset of the file;
determining a plurality of pattern sets, each pattern set comprising one or more patterns, each pattern describing one or more of a left set of contiguous characters occurring to the left of the selected set of contiguous characters, a right set of contiguous characters occurring to the right of the selected set of contiguous characters, and the selected set of contiguous characters;
for each of the plurality of pattern sets, assigning a plurality of scores comprising:
a first score indicative of regularity of the pattern set in rows of the file,
a second score indicative of a number of times the pattern set occurs in each row of the file, and
a third score indicative of uniformity of matches of the pattern set per row based on a deviation from an average number of matches per row of the file;
selecting a subset of the plurality of pattern sets based on the scores;
providing for display, via the user interface of the client device, results of an application of each of the selected subset of the plurality of pattern sets to each of at least one portion of the file;
receiving, via the user interface of the client device, a selection of one of the subset of the plurality of pattern sets for which results were provided for display; and
parsing the file using the selected pattern set.
7. The non-transitory computer readable storage medium of claim 6 , wherein the stored instructions cause the computer processor to further perform:
generating a tokenized version of at least a portion of the file, the tokenized version including a set of before tokens based on the left set of contiguous characters, a set of after tokens based on the right set of contiguous characters, and a set of selected tokens based on the selected set of contiguous characters.
8. The non-transitory computer readable storage medium of claim 6 , wherein the first score is based on a number of non-matching rows between rows that matched the selected pattern set for each pair of rows nearest to one another.
9. The non-transitory computer readable storage medium of claim 6 , wherein the stored instructions cause the processor to further perform:
providing to the user interface of the client device, information describing the selected subset of the plurality of pattern sets, the user interface configured to receive a selection of a pattern set.
10. The non-transitory computer readable storage medium of claim 6 , wherein the stored instructions cause the computer processor to further perform:
generating a new file based on data obtained by parsing the file; and
providing the new file as input to a program configured to analyze data obtained from the file.
11. A computer system, comprising:
a computer processor; and
a non-transitory computer readable storage medium storing instructions that when executed by a processor cause the computer processor to perform:
retrieving from a file storage, a subset of a file stored in the file storage;
sending for display, via a user interface of a client device, the subset of the file;
receiving, via the user interface of the client device, a selection of a set of contiguous characters of the subset of the file;
determining a plurality of pattern sets, each pattern set comprising one or more patterns, each pattern describing one or more of a left set of contiguous characters occurring to the left of the selected set of contiguous characters, a right set of contiguous characters occurring to the right of the selected set of contiguous characters, and the selected set of contiguous characters;
for each of the plurality of pattern sets, assigning a plurality of scores comprising:
a first score indicative of regularity of the pattern set in rows of the file,
a second score indicative of a number of times the pattern set occurs in each row of the file, and
a third score indicative of uniformity of matches of the pattern set per row based on a deviation from an average number of matches per row of the file;
selecting a subset of the plurality of pattern sets based on the scores;
providing for display, via the user interface of the client device, results of an application of each of the selected subset of the plurality of pattern sets to each of at least one portion of the file;
receiving, via the user interface of the client device, a selection of one of the subset of the plurality of pattern sets for which results were provided for display; and
parsing the file using the selected pattern set.
12. The computer system of claim 11 , wherein the stored instructions cause the computer processor to further perform:
generating a tokenized version of at least a portion of the file, the tokenized version including a set of before tokens based on the left set of contiguous characters, a set of after tokens based on the right set of contiguous characters, and a set of selected tokens based on the selected set of contiguous characters.
13. The computer system of claim 11 , wherein the first score is based on a number of non-matching rows between rows that matched the selected pattern set for each pair of rows nearest to one another.
14. The computer system of claim 11 , wherein the stored instructions cause the processor to further perform:
providing to the user interface of the client device, information describing the selected subset of the plurality of pattern sets, the user interface configured to receive a selection of a pattern set.
15. The computer system of claim 11 , wherein the stored instructions cause the computer processor to further perform:
generating a new file based on data obtained by parsing the file; and
providing the new file as input to a program configured to analyze data obtained from the file.