Automated data intake system

1. A non-transitory computer-readable medium having stored thereon computer-readable instructions that when executed by a computing device control the computing device to:

automatically read a registry file,

wherein the registry file includes a plurality of filename parameters,

wherein each filename parameter of the plurality of filename parameters identifies a matching filename pattern,

wherein an extract script indicator and a read file indicator are associated with each filename parameter,

wherein the extract script indicator indicates an extract script for a file having a filename that matches the matching filename pattern,

wherein the read file indicator indicates how to read the file having the filename that matches the matching filename pattern;

automatically determine whether unprocessed source data is stored in a predefined directory; and

based upon determining that unprocessed source data is stored in the predefined directory,

automatically select a source file from the unprocessed source data;

automatically select one parameter of the plurality of filename parameters read from the registry file by matching a filename of the selected source file to the matching filename pattern of the one parameter;

automatically select the extract script based on the extract script indicator associated with the selected one parameter;

automatically read data from the selected source file using the selected extract script and using the read file indicator associated with the selected one parameter; and

automatically output the read data to a different file than the source file and in a different format than a format used in the source file.

2. The non-transitory computer-readable medium of claim 1 , wherein the extract script for a first file having a first filename that matches a first matching filename pattern is different from the extract script for a second file having a second filename that matches a second matching filename pattern.

3. The non-transitory computer-readable medium of claim 1 , wherein the extract script indicator associated with the selected one parameter is blank indicating that the selected extract script is a default extractor.

4. The non-transitory computer-readable medium of claim 1 , wherein the registry file further includes a second-level filename parameter associated with each filename parameter of the plurality of filename parameters,

wherein each second-level filename parameter identifies a second matching filename pattern,

wherein the extract script indicator and the read file indicator are associated with the second-level filename parameter associated with each filename parameter of the plurality of filename parameters.

5. The non-transitory computer-readable medium of claim 1 , wherein the computer-readable instructions further control the computing device to:

determine whether a trigger file for the selected source file is stored in the predefined directory; and

based upon determining that the trigger file for the selected source file is stored in the predefined directory,

read the trigger file to define a data integrity test value for the selected source file, wherein the data integrity test value is selected from the group consisting of a number of records, a checksum value, and a modification date;

perform an integrity test by comparing the defined data integrity test value to a test value determined from the read data;

determine whether the performed integrity test fails; and

based upon determining that the performed integrity test fails, send an error notification message to a predefined address.

6. The non-transitory computer-readable medium of claim 1 , wherein the computer-readable instructions further control the computing device to:

determine whether a data layout description file for the selected source file is stored in the predefined directory; and

based upon determining that the data layout description file for the selected source file is stored in the predefined directory,

read the data layout description file to define a data quality test for the selected source file;

perform the defined data quality test on the read data;

determine whether the performed, defined data quality test fails; and

based upon determining that the performed, defined data quality test fails, send an error notification message to a predefined address.

7. The non-transitory computer-readable medium of claim 6 , wherein a first data quality test is defined for a first column of the read data, and a second data quality test is defined for a second column of the read data, wherein the first data quality test is different from the second data quality test.

8. The non-transitory computer-readable medium of claim 6 , wherein the data quality test is selected from the group consisting of a test on a range of values, a test for a null value, a test for a predefined data type, a test for only digits, a test for only characters, a test for only alphanumeric characters, a test for only values in a list, and a test for only values in a table.

9. The non-transitory computer-readable medium of claim 1 , wherein, before automatically reading the data, the computer-readable instructions further control the computing device to automatically decompress the selected source file based on a compression indicator associated with the selected one parameter.

10. The non-transitory computer-readable medium of claim 1 , wherein, before automatically reading the data, the computer-readable instructions further control the computing device to automatically decrypt the selected source file based on an encryption indicator associated with the selected one parameter.

11. The non-transitory computer-readable medium of claim 1 , wherein the computer-readable instructions further control the computing device to:

automatically select a pre-process program based on a pre-process program indicator associated with the selected one parameter; and

automatically execute the selected pre-process program with the selected source file before automatically reading the data from the selected source file.

12. The non-transitory computer-readable medium of claim 1 , wherein the computer-readable instructions further control the computing device to:

automatically select a post-process program based on a post-process program indicator associated with the selected one parameter; and

automatically execute the selected post-process program with the selected source file after automatically reading the data from the selected source file.

13. The non-transitory computer-readable medium of claim 1 , wherein the read file indicator is selected from the group consisting of a file type indicator, a delimiter type indicator, a first observation indicator, an encryption indicator, a compression indicator, a partitioned file indicator, a field type indicator, a field format indicator, and a field length indicator.

14. The non-transitory computer-readable medium of claim 1 , wherein the computer-readable instructions further control the computing device to:

determine whether a data layout description file for the selected source file is stored in the predefined directory; and

based upon determining that the data layout description file for the selected source file is stored in the predefined directory, read a data output description from the data layout description file, wherein the different format is based at least partially on the read data output description.

15. The non-transitory computer-readable medium of claim 14 , wherein the data output description includes at least one of a table name, a field order indicator, a column label indicator, a field type indicator, a field format indicator, and a field length indicator.

16. The non-transitory computer-readable medium of claim 1 , wherein the computer-readable instructions further control the computing device to:

automatically read a second registry file before automatically selecting the extract script when the filename of the selected source file does not match any matching filename pattern of the plurality of filename parameters,

wherein the second registry file includes a second plurality of filename parameters,

wherein each filename parameter of the second plurality of filename parameters identifies a second matching filename pattern,

wherein a second extract script indicator and a second read file indicator are associated with each filename parameter of the second plurality of filename parameters,

wherein the second extract script indicator indicates an extract script for a file having a filename that matches the second matching filename pattern,

wherein the second read file indicator indicates how to read the file having the filename that matches the second matching filename pattern; and

automatically select the one parameter of the second plurality of filename parameters read from the second registry file by matching the filename of the selected source file to the second matching filename pattern of the one parameter.

17. The non-transitory computer-readable medium of claim 1 , wherein the computer-readable instructions further control the computing device to:

determine whether the selected source file is expected based on a time parameter read from the registry file in association with the selected one parameter after automatically selecting the one parameter of the plurality of filename parameters; and

based upon determining that the selected source file is not expected based on the time parameter, send an error notification message to a predefined address.

18. The non-transitory computer-readable medium of claim 1 , wherein the computer-readable instructions further control the computing device to,

when unprocessed source data is not stored in the predefined directory, determine whether an expected receipt time has expired; and

based upon determining that the expected receipt time has expired, send an error notification message to a predefined address.

19. A computing device comprising:

a processor; and

a non-transitory computer-readable medium operably coupled to the processor, the computer-readable medium having computer-readable instructions stored thereon that, when executed by the processor, control the computing device to automatically read a registry file,

wherein the registry file includes a plurality of filename parameters,

wherein each filename parameter of the plurality of filename parameters identifies a matching filename pattern,

wherein an extract script indicator and a read file indicator are associated with each filename parameter,

wherein the extract script indicator indicates an extract script for a file having a filename that matches the matching filename pattern,

wherein the read file indicator indicates how to read the file having the filename that matches the matching filename pattern;

automatically determine whether unprocessed source data is stored in a predefined directory; and

based upon determining that unprocessed source data is stored in the predefined directory,

automatically select a source file from the unprocessed source data;

automatically select one parameter of the plurality of filename parameters read from the registry file by matching a filename of the selected source file to the matching filename pattern of the one parameter;

automatically select the extract script based on the extract script indicator associated with the selected one parameter;

automatically read data from the selected source file using the selected extract script and using the read file indicator associated with the selected one parameter; and

automatically output the read data to a different file than the source file and in a different format than a format used in the source file.

20. A method of automatically processing data received from a first computing device for use on a second computing device, the method comprising:

automatically reading, by a first computing device, a registry file,

wherein the registry file includes a plurality of filename parameters,

wherein each filename parameter of the plurality of filename parameters identifies a matching filename pattern,

wherein an extract script indicator and a read file indicator are associated with each filename parameter,

wherein the extract script indicator indicates an extract script for a file having a filename that matches the matching filename pattern,

wherein the read file indicator indicates how to read the file having the filename that matches the matching filename pattern;

automatically determining, by the first computing device, whether unprocessed source data is stored in a predefined directory; and

based upon determining that unprocessed source data is stored in the predefined directory,

automatically selecting, by the first computing device, a source file from the unprocessed source data;

automatically selecting, by the first computing device, one parameter of the plurality of filename parameters read from the registry file by matching a filename of the selected source file to the matching filename pattern of the one parameter;

automatically selecting, by the first computing device, the extract script based on the extract script indicator associated with the selected one parameter;

automatically reading, by the first computing device, data from the selected source file using the selected extract script and using the read file indicator associated with the selected one parameter; and

automatically outputting, by the first computing device, the read data to a different file than the source file and in a different format than a format used in the source file.