Enhanced data collection techniques

1. A data collection system, comprising:

a plurality of data sources that produce data, wherein at least some of the plurality of data sources produce data in different formats;

a plurality of files, wherein each of the plurality of files is associated with a different size file line;

a plurality of listeners stored in a non-transitory computer readable medium that, when executed by a computer processor, receive the data from the plurality of data sources, identify the data, parse the data, convert the data into a generic format, validate the converted data, and write the converted data to the plurality of files based on a line size of the data, wherein the plurality of listeners are implemented using a script programming language;

a plurality of loaders stored in a non-transitory computer readable medium that, when executed by a computer processor, load the converted data into a database, wherein the plurality of loaders are implemented using an object oriented programming language, and wherein each of the plurality of loaders loads file lines of a certain size from one of a plurality of files; and

a plurality of JAVA virtual machines, wherein each of the plurality of loaders is executed in a different one of the plurality of JAVA virtual machines, and wherein each of the plurality of JAVA virtual machines is tuned differently based on the certain size file lines being loaded by the corresponding loader of the plurality of loaders operating therein.

2. The data collection system of claim 1 , wherein the script programming language is Perl.

3. The data collection system of claim 1 , wherein the object oriented programming language is JAVA.

4. The data collection system of claim 1 , wherein the generic format is an extensible markup language based format.

5. A computer-implemented method for collecting data, comprising:

receiving, by a plurality of listeners implemented using a script programming language, stored on a non-transitory computer readable medium, and executed by a computer processor, data from a plurality of data sources, wherein the data comprises different formats;

identifying, by the plurality of listeners, the data;

parsing, by the plurality of listeners, the data;

converting, by the plurality of listeners, the data into extensible markup language data sets;

validating, by the plurality of listeners, the extensible markup language data sets;

writing, by the plurality of listeners the extensible markup language data sets to a plurality of files based on a line size of the data, wherein each of the plurality of files is associated with a different size file line; and

loading, by a plurality of loaders implemented using an object oriented programming language, stored on a non-transitory computer readable medium, and executed by a computer processor, the extensible markup language data sets into a database, wherein each of the plurality of loaders loads file lines of a certain size from one of the plurality of files, wherein each of the plurality of loaders operate in a different one of a plurality of JAVA virtual machines, and wherein each of the plurality of JAVA virtual machines is tuned differently based on the certain size file lines being loaded by the corresponding loader of the plurality of loaders operating therein.

6. The computer-implemented method of claim 5 , wherein the script programming language is Perl.

7. The computer-implemented method of claim 5 , wherein the object oriented programming language is JAVA.

8. A data collection system, comprising:

a plurality of data sources that produce data, wherein at least some of the plurality of data sources produce data in different formats;

a plurality of files, wherein each of the plurality of files is associated with a different size file line;

a plurality of listeners stored in a non-transitory computer readable medium that, when executed by a computer processor, receive the data from the plurality of data sources, identify the data, parse the data, convert the data into extensible markup language data sets, validate the extensible markup language data sets, and write the extensible markup language data sets to the plurality of files based on a line size of the data, wherein the plurality of listeners are implemented using a script programming language;

a plurality of loaders stored in a non-transitory computer readable medium that, when executed by a computer processor, load the extensible markup language data sets into a database, wherein each of the plurality of loaders loads file lines of a certain size from one of the plurality of files, wherein the plurality of loaders are implemented using an object oriented programming language; and

a plurality of JAVA virtual machines, wherein each of the plurality of loaders operates in a different one of the plurality of JAVA virtual machines, and wherein each of the plurality of JAVA virtual machines is tuned differently based on the certain size file lines being loaded by the corresponding loader of the plurality of loaders operating therein.

9. The data collection system of claim 8 , wherein each the plurality of files is associated with a different range of file line sizes.

10. The data collection system of claim 9 , wherein each of the plurality of loaders loads file lines of a certain range of sizes from one of the plurality of files.

11. The data collection system of claim 8 , wherein each of the plurality of JAVA virtual machines is tuned differently by varying at least one of a heap size, a number of threads, and a number of data connections for each of the plurality of JAVA virtual machines.