Systems and methods for importing data from data sources over a network while correcting and transforming the data
Systems and methods for importing data from data sources over a network while correcting and transforming the data are described. A data migration server receives data from various data sources. The data migration server repairs and transforms the received data before transmitting the data to a new data server. In some embodiments, the data migration server repairs data before formatting the data, and in other embodiments, the data migration server formats data before repairing the data. The data migration server also verifies the accuracy of the data, either by comparing the data against data requested and received from third-party servers or in reference to other data from the data sources.
1. A method of repairing and migrating data from a data source over a network, the method comprising:
receiving, by a server, a first data of a plurality of data stored in a first data source;
identifying, by the server, a relationship between the first data and one or more additional data received from one or more additional data sources;
grouping, by the server, based on the relationship, the first data and the one or more additional data;
adding, by the server, to the first data, an indicator of the grouping;
performing, by the server, a first transformation upon the first data to repair the first data based on at least one of the additional data in the grouping;
storing, in a memory element provided by the server, the repaired first data;
performing, by the server, a second transformation upon the repaired first data to format the repaired first data; and
transmitting, by the server, the repaired and formatted first data to a new data server.
2. The method of claim 1 , wherein the first and second transformations are determined according to a set of rules and wherein a type of repair performed in the first transformation is determined from a plurality of repair types.
3. The method of claim 1 , wherein the second transformation formats the repaired first data into an intermediate format.
4. The method of claim 3 , further comprising:
receiving, by the server, a second data of a plurality of data stored in a second data source;
performing, by the server, a third transformation different from the second transformation upon the second data to format the second data into the intermediate format.
5. The method of claim 4 , further comprising:
performing, by the server, a fourth transformation upon the first and second data in the intermediate format to format the first and second data into a final format; and
storing, in the memory element provided by the server, the first and second data in the final format.
6. The method of claim 1 , further comprising:
transmitting, by the server, information according to the data in the group to a second server to verify the accuracy of the data in the group;
receiving, by the server, a response from the second server indicating the accuracy of the data; and
correcting, by the server, at least one data in the group according to the response.
7. The method of claim 1 , further comprising:
retrieving, by the server, a second data from the plurality of data that is stored in the first data source;
performing, by the server, an operation upon the first and second data to determine the accuracy of the first data; and
correcting, by the server, the first data according to the determination.
8. The method of claim 1 , wherein adding the indicator of the grouping comprises adding, to the first data, a field with a value held in common by other members of the grouping.
9. A method of repairing and migrating data from a data source over a network, the method comprising:
receiving, by a server, a first data of a plurality of data stored in a data source;
performing, by the server, a first transformation upon the first data to format the first data;
storing, in a memory element provided by the server, the formatted first data;
identifying, by the server, a relationship between the formatted first data and one or more additional data received from one or more additional data sources;
grouping, by the server, based on the relationship, the formatted first data and the one or more additional data;
adding, by the server, to the formatted first data, an indicator of the grouping;
performing, by the server, a second transformation upon the formatted first data to repair the formatted first data based on at least one of the additional data in the grouping; and
transmitting, by the server, the formatted and repaired first data to a new data server.
10. The method of claim 9 , wherein the first and second transformations are determined according to a set of rules and wherein a type of repair performed in the second transformation is determined from a plurality of repair types.
11. The method of claim 9 , wherein the first transformation formats the first data into an intermediate format.
12. The method of claim 11 , further comprising:
receiving, by the server, a second data of a plurality of data stored in a second data source;
performing, by the server, a third transformation different from the first transformation upon the second data to format the second data into the intermediate format.
13. The method of claim 12 , further comprising:
performing, by the server, a fourth transformation upon the first and second data in the intermediate format to format the first and second data into a final format; and
storing, in the memory element provided by the server, the first and second data in the final format.
14. The method of claim 9 , further comprising:
transmitting, by the server, information according to the data in the group to a second server to verify the accuracy of the data in the group;
receiving, by the server, a response from the second server indicating the accuracy of the data; and
correcting, by the server, at least one data in the group according to the response.
15. The method of claim 9 , further comprising:
retrieving, by the server, a second data from the plurality of data that is stored in the first data source;
performing, by the server, an operation upon the first and second data to determine the accuracy of the first data; and
correcting, by the server, the first data according to the determination.
16. The method of claim 9 , wherein adding the indicator of the grouping comprises adding, to the first data, a field with a value held in common by other members of the grouping.
17. A computer implemented system for repairing and migrating data from a data source over a network, the system comprising:
a transceiver that receives a first data of a plurality of data stored in a data source and transmits repaired and formatted first data to a new data server;
a first transformation engine that performs a first transformation upon the first data to repair the first data, the first transformation comprising:
identifying a relationship between the first data and one or more additional data received from one or more additional data sources,
grouping, based on the relationship, the first data and the one or more additional data,
adding, to the first data, an indicator of the grouping, and
repairing the first data based on at least one of the additional data in the grouping;
a second transformation engine that performs a second transformation upon the repaired first data to format the repaired first data; and
a memory element that stores the repaired first data or the repaired and formatted first data.
18. The system of claim 17 , wherein the second transformation formats the repaired first data into an intermediate format.
19. The system of claim 18 , further comprising a third transformation engine that performs a third transformation different from the second transformation upon second data from a second data source to format the second data into the intermediate format.
20. The system of claim 19 , further comprising a fourth transformation engine that performs a fourth transformation upon the first and second data in the intermediate format to format the first and second data into a final format, wherein the first and second data in the final format are stored in the memory element.
21. The system of claim 17 , further comprising a processor that transmits information according to the data in the group to a server to verify the accuracy of the data in the group, receives a response from the second server indicating the accuracy of the data and corrects at least one data in the group according to the response.
22. The system of claim 17 , further comprising a processor that retrieves a second data from the plurality of data that is stored in the first data source, performs an operation upon the first and second data to determine the accuracy of the first data, and corrects the first data according to the determination.
23. The computer implemented system of claim 17 , wherein adding the indicator of the grouping comprises adding, to the first data, a field with a value held in common by other members of the grouping.
24. A computer implemented system for repairing and migrating data from a data source over a network, the system comprising:
a transceiver that receives a first data of a plurality of data stored in a data source and transmits formatted and repaired first data to a new data server;
a first transformation engine that performs a first transformation upon the first data to format the first data;
a second transformation engine that performs a second transformation upon the formatted first data to repair the formatted first data, the second transformation comprising:
identifying a relationship between the formatted first data and one or more additional data received from one or more additional data sources,
grouping, based on the relationship, the formatted first data and the one or more additional data,
adding, to the formatted first data, an indicator of the grouping, and
repairing the formatted first data based on at least one of the additional data in the grouping; and
a memory element that stores the formatted first data or the formatted and repaired first data.
25. The system of claim 24 , wherein the first transformation formats the first data into an intermediate format.
26. The system of claim 25 , further comprising a third transformation engine that performs a third transformation different from the first transformation upon second data from a second data source to format the second data into the intermediate format.
27. The system of claim 26 , further comprising a fourth transformation engine that performs a fourth transformation upon the first and second data in the intermediate format to format the first and second data into a final format, wherein the first and second data in the final format are stored in the memory element.
28. The system of claim 24 , further comprising a processor that transmits information according to the data in the group to a server to verify the accuracy of the data in the group, receives a response from the second server indicating the accuracy of the data and corrects at least one data in the group according to the response.
29. The system of claim 24 , further comprising a processor that retrieves a second data from the plurality of data that is stored in the first data source, performs an operation upon the first and second data to determine the accuracy of the first data, and corrects the first data according to the determination.
30. The computer implemented system of claim 24 , wherein adding the indicator of the grouping comprises adding, to the first data, a field with a value held in common by other members of the grouping.