Systems and methods for data transformation
Systems and methods for transforming data for a batch-based legacy software program into data suitable for processing by a streaming-based new software program. In some aspects, the system generates an information stream for multiple threads from batch information received from the legacy software program. The system processes the information stream for the multiple threads using the new software program. If there is a mismatch between outputs of the legacy and new software programs for at least one thread, the system adjusts a data model of the new software program to address the mismatch and generates an updated information stream for processing by the new software program.
1. A system for transforming data from a batch-based legacy software program for processing by a streaming-based new software program, the system comprising:
one or more processors; and
a non-transitory, computer-readable medium comprising instructions that, when executed by the one or more processors, cause operations comprising:
receiving batch information for a plurality of threads to be processed by a new software program, the batch information including for each thread input data and output data captured from a legacy software program during a particular time period;
generating, from the batch information, an information stream based on a data model for the new software program, the information stream including at least some of the input data for the plurality of threads, the data model for the new software program being different from a data model for the legacy software program;
processing, using the new software program, the information stream for the plurality of threads to generate corresponding output data for the particular time period;
comparing the output data from the new software program and the output data from the legacy software program for the particular time period;
based on the comparison, determining that at least one thread has a mismatch between the output data from the new software program and the output data from the legacy software program;
adjusting the data model for the new software program based on the mismatch for the at least one thread;
generating, from the batch information, an updated information stream based on the adjusted data model for the new software program; and
processing, using the new software program, the updated information stream for the plurality of threads to generate corresponding output data for the particular time period.
2. The system of claim 1 , wherein the data model for the new software program includes a streaming data model, and wherein the data model for the legacy software program includes a batch processing data model.
3. The system of claim 1 , wherein the at least some of the input data for the plurality of threads for the information stream includes a combination of the input data for the plurality of threads.
4. A method comprising:
receiving batch information for a plurality of threads to be processed by a new software program, the batch information including for each thread input data and output data captured from a legacy software program during a particular time period;
generating, from the batch information, an information stream based on a data model for the new software program, the information stream including at least some of the input data for the plurality of threads;
processing, using the new software program, the information stream for the plurality of threads to generate corresponding output data for the particular time period; and
based on determining that at least one thread has a mismatch between the output data from the new software program and the output data from the legacy software program, adjusting the data model for the new software program based on the mismatch for the at least one thread.
5. The method of claim 4 , wherein the at least some of the input data for the plurality of threads for the information stream includes a combination of the input data for the plurality of threads.
6. The method of claim 4 , wherein the data model for the new software program is different from a data model for the legacy software program.
7. The method of claim 6 , wherein the data model for the new software program includes a streaming data model, and wherein the data model for the legacy software program includes a batch processing data model.
8. The method of claim 4 , further comprising:
comparing the output data from the new software program and the output data from the legacy software program for the particular time period; and
based on the comparison, determining that at least one thread has a mismatch between the output data from the new software program and the output data from the legacy software program.
9. The method of claim 4 , further comprising:
generating, from the batch information, an updated information stream based on the adjusted data model for the new software program; and
processing, using the new software program, the updated information stream for the plurality of threads to generate corresponding output data for the particular time period.
10. A non-transitory, computer-readable medium comprising instructions that, when executed by one or more processors, cause operations comprising:
receiving batch information for a plurality of threads to be processed by a new software program, the batch information including for each thread input data and output data captured from a legacy software program during a particular time period;
generating, from the batch information, an information stream based on a data model for the new software program, the information stream including at least some of the input data for the plurality of threads;
processing, using the new software program, the information stream for the plurality of threads to generate corresponding output data for the particular time period; and
based on determining that at least one thread has a mismatch between the output data from the new software program and the output data from the legacy software program, adjusting the data model for the new software program based on the mismatch for the at least one thread.
11. The non-transitory, computer-readable medium of claim 10 , wherein the at least some of the input data for the plurality of threads for the information stream includes a combination of the input data for the plurality of threads.
12. The non-transitory, computer-readable medium of claim 10 , wherein the data model for the new software program is different from a data model for the legacy software program.
13. The non-transitory, computer-readable medium of claim 12 , wherein the data model for the new software program includes a streaming data model, and wherein the data model for the legacy software program includes a batch processing data model.
14. The non-transitory, computer-readable medium of claim 10 , further comprising:
comparing the output data from the new software program and the output data from the legacy software program for the particular time period; and
based on the comparison, determining that at least one thread has a mismatch between the output data from the new software program and the output data from the legacy software program.
15. The non-transitory, computer-readable medium of claim 10 , further comprising:
generating, from the batch information, an updated information stream based on the adjusted data model for the new software program; and
processing, using the new software program, the updated information stream for the plurality of threads to generate corresponding output data for the particular time period.