Data unit test-based data management system
An improved unit test framework that validates large datasets generated by a data management system is described herein. Typical unit test frameworks validate functions. However, the improved unit test framework validates the underlying data. For example, after each step of a data transformation process implemented by the data management system, the data management system can execute a data unit test that loads data sets into memory, checks a set of preconditions, and applies unit test logic to the loaded data sets. In some embodiments, the data management system executes the data unit tests asynchronously with the data transformation processes and therefore do not interfere with the data transformation processes. Rather, the data management system generates and transmits a notification when any step of the data transformation process fails a particular data unit test.
1. A method comprising:
asynchronous from and at least partially concurrent to operation of a first data transformation process performed by one or more data processors and operation of a second data transformation process performed by the one or more data processors, by one or more computer hardware processors separate from the one or more data processors:
loading, by the one or more computer hardware processors, a data set into memory, wherein the data set is generated by the first data transformation process executed by the one or more data processors and forwarded by the first data transformation process to the second data transformation process executed by the one or more data processors;
applying, by the one or more computer hardware processors, a first condition to the data set to obtain a subset of the data set;
executing, by the one or more computer hardware processors, a test function comprising unit test logic on the subset of the data set, wherein the unit test logic includes a matcher; and
determining, by the one or more computer hardware processors, that the subset of the data set is valid in response to execution of the test function on the subset of the data set;
wherein, in response to the determination that the data set is valid, the first and second data transformation processes by the one or more data processors are uninterrupted.
2. The method of claim 1 , wherein the first condition corresponds to a first column in the data set.
3. The method of claim 2 , further comprising:
determining that a value of a variable of the data set satisfies a second condition; and
determining that the data set is valid in response to a determination that the data set satisfies the first condition and that the value of the variable satisfies the second condition.
4. The method of claim 1 , further comprising applying the test function to a second data set prior to execution of the test function on the subset of the data set as a previous step before the first data transformation process.
5. A system comprising:
one or more computer hardware processors;
one or more data processors that are separate from the one or more computer hardware processors; and
a storage device configured to store computer-executable instructions, the computer-executable instructions, when executed by the one or more computer hardware processors, cause the system to at least, asynchronous and at least partially concurrent to operation of a first data transformation process by the one or more data processors and operation of a second data transformation process by the one or more data processors:
load, by the one or more computer hardware processors, a data set into memory, where the data set is generated by the first data transformation process and forwarded by the first data transformation process to the second data transformation process;
apply, by the one or more computer hardware processors, a first condition to the data set to obtain a subset of the data set;
execute, by the one or more computer hardware processors, a test function comprising unit test logic on the subset of the data set, wherein the unit test logic includes a matcher; and
determine, by the one or more computer hardware processors, that the subset of the data set is valid in response to execution of the test function on the subset of the data set;
wherein, in response to the determination that the data set is valid, the first and second data transformation processes by the one or more data processors are uninterrupted.
6. The system of claim 5 , wherein a first data processor is configured to perform the first data transformation process.
7. The system of claim 6 , wherein a second data processor separate from the first data processor is configured to perform the second data transformation process.
8. The system of claim 5 , wherein the first condition corresponds to a first column in the data set.
9. The system of claim 5 , wherein the computer-executable instructions, when executed by the one or more computer hardware processors, further cause the system to at least:
determine that a value of a variable of the data set satisfies a second condition; and
determine that the data set is valid in response to a determination that the data set satisfies the first condition and that the value of the variable satisfies the second condition.
10. One or more non-transitory, computer-readable storage media storing computer-executable instructions, which if performed by one or more computer hardware processors that are separate from one or more data processors, cause the one or more computer hardware processors to at least:
asynchronous from and at least partially concurrent to operation of a first data transformation process performed by the one or more data processors and operation of a second data transformation process performed by the one or more data processors,
load, by the one or more computer hardware processors, a data set into memory, wherein the data set is generated by the first data transformation process and forwarded by the first data transformation process to the second data transformation process;
apply, by the one or more computer hardware processors, a first condition to the data set to obtain a subset of the data set;
execute, by the one or more computer hardware processors, a test function comprising unit test logic on the subset of the data set, wherein the unit test logic includes a matcher;
determine, by the one or more computer hardware processors, that the subset of the data set is valid in response to execution of the test function on the subset of the data set;
wherein, in response to the determination that the data set is valid, the first and second data transformation processes by the one or more data processors are uninterrupted.
11. The one or more non-transitory, computer-readable storage media of claim 10 , wherein the first condition corresponds to a first column in the data set.
12. The one or more non-transitory, computer-readable storage media of claim 10 , wherein the computer-executable instructions, when performed by the one or more computer hardware processors, further cause the one or more computer hardware processors to at least:
determine that a value of a variable of the data set satisfies a second condition; and
determine that the data set is valid in response to a determination that the data set satisfies the first condition and that the value of the variable satisfies the second condition.
13. Method comprising:
asynchronous from and at least partially concurrent to a first data processing operation performed by one or more data processors and a second data processing operation performed by the one or more data processors,
by one or more computer hardware processors separate from the one or more data processors,
loading, by the one or more computer hardware processors, a data set generated by the first data processing operation executed by the one or more data processors and forwarded by the first data processing operation to the second data processing operation executed by the one or more data processors;
applying, by the one or more computer hardware processors, a first condition to the data set to obtain a subset of the data set;
executing, by the one or more computer hardware processors, a test function comprising unit test logic on the subset of the data set, wherein the unit test logic includes a matcher; and
determining, by the one or more computer hardware processors, that the subset of the data set is valid in response to execution of the test function on the subset of the data set;
wherein, in response to the determination that the data set is valid, the first and second data processing operations by the one or more data processors are uninterrupted.