Validating data integrations using a secondary data store
A data integration system provides a secondary validation data store that may be used as a staging area for testing and validating data integrations for a set of structured data. Ontology data that defines a schema for integrated data may be modified and tweaked. The secondary validation data store is more efficient than a production system for such schema modifications. Once the ontology data is satisfactory, the structured data and the ontology data may be loaded into a production system.
1. A method, comprising:
receiving a first set of data comprising a plurality of unstructured data records;
receiving a schema that describes characteristics of the plurality of unstructured data records and relationships between one or more data records of the plurality of unstructured data records;
generating, based on the first set of data and the schema, a first plurality of structured data records, wherein the first plurality of structured data records organizes the plurality of unstructured data records based on the schema;
causing to be displayed, at a client computing device, the first plurality of structured data records;
receiving schema modification instructions;
in response to receiving the schema modification instructions, modifying the schema based on the schema modification instructions;
generating, based on the modified schema and the first set of data, a second plurality of structured data records, wherein the second plurality of structured data records organizes the plurality of unstructured data records based on modified schema;
in response to receiving the schema modification instructions, causing to be displayed, at the client computing device, the second plurality of structured data records;
wherein the method is performed using one or more processors.
2. The method of claim 1 , further comprising:
extracting a second set of data from one or more data sources; and
applying a data modification rule to the second set of data to generate, at least in part, the first set of data.
3. The method of claim 2 , wherein the data modification rule comprises a rule for removing a field from the second set of data.
4. The method of claim 2 , wherein the data modification rule comprises a rule for normalizing a value for a field from the second set of data.
5. The method of claim 2 , wherein the data modification rule comprises a rule for joining the second set of data with a third set of data.
6. The method of claim 2 , wherein the second set of data is a file of comma separated values.
7. The method of claim 1 , wherein the first set of data is a set of data in a JavaScript Object Notation (JSON) format.
8. The method of claim 1 , further comprising:
in response to receiving schema approval input, causing the first set of data and the schema to be transmitted to a second system for processing.
9. One or more non-transitory computer-readable media storing instructions, which when executed by one or more hardware processors cause:
receiving a first set of data comprising a plurality of unstructured data records;
receiving a schema that describes characteristics of the plurality of unstructured data records and relationships between one or more data records of the plurality of unstructured data records;
storing a first set of data using a first system, wherein the first system comprises a schema that describes objects and properties in the first set of data;
generating, based on the first set of data and the schema, a first plurality of structured data records, wherein the first plurality of structured data records organizes the plurality of unstructured data records based on the schema;
causing to be displayed, at a client computing device, the first plurality of structured data records;
receiving schema modification instructions;
in response to receiving the schema modification instructions, modifying the schema based on the schema modification instructions;
generating, based on the modified schema and the first set of data, a second plurality of structured data records, wherein the second plurality of structured data records organizes the plurality of unstructured data records based on modified schema;
in response to receiving the schema modification instructions, causing to be displayed, at the client computing device, the second plurality of structured data records.
10. The one or more non-transitory computer-readable media of claim 9 , the instructions further comprising instructions, which when executed by one or more hardware processors cause:
extracting a second set of data from one or more data sources; and
applying a data modification rule to the second set of data to generate, at least in part, the first set of data.
11. The one or more non-transitory computer-readable media of claim 10 , wherein the data modification rule comprises a rule for removing a field from the second set of data.
12. The one or more non-transitory computer-readable media of claim 10 , wherein the data modification rule comprises a rule for normalizing a value for a field from the second set of data.
13. The one or more non-transitory computer-readable media of claim 10 , wherein the data modification rule comprises a rule for joining the second set of data with a third set of data.
14. The one or more non-transitory computer-readable media of claim 10 , wherein the second set of data is a file of comma separated values.
15. The one or more non-transitory computer-readable media of claim 9 , wherein the first set of data is a set of data in a JavaScript Object Notation (JSON) format.
16. The one or more non-transitory computer-readable media of claim 9 , the instructions further comprising instructions, which when executed by one or more hardware processors cause:
in response to receiving schema approval input, causing the first set of data and the schema to be transmitted to a second system for processing.
17. A method comprising:
receiving a first set of data, wherein the first set of data comprises a plurality of data fields;
analyzing a particular data field of the plurality of data fields to determine a particular property type of the data field;
using the particular property type, at least in part, to generate a database schema;
storing the first set of data in a secondary data store using the database schema;
wherein the method is performed using one or more processors.
18. The method of claim 17 , further comprising:
indexing the secondary data store asynchronously.
19. The method of claim 17 , wherein the first set of data is formatted in a JavaScript Object Notation (JSON) format.
20. The method of claim 17 , wherein the secondary data store does not store logging data.