Phased importing of objects
In general, the subject matter described in this specification can be embodied in methods, systems, and program products for importing objects. A computer system processes structured electronic documents that define objects to be imported into a database in multiple phases. In a first phase, the computer system analyzes the structured electronic documents to determine whether the objects to be imported are related to one or more other objects that have yet to be stored in the database. If so, the computer system generates contextual information that defines the relationships between the objects. In a second phase, the computer system establishes relationships in the database between related objects using the contextual information.
1. A computer-implemented method for importing objects, the method comprising:
identifying, using a computer system, a plurality of structured electronic documents, each of which defines an object to be imported into a database;
processing, using the computer system, the plurality of structured electronic documents in a first phase that comprises, for each of the structured electronic documents:
analyzing the structured electronic document to obtain an identifier associated with the object to be imported, the identifier being incompatible for use as an index in the database;
generating a converted identifier from the identifier associated with the object to be imported, the converted identifier being compatible for use as an index in the database;
analyzing the structured electronic document to determine whether the object to be imported is related to one or more other objects defined by the plurality of structured electronic documents that have yet to be stored in the database, and if so, generating contextual information that defines a relationship between the object to be imported and the one or more other objects, the contextual information including the converted identifier and a relational identifier for each of the one or more other objects; and
storing the object defined by the structured electronic document in the database; and
processing, using the computer system, the contextual information in a second phase that comprises at least establishing relationships in the database between related objects using the contextual information;
wherein the processing of the first phase either: i) precedes the processing of the second phase; or ii) occurs in parallel with the processing of the second phase.
2. The method of claim 1 , further comprising, during the first phase processing, determining that the object to be imported is related to an object that is stored in the database, and establishing a relationship in the database between the object to be imported and the object that is stored in the database.
3. The method of claim 1 , wherein generating the converted identifier comprises hashing the identifier associated with the object to be imported.
4. The method of claim 1 , further comprising simulating at least a portion of the first phase processing and the second phase processing without storing the objects in the database.
5. The method of claim 4 , wherein a subset of the plurality of structured electronic documents is used during the simulation, the subset being fewer than all of the plurality of structured electronic documents.
6. A computer-readable storage device storing instructions that, when executed by one or more processing devices, perform operations comprising:
identifying, using a computer system, a plurality of structured electronic documents, each of which defines an object to be imported into a database;
processing, using the computer system, the plurality of structured electronic documents in a first phase that comprises, for each of the structured electronic documents:
analyzing the structured electronic document to obtain an identifier associated with the object to be imported, the identifier being incompatible for use as an index in the database;
generating a converted identifier from the identifier associated with the object to be imported, the converted identifier being compatible for use as an index in the database;
analyzing the structured electronic document to determine whether the object to be imported is related to one or more other objects defined by the plurality of structured electronic documents that have yet to be stored in the database, and if so, generating contextual information that defines a relationship between the object to be imported and the one or more other objects, the contextual information including the converted identifier and a relational identifier for each of the one or more other objects; and
storing the object defined by the structured electronic document in the database; and
processing, using the computer system, the contextual information in a second phase that comprises at least establishing relationships in the database between related objects using the contextual information;
wherein the processing of the first phase either: i) precedes the processing of the second phase; or ii) occurs in parallel with the processing of the second phase.
7. The computer-readable storage device of claim 6 , wherein the operations further comprise, during the first phase processing, determining that the object to be imported is related to an object that is stored in the database, and establishing a relationship in the database between the object to be imported and the object that is stored in the database.
8. The computer-readable storage device of claim 6 , wherein generating the converted identifier comprises hashing the identifier associated with the object to be imported.
9. The computer-readable storage device of claim 6 , wherein the operations further comprise simulating at least a portion of the first phase processing and the second phase processing without storing the objects in the database.
10. The computer-readable storage device of claim 9 , wherein a subset of the plurality of structured electronic documents is used during the simulation, the subset being fewer than all of the plurality of structured electronic documents.
11. A system comprising:
one or more programmable processors; and
a computer-readable storage device storing instructions that, when executed by the one or more programmable processors, perform operations comprising:
identifying, using a computer system, a plurality of structured electronic documents, each of which defines an object to be imported into a database;
processing, using the computer system, the plurality of structured electronic documents in a first phase that comprises, for each of the structured electronic documents:
analyzing the structured electronic document to obtain an identifier associated with the object to be imported, the identifier being incompatible for use as an index in the database;
generating a converted identifier from the identifier associated with the object to be imported, the converted identifier being compatible for use as an index in the database;
analyzing the structured electronic document to determine whether the object to be imported is related to one or more other objects defined by the plurality of structured electronic documents that have yet to be stored in the database, and if so, generating contextual information that defines a relationship between the object to be imported and the one or more other objects, the contextual information including the converted identifier and a relational identifier for each of the one or more other objects; and
storing the object defined by the structured electronic document in the database; and
processing, using the computer system, the contextual information in a second phase that comprises at least establishing relationships in the database between related objects using the contextual information;
wherein the processing of the first phase either: i) precedes the processing of the second phase; or ii) occurs in parallel with the processing of the second phase.
12. The system of claim 11 , wherein the operations further comprise, during the first phase processing, determining that the object to be imported is related to an object that is stored in the database, and establishing a relationship in the database between the object to be imported and the object that is stored in the database.
13. The system of claim 11 , wherein generating the converted identifier comprises hashing the identifier associated with the object to be imported.
14. The system of claim 11 , wherein the operations further comprise simulating at least a portion of the first phase processing and the second phase processing without storing the objects in the database.
15. The system of claim 14 , wherein a subset of the plurality of structured electronic documents is used during the simulation, the subset being fewer than all of the plurality of structured electronic documents.