Document architecture with efficient storage
A document architecture system includes an intelligent document processing engine that can receive raw data streamed from external data source(s), analyze the raw data to identify triggering events, apply rules to the triggering events identified, break down the raw data accordingly, and generating messages in an internal format to represent the pieces of data without any formatting or layout information. The messages can be augmented with custom metadata and stored as fragments that reference, but that do not physically incorporate, the messages. The stored fragments can be used by the document process engine to intelligently render documents and versions thereof. Some embodiments store only unique fragments, resulting in a significant reduction in storage requirements and an extremely efficient way of processing huge amounts of data (e.g., millions of documents) for document production. The messages and fragments are in internal formats under control of the document architecture system.
1. A method for efficient storage of document fragments, comprising:
receiving, by a document processing engine operating in a document architecture, raw data from a data source;
analyzing, by the document processing engine, the raw data to identify triggering events;
applying, by the document processing engine, rules to the triggering events identified in the raw data;
based on the analyzing and the applying by the document processing engine, breaking down the raw data into pieces of data and, for each piece of data, generating a message having a format-independent data structure, the message containing system-defined properties describing the each piece of data;
persisting, by the document processing engine, the message in a first data storage;
generating, by the document processing engine, a fragment that references the message in the first data storage, the fragment having a presentation-independent format;
determining, by the document processing engine, whether the fragment is unique among previously generated fragments; and
storing, by the document processing engine, the fragment in the second data storage responsive to a determination that the fragment is unique.
2. The method according to claim 1 , wherein the determining whether the fragment is unique comprises:
applying a hash function to the fragment to generate a hash; and
comparing the hash with hashes stored in a hash table or data structure to determine whether an identical fragment already exists in the second data storage.
3. The method according to claim 1 , wherein the generating the fragment comprises breaking a document definition file apart into fragments.
4. The method according to claim 3 , wherein the document definition file has an associated metadata model that describes how to render a document, or a version or versions thereof, from the fragments.
5. The method according to claim 4 , wherein the metadata model is accessible through a metadata model editor, a metadata model application programming interface, or a tenant-specific service.
6. The method according to claim 4 , wherein the metadata model is accessible at design time for defining a rendered document type or at runtime for mapping input data to typed metadata.
7. The method according to claim 4 , wherein the metadata model comprises an inheritance pattern and an aggregation pattern for describing custom metadata types.
8. A system for efficient storage of document fragments, comprising:
a processor;
a non-transitory computer-readable medium; and
stored instructions translatable by the processor for:
receiving raw data from a data source;
analyzing the raw data to identify triggering events;
applying rules to the triggering events identified in the raw data;
based on the analyzing and the applying, breaking down the raw data into pieces of data and, for each piece of data, generating a message having a format-independent data structure, the message containing system-defined properties describing the each piece of data;
persisting the message in a first data storage;
generating a fragment that references the message in the first data storage, the fragment having a presentation-independent format;
determining whether the fragment is unique among previously generated fragments; and
storing the fragment in the second data storage responsive to a determination that the fragment is unique.
9. The system of claim 8 , wherein the determining whether the fragment is unique comprises:
applying a hash function to the fragment to generate a hash; and
comparing the hash with hashes stored in a hash table or data structure to determine whether an identical fragment already exists in the second data storage.
10. The system of claim 8 , wherein the generating the fragment comprises breaking a document definition file apart into fragments.
11. The system of claim 10 , wherein the document definition file has an associated metadata model that describes how to render a document, or a version or versions thereof, from the fragments.
12. The system of claim 11 , wherein the metadata model is accessible through a metadata model editor, a metadata model application programming interface, or a tenant-specific service.
13. The system of claim 11 , wherein the metadata model is accessible at design time for defining a rendered document type or at runtime for mapping input data to typed metadata.
14. The system of claim 11 , wherein the metadata model comprises an inheritance pattern and an aggregation pattern for describing custom metadata types.
15. A computer program product for efficient storage of document fragments, the computer program product comprising a non-transitory computer-readable medium storing instructions translatable by a processor for:
receiving raw data from a data source;
analyzing the raw data to identify triggering events;
applying rules to the triggering events identified in the raw data;
based on the analyzing and the applying, breaking down the raw data into pieces of data and, for each piece of data, generating a message having a format-independent data structure, the message containing system-defined properties describing the each piece of data;
persisting the message in a first data storage;
generating a fragment that references the message in the first data storage, the fragment having a presentation-independent format;
determining whether the fragment is unique among previously generated fragments; and
storing the fragment in the second data storage responsive to a determination that the fragment is unique.
16. The computer program product of claim 15 , wherein the determining whether the fragment is unique comprises:
applying a hash function to the fragment to generate a hash; and
comparing the hash with hashes stored in a hash table or data structure to determine whether an identical fragment already exists in the second data storage.
17. The computer program product of claim 15 , wherein the generating the fragment comprises breaking a document definition file apart into fragments.
18. The computer program product of claim 17 , wherein the document definition file has an associated metadata model that describes how to render a document, or a version or versions thereof, from the fragments.
19. The computer program product of claim 18 , wherein the metadata model is accessible through a metadata model editor, a metadata model application programming interface, or a tenant-specific service.
20. The computer program product of claim 18 , wherein the metadata model is accessible at design time for defining a rendered document type or at runtime for mapping input data to typed metadata and wherein the metadata model comprises an inheritance pattern and an aggregation pattern for describing custom metadata types.