Document architecture with efficient storage
A document architecture system includes an intelligent document processing engine that can receive raw data streamed from external data source(s), analyze the raw data to identify triggering events, apply rules to the triggering events identified, break down the raw data accordingly, and generating messages in an internal format to represent the pieces of data without any formatting or layout information. The messages can be augmented with custom metadata and stored as fragments that reference, but that do not physically incorporate, the messages. The stored fragments can be used by the document process engine to intelligently render documents and versions thereof. Some embodiments store only unique fragments, resulting in a significant reduction in storage requirements and an extremely efficient way of processing huge amounts of data (e.g., millions of documents) for document production. The messages and fragments are in internal formats under control of the document architecture system.
1. A method, comprising:
processing, by a computer, raw data streamed from a data source into structured messages containing metadata describing patterns in the raw data, the structure messages constructed on a format-independent data structure;
generating, by the computer, fragments having a presentation-independent format, each of the fragments referencing one of the structure messages; and
for each respective fragment of the fragments:
determining whether the respective fragment matches any fragment stored in a fragment storage; and
storing the respective fragment in the fragment storage only if the respective fragment does not match any fragment stored in the fragment storage.
2. The method according to claim 1 , wherein determining whether the respective fragment matches any fragment stored in the fragment storage comprises:
applying a hash function to the respective fragment so as to generate a hash; and
comparing the hash with hashes stored in a tracking table or data structure, the hashes corresponding to unique fragments stored in the fragment storage.
3. The method according to claim 1 , wherein determining whether the respective fragment matches any fragment stored in the fragment storage comprises:
applying a cryptographic hash function to the respective fragment so as to generate a checksum; and
comparing the checksum for the respective fragment with checksums of unique fragments stored in the fragment storage.
4. The method according to claim 1 , further comprising:
prior to storing a unique fragment or a set of unique fragments to the fragment storage, applying a cryptographic hash function to the unique fragment or the set of unique fragments so as to generate a first signature; and
storing the first signature with the unique fragment or the set of unique fragments.
5. The method according to claim 4 , further comprising:
retrieving the unique fragment or the set of unique fragments from the fragment storage;
applying the cryptographic hash function to the unique fragment or the set of unique fragments retrieved from the fragment storage so as to generate a second signature;
comparing the first signature with the second signature; and
based on the comparing, determining whether the unique fragment or the set of unique fragments retrieved from the fragment storage has been altered.
6. The method according to claim 1 , further comprising:
retrieving a set of fragments from the fragment storage;
assembling a document from the set of fragments based on instructions from a document definition associated with the document;
processing the document based on custom configuration used by a post formatting component so as to produce a final document, the post formatting component determining a correct final format, a correct final size, and a correct final resolution for an output channel; and
passing the final document to a driver or renderer for rendering the final document into a rendition of the document, the rendition having the correct final format, the correct final size, and the correct final resolution for the output channel.
7. The method according to claim 6 , wherein the correct final format comprises a device-specific format for the output channel.
8. A system, comprising:
a processor;
a non-transitory computer-readable medium; and
instructions stored on the non-transitory computer-readable medium and translatable by the processor for:
processing raw data streamed from a data source into structured messages containing metadata describing patterns in the raw data, the structure messages constructed on a format-independent data structure;
generating fragments having a presentation-independent format, each of the fragments referencing one of the structure messages; and
for each respective fragment of the fragments:
determining whether the respective fragment matches any fragment stored in a fragment storage; and
storing the respective fragment in the fragment storage only if the respective fragment does not match any fragment stored in the fragment storage.
9. The system of claim 8 , wherein determining whether the respective fragment matches any fragment stored in the fragment storage comprises:
applying a hash function to the respective fragment so as to generate a hash; and
comparing the hash with hashes stored in a tracking table or data structure, the hashes corresponding to unique fragments stored in the fragment storage.
10. The system of claim 8 , wherein determining whether the respective fragment matches any fragment stored in the fragment storage comprises:
applying a cryptographic hash function to the respective fragment so as to generate a checksum; and
comparing the checksum for the respective fragment with checksums of unique fragments stored in the fragment storage.
11. The system of claim 8 , wherein the instructions are further translatable by the processor for:
prior to storing a unique fragment or a set of unique fragments to the fragment storage, applying a cryptographic hash function to the unique fragment or the set of unique fragments so as to generate a first signature; and
storing the first signature with the unique fragment or the set of unique fragments.
12. The system of claim 11 , wherein the instructions are further translatable by the processor for:
retrieving the unique fragment or the set of unique fragments from the fragment storage;
applying the cryptographic hash function to the unique fragment or the set of unique fragments retrieved from the fragment storage so as to generate a second signature;
comparing the first signature with the second signature; and
based on the comparing, determining whether the unique fragment or the set of unique fragments retrieved from the fragment storage has been altered.
13. The system of claim 8 , wherein the instructions are further translatable by the processor for:
retrieving a set of fragments from the fragment storage;
assembling a document from the set of fragments based on instructions from a document definition associated with the document;
processing the document based on custom configuration used by a post formatting component so as to produce a final document, the post formatting component determining a correct final format, a correct final size, and a correct final resolution for an output channel; and
passing the final document to a driver or renderer for rendering the final document into a rendition of the document, the rendition having the correct final format, the correct final size, and the correct final resolution for the output channel.
14. The system of claim 13 , wherein the correct final format comprises a device-specific format for the output channel.
15. A computer program product comprising a non-transitory computer-readable medium storing instructions translatable by a processor for:
processing raw data streamed from a data source into structured messages containing metadata describing patterns in the raw data, the structure messages constructed on a format-independent data structure;
generating fragments having a presentation-independent format, each of the fragments referencing one of the structure messages; and
for each respective fragment of the fragments:
determining whether the respective fragment matches any fragment stored in a fragment storage; and
storing the respective fragment in the fragment storage only if the respective fragment does not match any fragment stored in the fragment storage.
16. The computer program product of claim 15 , wherein determining whether the respective fragment matches any fragment stored in the fragment storage comprises:
applying a hash function to the respective fragment so as to generate a hash; and
comparing the hash with hashes stored in a tracking table or data structure, the hashes corresponding to unique fragments stored in the fragment storage.
17. The computer program product of claim 15 , wherein determining whether the respective fragment matches any fragment stored in the fragment storage comprises:
applying a cryptographic hash function to the respective fragment so as to generate a checksum; and
comparing the checksum for the respective fragment with checksums of unique fragments stored in the fragment storage.
18. The computer program product of claim 15 , wherein the instructions are further translatable by the processor for:
prior to storing a unique fragment or a set of unique fragments to the fragment storage, applying a cryptographic hash function to the unique fragment or the set of unique fragments so as to generate a first signature; and
storing the first signature with the unique fragment or the set of unique fragments.
19. The computer program product of claim 18 , wherein the instructions are further translatable by the processor for:
retrieving the unique fragment or the set of unique fragments from the fragment storage;
applying the cryptographic hash function to the unique fragment or the set of unique fragments retrieved from the fragment storage so as to generate a second signature;
comparing the first signature with the second signature; and
based on the comparing, determining whether the unique fragment or the set of unique fragments retrieved from the fragment storage has been altered.
20. The computer program product of claim 15 , wherein the instructions are further translatable by the processor for:
retrieving a set of fragments from the fragment storage;
assembling a document from the set of fragments based on instructions from a document definition associated with the document;
processing the document based on custom configuration used by a post formatting component so as to produce a final document, the post formatting component determining a correct final format, a correct final size, and a correct final resolution for an output channel; and
passing the final document to a driver or renderer for rendering the final document into a rendition of the document, the rendition having the correct final format, the correct final size, and the correct final resolution for the output channel.