Flexible and scalable artificial intelligence and analytics platform with advanced content analytics and data ingestion
Disclosed is a flexible and scalable artificial intelligence and analytics platform with advanced content analytics and content ingestion. Disparate contents can be ingested into a content analytics system of the platform through a content ingestion pipeline operated by a sophisticated text mining engine. Prior to persistence, editorial metadata can be extracted and semantic metadata inferred to gain insights across the disparate contents. The editorial metadata and the semantic metadata can be dynamically mapped, as the disparate contents are crawled from disparate sources, to an internal ingestion pipeline document conforming to a uniform mapping schema that specifies master metadata of interest. For persistence, the semantic metadata in the internal ingestion pipeline document can be mapped to metadata tables conforming to a single common data model of a central repository. In this way, ingested metadata can be leveraged across the platform, for instance, for trend analysis, mood detection, model building, etc.
1. A method for content ingestion, comprising:
generating, by a text mining engine operating a content ingestion pipeline in a system having a processor and a non-transitory computer-readable medium, metadata from contents obtained or received through real-time data feeds from content sources;
capturing, by the text mining engine, the metadata or a portion thereof in an internal ingestion pipeline document, wherein the internal ingestion pipeline document conforms to a uniform mapping schema that defines a set of master metadata of interest and wherein the capturing comprises:
dynamically mapping the metadata to the set of master metadata of interest so as to produce matched metadata; and
populating the internal ingestion pipeline document with the matched metadata; and
mapping, by the text mining engine, the matched metadata in the internal ingestion pipeline document to metadata tables in a central repository to thereby persist the matched metadata in the central repository.
2. The method according to claim 1 , wherein the internal ingestion pipeline document is created utilizing the uniform mapping schema.
3. The method according to claim 1 , wherein the internal ingestion pipeline document is one of a plurality of internal ingestion pipeline documents utilized by the text mining engine for capturing information of interest in ingesting the contents through the content ingestion pipeline.
4. The method according to claim 1 , wherein the mapping the matched metadata in the internal ingestion pipeline document to the metadata tables in the central repository utilizes a single common data model of the central repository.
5. The method according to claim 1 , further comprising:
applying text mining functions to the contents, wherein the generating the metadata from the contents comprises capturing, in the internal ingestion pipeline document, outputs from the applying the text mining functions to the contents.
6. The method according to claim 5 , wherein the text mining functions comprise at least one of language detection, concept extraction, categorization, topic, classification, sentiment analysis, summarization, or entity extraction.
7. The method according to claim 1 , further comprising:
storing information about the internal ingestion pipeline document in a document table.
8. A system, comprising:
a processor;
a non-transitory computer-readable medium; and
instructions stored on the non-transitory computer-readable medium and translatable by the processor for:
generating metadata in ingesting contents through a content ingestion pipeline, the contents obtained or received through real-time data feeds from content sources;
capturing the metadata or a portion thereof in an internal ingestion pipeline document, wherein the internal ingestion pipeline document conforms to a uniform mapping schema that defines a set of master metadata of interest and wherein the capturing comprises:
dynamically mapping the metadata to the set of master metadata of interest so as to produce matched metadata; and
populating the internal ingestion pipeline document with the matched metadata; and
mapping the matched metadata in the internal ingestion pipeline document to metadata tables in a central repository to thereby persist the matched metadata in the central repository.
9. The system of claim 8 , wherein the internal ingestion pipeline document is created utilizing the uniform mapping schema.
10. The system of claim 8 , wherein the internal ingestion pipeline document is one of a plurality of internal ingestion pipeline documents utilized by the text mining engine for capturing information of interest in ingesting the contents through the content ingestion pipeline.
11. The system of claim 8 , wherein the mapping the matched metadata in the internal ingestion pipeline document to the metadata tables in the central repository utilizes a single common data model of the central repository.
12. The system of claim 8 , wherein the instructions are further translatable by the processor for:
applying text mining functions to the contents, wherein the generating the metadata from the contents comprises capturing, in the internal ingestion pipeline document, outputs from the applying the text mining functions to the contents.
13. The system of claim 12 , wherein the text mining functions comprise at least one of language detection, concept extraction, categorization, topic, classification, sentiment analysis, summarization, or entity extraction.
14. The system of claim 8 , wherein the instructions are further translatable by the processor for:
storing information about the internal ingestion pipeline document in a document table.
15. A computer program product comprising a non-transitory computer-readable medium storing instructions translatable by a processor for:
generating metadata in ingesting contents through a content ingestion pipeline, the contents obtained or received through real-time data feeds from content sources;
capturing the metadata or a portion thereof in an internal ingestion pipeline document, wherein the internal ingestion pipeline document conforms to a uniform mapping schema that defines a set of master metadata of interest and wherein the capturing comprises:
dynamically mapping the metadata to the set of master metadata of interest so as to produce matched metadata; and
populating the internal ingestion pipeline document with the matched metadata; and
mapping the matched metadata in the internal ingestion pipeline document to metadata tables in a central repository to thereby persist the matched metadata in the central repository.
16. The computer program product of claim 15 , wherein the internal ingestion pipeline document is created utilizing the uniform mapping schema.
17. The computer program product of claim 15 , wherein the internal ingestion pipeline document is one of a plurality of internal ingestion pipeline documents utilized by the text mining engine for capturing information of interest in ingesting the contents through the content ingestion pipeline.
18. The computer program product of claim 15 , wherein the mapping the matched metadata in the internal ingestion pipeline document to the metadata tables in the central repository utilizes a single common data model of the central repository.
19. The computer program product of claim 15 , wherein the instructions are further translatable by the processor for:
applying text mining functions to the contents, wherein the generating the metadata from the contents comprises capturing, in the internal ingestion pipeline document, outputs from the applying the text mining functions to the contents.
20. The computer program product of claim 19 , wherein the text mining functions comprise at least one of language detection, concept extraction, categorization, topic, classification, sentiment analysis, summarization, or entity extraction.