CONTENT DISCOVERY AND INGESTION

1 . A method comprising:

discovering, by a data processing system, data files from one or more content repositories;

identifying, by the data processing system, key terms in the data files;

for each of the identified key terms:

determining, by the data processing system, a frequency of occurrence of the key term in the corresponding data file; and

determining, by the data processing system, locations of the key term in the corresponding data file;

generating, by the data processing system, a plurality of knowledge units from the data files based on the determined frequencies of occurrence and the determined locations of the key terms; and

storing, by the data processing system, the plurality of knowledge units in a data store.

2 . The method of claim 1 , further comprising:

converting the data files into a common data format.

3 . The method of claim 1 , further comprising:

associating each of the knowledge units with a term vector that includes one or more key terms in the corresponding knowledge unit.

4 . The method of claim 3 , further comprising:

selecting one of the knowledge units;

performing a similarity mapping between the selected knowledge unit and the other knowledge units based on a knowledge unit distance metric computed between the term vector of the selected knowledge unit and the term vector of each of the other knowledge units;

identifying one or more of the other knowledge units that has the knowledge unit distance metric being below a predetermined threshold distance; and

combining the selected knowledge unit and the identified one or more of the other knowledge units into a knowledge pack.

5 . The method of claim 1 , wherein generating the plurality of knowledge units includes segmenting a data file into knowledge segments, and forming one or more of the knowledge units from the knowledge segments.

6 . The method of claim 5 , wherein the data file is segmented based on organization of content within the data file.

7 . The method of claim 5 , wherein the data file includes unstructured content, and the data file is segmented based on the determined locations of key terms in the data file.

8 . A non-transitory computer-readable storage memory storing a plurality of instructions executable by one or more processors, the plurality of instructions comprising:

instructions that cause the one or more processors to discover data files from one or more content repositories;

instructions that cause the one or more processors to identify key terms in the data files;

instructions that cause the one or more processors to, for each of the identified key terms, determine a frequency of occurrence of the key term in the corresponding data file, and determine locations of the key term in the corresponding data file;

instructions that cause the one or more processors to generate a plurality of knowledge units from the data files based on the determined frequencies of occurrence and the determined locations of the key terms; and

instructions that cause the one or more processors to store the plurality of knowledge units in a data store.

9 . The non-transitory computer-readable storage memory of claim 8 , wherein the plurality of instructions further comprises:

instructions that cause the one or more processors to convert the data files into a common data format.

10 . The non-transitory computer-readable storage memory of claim 8 , wherein the plurality of instructions further comprises:

instructions that cause the one or more processors to associate each of the knowledge units with a term vector that includes one or more key terms in the corresponding knowledge unit.

11 . The non-transitory computer-readable storage memory of claim 10 , wherein the plurality of instructions further comprises:

instructions that cause the one or more processors to select one of the knowledge units;

instructions that cause the one or more processors to perform a similarity mapping between the selected knowledge unit and the other knowledge units based on a knowledge unit distance metric computed between the term vector of the selected knowledge unit and the term vector of each of the other knowledge units;

instructions that cause the one or more processors to identify one or more of the other knowledge units that has the knowledge unit distance metric being below a predetermined threshold distance; and

instructions that cause the one or more processors to combine the selected knowledge unit and the identified one or more of the other knowledge units into a knowledge pack.

12 . The non-transitory computer-readable storage memory of claim 8 , wherein the plurality of instructions further comprises:

instructions that cause the one or more processors to segment a data file into knowledge segments; and

instructions that cause the one or more processors to form one or more of the knowledge units from the knowledge segments.

13 . The non-transitory computer-readable storage memory of claim 12 , wherein the data file is segmented based on organization of content within the data file.

14 . The non-transitory computer-readable storage memory of claim 12 , wherein the data file includes unstructured content, and the data file is segmented based on the determined locations of key terms in the data file.

15 . A system comprising:

one or more processors; and

a memory coupled with and readable by the one or more processors, the memory configured to store a set of instructions which, when executed by the one or more processors, causes the one or more processors to:

discover data files from one or more content repositories;

identify key terms in the data files;

for each of the identified key terms:

determine a frequency of occurrence of the key term in the corresponding data file; and

determine locations of the key term in the corresponding data file;

generate a plurality of knowledge units from the data files based on the determined frequencies of occurrence and the determined locations of the key terms; and

store the plurality of knowledge units in a data store.

16 . The system of claim 15 , wherein the set of instructions further comprises instructions, which when executed by the one or more processors, causes the one or more processors to:

convert the data files into a common data format.

17 . The system of claim 15 , wherein the set of instructions further comprises instructions, which when executed by the one or more processors, causes the one or more processors to:

associate each of the knowledge units with a term vector that includes one or more key terms in the corresponding knowledge unit.

18 . The system of claim 17 , wherein the set of instructions further comprises instructions, which when executed by the one or more processors, causes the one or more processors to:

select one of the knowledge units;

perform a similarity mapping between the selected knowledge unit and the other knowledge units based on a knowledge unit distance metric computed between the term vector of the selected knowledge unit and the term vector of each of the other knowledge units;

identify one or more of the other knowledge units that has the knowledge unit distance metric being below a predetermined threshold distance; and

combine the selected knowledge unit and the identified one or more of the other knowledge units into a knowledge pack.

19 . The system of claim 15 , wherein the set of instructions further comprises instructions, which when executed by the one or more processors, causes the one or more processors to:

segment a data file into knowledge segments; and

forming one or more of the knowledge units from the knowledge segments.

20 . The system of claim 19 , wherein the data file is segmented based on organization of content within the data file, or the determined locations of key terms in the data file.