Domain dictionary constructing method and apparatus

1 . A domain dictionary constructing method, implemented by an electronic device, comprising:

providing, to an n-gram model, a domain corpus sample for segmenting the domain corpus sample to obtain a first character segment set, wherein the first character segment set comprises at least one first character segment;

calculating an inter-character correlation index of the at least one first character segment;

determining a second character segment set according to the correlation index, wherein the second character segment set comprises a first character segment of which the correlation index is greater than or equal to a preset threshold;

determining a third character segment set according to the second character segment set, wherein the third character segment set comprises the second character segment set and second character segments of the domain corpus sample, wherein each of the second character segments is a character segment that is the same as a character segment obtained by combining at least two character segments of the second character segment set;

constructing a domain dictionary according to the third character segment set; and

using the domain dictionary for one or more of: domain classification, keyword extraction, construction of a knowledge graph, domain word expansion, and construction of domain word knowledge bases.

2 . The method according to claim 1 , wherein the calculating the inter-character correlation index of the first character segment comprises:

calculating a sub correlation index of each sub-segment set of the first character segment, wherein the sub-segment set comprises at least two sub-segments in the first character segment;

calculating the inter-character correlation index of the first character segment according to the sub correlation index.

3 . The method according to claim 2 , wherein the calculating the sub correlation index of each sub segment set of the first character segment comprises:

calculating the sub correlation index of each sub-segment set of the first character segment based on a first calculating formula:

P

ik

=

NUM

⁡

(

i

)

∏

j

=

1

J

k

NUM

⁡

(

j

)

wherein NUM(i) represents a number of occurrences or a probability of occurrence of a-th first character segment of the first character segment set in the first character segment set, J k represents a number of sub-segments of a-th sub-segment set of the-th first character segment, NUM(j) represents a number of occurrences or a probability of occurrence of an j-th sub-segment of the-th sub-segment set in the first character segment set, P ik represents a sub correlation index of the-th sub-segment set of the-th first character segment, wherein i and k are both positive integers.

4 . The method according to claim 2 , wherein the calculating the inter-character correlation index of the first character segment according to the sub correlation index comprises:

determining a minimum sub correlation index as the inter-character correlation index of the first character segment.

5 . The method according to claim 1 , wherein the segmenting the domain corpus sample to obtain the first character segment set comprises:

segmenting the domain corpus sample by using different segmentation lengths respectively to obtain the first character segment set.

6 . The method according to claim 1 , wherein a maximum segmentation length for the domain corpus sample is L, and L is a positive integer; wherein the constructing the domain dictionary according to the third character segment set comprises:

deleting third character segments from the third character segment set, wherein each of the third character segments satisfies any one of the following conditions: having a length less than or equal to L and not belonging to the second character segment set; having a length greater than L and at least one of at least two L character segments obtained by segmenting with the maximum segmentation length not belonging to the second character segment set;

constructing the domain dictionary according to the third character segment set after deleting the third character segments.

7 . An electronic device, comprising: a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, causes the processor to:

provide, to an n-gram model, a domain corpus sample for segmenting the domain corpus sample to obtain a first character segment set, wherein the first character segment set comprises at least one first character segment;

calculate an inter-character correlation index of the at least one first character segment;

determine a second character segment set according to the correlation index, wherein the second character segment set comprises a first character segment of which the correlation index is greater than or equal to a preset threshold;

determine a third character segment set according to the second character segment set, wherein the third character segment set comprises the second character segment set and second character segments of the domain corpus sample, wherein each of the second character segments is a character segment that is the same as a character segment obtained by combining at least two character segments of the second character segment set;

construct a domain dictionary according to the third character segment set; and

use the domain dictionary for one or more of: domain classification, keyword extraction, construction of a knowledge graph, domain word expansion, and construction of domain word knowledge bases.

8 . The electronic device according to claim 7 , wherein the computer program further causes the processor to:

calculate a sub correlation index of each sub-segment set of the first character segment, wherein the sub-segment set comprises at least two sub-segments in the first character segment;

calculate the inter-character correlation index of the first character segment according to the sub correlation index.

9 . The electronic device according to claim 8 , wherein the computer program further causes the processor to:

calculate the sub correlation index of each sub-segment set of the first character segment based on a first calculating formula:

P

ik

=

NUM

⁡

(

i

)

∏

j

=

1

J

k

NUM

⁡

(

j

)

wherein NUM(i) represents a number of occurrences or a probability of occurrence of a-th first character segment of the first character segment set in the first character segment set, J k represents a number of sub-segments of a-th sub-segment set of the-th first character segment, NUM(j) represents a number of occurrences or a probability of occurrence of an j-th sub-segment of the-th sub-segment set in the first character segment set, P ik represents a sub correlation index of the-th sub segment set of the-th first character segment, wherein i and k are both positive integers.

10 . The electronic device according to claim 8 , wherein the computer program further causes the processor to:

determine a minimum sub correlation index as the inter-character correlation index of the first character segment.

11 . The electronic device according to claim 7 , wherein the computer program further causes the processor to:

segment the domain corpus sample by using different segmentation lengths respectively to obtain the first character segment set.

12 . The electronic device according to claim 7 , wherein a maximum segmentation length for the domain corpus sample is L, and L is a positive integer; wherein the computer program further causes the processor to:

delete third character segments from the third character segment set, wherein each of the third character segments satisfies any one of the following conditions: having a length less than or equal to L and not belonging to the second character segment set; having a length greater than L and at least one of at least two L character segments obtained by segmenting with the maximum segmentation length not belonging to the second character segment set;

construct the domain dictionary according to the third character segment set after deleting the third character segments.

13 . A non-transitory computer-readable storage medium, storing a computer program therein, wherein the computer program, when executed by a processor of an electronic device, causes the processor to execute following operations:

providing, to an n-gram model, a domain corpus sample for segmenting the domain corpus sample to obtain a first character segment set, wherein the first character segment set comprises at least one first character segment;

calculating an inter-character correlation index of the at least one first character segment;

determining a second character segment set according to the correlation index, wherein the second character segment set comprises a first character segment of which the correlation index is greater than or equal to a preset threshold;

determining a third character segment set according to the second character segment set, wherein the third character segment set comprises the second character segment set and second character segments of the domain corpus sample, wherein each of the second character segments is a character segment that is the same as a character segment obtained by combining at least two character segments of the second character segment set;

constructing a domain dictionary according to the third character segment set; and

using the domain dictionary for one or more of: domain classification, keyword extraction, construction of a knowledge graph, domain word expansion, and construction of domain word knowledge bases.

14 . The non-transitory computer-readable storage medium according to claim 13 , wherein the computer program further causes the processor to execute following operations:

calculating a sub correlation index of each sub-segment set of the first character segment, wherein the sub-segment set comprises at least two sub-segments in the first character segment;

calculating the inter-character correlation index of the first character segment according to the sub correlation index.

15 . The non-transitory computer-readable storage medium according to claim 14 , wherein the computer program further causes the processor to execute following operation:

calculating the sub correlation index of each sub-segment set of the first character segment based on a first calculating formula:

P

ik

=

NUM

⁡

(

i

)

∏

j

=

1

J

k

NUM

⁡

(

j

)

wherein NUM(i) represents a number of occurrences or a probability of occurrence of a-th first character segment of the first character segment set in the first character segment set, J k represents a number of sub-segments of a-th sub-segment set of the-th first character segment, NUM(j) represents a number of occurrences or a probability of occurrence of an j-th sub-segment of the-th sub-segment set in the first character segment set, P ik represents a sub correlation index of the-th sub segment set of the-th first character segment, wherein i and k are both positive integers.

16 . The non-transitory computer-readable storage medium according to claim 14 , wherein the computer program further causes the processor to execute following operation:

determining a minimum sub correlation index as the inter-character correlation index of the first character segment.

17 . The non-transitory computer-readable storage medium according to claim 13 , wherein the computer program further causes the processor to execute following operation:

segmenting the domain corpus sample by using different segmentation lengths respectively to obtain the first character segment set.

18 . The non-transitory computer-readable storage medium according to claim 13 , wherein a maximum segmentation length for the domain corpus sample is L, and L is a positive integer; wherein the computer program further causes the processor to execute following operations:

deleting third character segments from the third character segment set, wherein each of the third character segments satisfies any one of the following conditions: having a length less than or equal to L and not belonging to the second character segment set; having a length greater than L and at least one of at least two L character segments obtained by segmenting with the maximum segmentation length not belonging to the second character segment set;

constructing the domain dictionary according to the third character segment set after deleting the third character segments.