Sliding window to detect entities in corpus using natural language processing
According to one or more embodiments of the present invention, an input request to a natural language processing (NLP) system is optimized. A window-size is selected for annotating an input corpus. The corpus is divided into partitions of the window-size, each partition processed separately. Further, a first set of entities is identified in a first partition, and a second set of entities in a second partition. Further, a third partition containing a first segment and a second segment is determined. The first segment overlaps the first partition, and the second segment overlaps the second partition. The method further includes identifying a third set of entities in the third partition. In response to the third set of entities being distinct from a set of entities from the first segment and the second segment, the window-size is adjusted. The input request for the NLP system is generated using the adjusted window-size.
1. A computer-implemented method for optimizing a window-size for an input request that is sent to a natural language processing (NLP) system, the method comprising:
selecting, by a processor, a window-size for identifying entities in an input corpus, the input corpus being divided into a plurality of partitions of the window-size, each partition being processed separately;
identifying, by the processor, a first set of entities in a first partition;
identifying, by the processor, a second set of entities in a second partition;
determining, by the processor, a third partition that comprises a first segment and a second segment, the first segment overlaps the first partition, and the second segment overlaps the second partition;
identifying, by the processor, a third set of entities in the third partition;
determining the third set of entities is distinct from a set of entities corresponding to the first segment and the second segment;
adjusting, by the processor, the window-size based on determining the third set of entities is distinct, determining the third set of entities is no longer distinct from the set of entities corresponding to the first segment and the second segment based on the adjusting the window-size, wherein the third set of entities is determined to be no longer distinct when the window-size produces a smallest overlap between the third partition and the first partition providing the same results;
and
generating, by the processor, the input request for the NLP system using the window-size that has been adjusted.
2. The computer-implemented method of claim 1 , wherein the first partition and the second partition are consecutive partitions.
3. The computer-implemented method of claim 1 , wherein the first segment in the third partition is based on an overlap-size.
4. The computer-implemented method of claim 1 , wherein adjusting the window-size comprises decreasing the window-size.
5. The computer-implemented method of claim 1 , wherein adjusting the window-size comprises increasing the window-size.
6. The computer-implemented method of claim 1 , wherein the processor is part of the NLP system.
7. The computer-implemented method of claim 1 , wherein the processor receives the input request for annotating the input corpus, and in response:
generates a plurality of input requests based on the window-size that is adjusted; and
sends, to the NLP system, the plurality of input requests.
8. The computer-implemented method of claim 1 , wherein the processor receives, from a user computer system, a request for determining the window-size for annotating the input corpus, and in response, outputs the window-size that is adjusted to the user computer system for generating the input request for the NLP system.
9. A system comprising:
a memory having computer readable instructions; and
one or more processors for executing the computer readable instructions, the computer readable instructions controlling the one or more processors to perform operations comprising:
selecting a window-size for identifying entities in an input corpus, the input corpus being divided into a plurality of partitions of the window-size, each partition being processed separately;
identifying a first set of entities in a first partition;
identifying a second set of entities in a second partition;
determining a third partition that comprises a first segment and a second segment, the first segment overlaps the first partition, and the second segment overlaps the second partition;
identifying a third set of entities in the third partition; determining the third set of entities is distinct from a set of entities corresponding to the first segment and the second segment;
adjusting the window-size based on determining the third set of entities is distinct, determining the third set of entities is no longer distinct from the set of entities corresponding to the first segment and the second segment based on the adjusting the window-size, wherein the third set of entities is determined to be no longer distinct when the window-size produces a smallest overlap between the third partition and the first partition providing the same results;
and
generating the input request for the NLP system using the window-size that has been adjusted.
10. The system of claim 9 , wherein the first partition and the second partition are consecutive partitions.
11. The system of claim 9 , wherein the first segment in the third partition is based on an overlap-size.
12. The system of claim 9 , wherein adjusting the window-size comprises decreasing the window-size.
13. The system of claim 9 , wherein adjusting the window-size comprises increasing the window-size.
14. The system of claim 9 , wherein the one or more processors receive the input request for annotating the input corpus, and in response:
generate a plurality of input requests based on the window-size that is adjusted; and
send, to the NLP system, the plurality of input requests.
15. The system of claim 9 , wherein the one or more processors are part of the NLP system.
16. A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to perform operations comprising:
selecting a window-size for identifying entities in an input corpus, the input corpus being divided into a plurality of partitions of the window-size, each partition being processed separately;
identifying a first set of entities in a first partition;
identifying a second set of entities in a second partition;
determining a third partition that comprises a first segment and a second segment, the first segment overlaps the first partition, and the second segment overlaps the second partition;
identifying a third set of entities in the third partition;
determining the third set of entities is distinct from a set of entities corresponding to the first segment and the second segment;
adjusting the window-size based on determining the third set of entities is distinct, determining the third set of entities is no longer distinct from the set of entities corresponding to the first segment and the second segment based on the adjusting the window-size, wherein the third set of entities is determined to be no longer distinct when the window-size produces a smallest overlap between the third partition and the first partition providing the same results;
and
generating the input request for the NLP system using the window-size that has been adjusted.
17. The computer program product of claim 16 , wherein the first partition and the second partition are consecutive partitions.
18. The computer program product of claim 16 , wherein the first segment in the third partition is based on an overlap-size.
19. The computer program product of claim 16 , wherein adjusting the window-size comprises decreasing the window-size.
20. The computer program product of claim 16 , wherein adjusting the window-size comprises increasing the window-size.