AUTOMATED IDENTIFICATION OF RECURRING TEXT
In embodiments, one or more computer-readable media may have instructions stored thereon which, when executed by a processor of a computing device, provide the computing device with a recurring text identification service. The recurring text identification service may be configured, in some embodiments, to receive a request to identify recurring text within a plurality of documents. The recurring text identification service may be further configured to analyze individual segments of the plurality of documents to generate segment identifiers respectively associated with the segments. In embodiments, the segment identifiers may be based on content of the segments. In embodiments, segments with the same content may have equivalent segment identifiers. The recurring text identification service may further be configured to generate a distribution of the segment identifiers and may enable the distribution of segment identifiers to be used to streamline identification of recurring text within the plurality of documents.
1 . One or more computer-readable media having instructions stored thereon which, when executed by a processor of a computing device, cause the computing device to provide a recurring text identification service configured to:
receive a request to identify recurring text within a plurality of documents;
analyze individual segments of the plurality of documents to generate segment identifiers respectively associated with the segments, wherein the segment identifiers are based at least in part on content of the segments, and wherein segments with the same content have equivalent segment identifiers;
generate a distribution of the segment identifiers; and
enable the distribution of segment identifiers to be used to streamline identification of recurring text within the plurality of documents.
2 . The computer-readable media of claim 1 , wherein to enable the distribution of segment identifiers the recurring text identification service is further configure to create and output a report of the distribution of the segment identifiers.
3 . The computer-readable media of claim 2 , wherein to output comprises output to a display with the segment identifiers being selectable by a user, wherein the recurring text identification service is further configured to receive segment identifier selections of the user, and wherein the recurring text identification service is also further configured to streamline identification of recurring text within the plurality of documents by inclusion of only segments of the plurality of documents having a selected or equivalent segment identifier as recurring text.
4 . The computer-readable media of claim 3 , wherein the recurring text identification service is further configured to generate a plurality of indices to index only those segments not included as recurring text to facilitate searching for content within the plurality of documents.
5 . The computer-readable media of claim 1 , wherein generation of a segment identifier for a segment includes application of a hash function to the content of the segment.
6 . The computer-readable media of claim 5 , wherein the hash function is a message-digest 5 (MD5) hash function.
7 . The computer-readable media of claim 1 , wherein the recurring text identification service is further configured to partition each of the plurality of documents into a plurality of segments; wherein partition of each document is based at least in part on paragraph break indicators contained within the document.
8 . The computer-readable media of claim 1 , wherein the recurring text identification service is further configured to determine whether the content of each individual segment meets one or more analysis conditions and only analyzing the individual segment if the segment meets the one or more analysis conditions.
9 . The computer-readable media of claim 8 , wherein the one or more analysis conditions include at least one of a character length of the content of a segment or a predefined character pattern of the respective segment.
10 . A system for identifying recurring text contained within one or more documents comprising:
a processor; and
a recurring text identification service configured to cause the processor to:
receive a request to identify recurring text within a plurality of documents;
analyze individual segments of the plurality of documents to generate segment identifiers respectively associated with the segments, wherein the segment identifiers are based at least in part on content of the segments, and wherein segments with the same content have equivalent segment identifiers;
generate a distribution of the segment identifiers; and
enable the distribution of segment identifiers to be used to streamline identification of recurring text within the plurality of documents.
11 . The system of claim 10 , wherein to enable the distribution of segment identifiers the recurring text identification service further configures the processor to create and output a report of the distribution of the segment identifiers.
12 . The system of claim 11 , wherein the system further comprises a display and to output comprises output to the display with the segment identifiers being selectable by a user, wherein the recurring text identification service further configures the processor to receive segment identifier selections of the user, and wherein the recurring text identification service also further configures the processor to streamline identification of recurring text within the plurality of documents by inclusion of only segments of the plurality of documents having a selected or equivalent segment identifier as recurring text.
13 . The system of claim 12 , wherein the recurring text identification service further configures the processor to generate a plurality of indices to index only those segments not included as recurring text to facilitate searching for content within the plurality of documents.
14 . The system of claim 10 , wherein generation of a segment identifier for a segment includes application of a hash function to the content of the segment.
15 . The system of claim 14 , wherein the hash function is a message-digest 5 (MD5) hash function.
16 . The system of claim 10 , wherein the recurring text identification service further configures the processor to partition each of the plurality of documents into a plurality of segments; wherein partition of each document is based at least in part on paragraph break indicators contained within the document.
17 . The system of claim 10 , wherein the recurring text identification service further configures the processor to determine whether the content of each individual segment meets one or more analysis conditions and only analyzing the individual segment if the segment meets the one or more analysis conditions.
18 . The system of claim 17 , wherein the one or more analysis conditions include at least one of a character length of the content of a segment or a predefined character pattern of the respective segment.
19 . A computer-implemented method for identifying recurring text in one or more documents comprising:
receiving, by a recurring text identification service of a computing device, a request to identify recurring text within a plurality of documents;
analyzing, by the recurring text identification service, individual segments of the plurality of documents to generate segment identifiers respectively associated with the segments, wherein the segment identifiers are based at least in part on content of the segments, and wherein segments with the same content have equivalent segment identifiers;
generating, by the recurring text identification service, a distribution of the segment identifiers; and
enabling, by the recurring text identification service, the distribution of segment identifiers to be used in streamlining identification of recurring text within the plurality of documents.
20 . The computer-implemented method of claim 19 , wherein enabling the distribution of segment identifiers further comprises creating and outputting a report of the distribution of the segment identifiers.
21 . The computer-implemented method of claim 20 , wherein outputting comprises outputting to a display with the segment identifiers being selectable by a user, and further comprising receiving, by the recurring text identification service, segment identifier selections of the user and wherein streamlining further comprises including, by the recurring text identification service, only segments of the plurality of documents having selected or equivalent segment identifier from further processing as recurring text.
22 . The computer-implemented method of claim 21 , further comprising generating, by the recurring text identification service, a plurality of indices to index only those segments not included as recurring text to facilitate searching for content within the plurality of documents.
23 . The computer-implemented method of claim 19 , wherein generating a segment identifier for a segment includes applying a hash function to the content of the segment.
24 . The computer-implemented method of claim 23 , wherein the hash function is a message-digest 5 (MD5) hash function.
25 . The computer-implemented method of claim 19 , further comprising partitioning, by the recurring text identification service, each of the plurality of documents into a plurality of segments; wherein partitioning each document is based at least in part on paragraph break indicators contained within the document.
26 . The computer-implemented method of claim 19 , further comprising determining whether the content of each individual segment meets one or more analysis conditions and only analyzing the individual segment if the segment meets the one or more analysis conditions.
27 . The computer-implemented method of claim 26 , wherein the one or more analysis conditions include at least one of a character length of the content of a segment or a predefined character pattern of the respective segment.