Enhanced string match matrix generation
A method for remediating duplication errors associated with first and second documents includes obtaining a first document and a second document, generating a corresponding first document string set and a second document string set, generating a second document string subset that is a proper subset of the second document string set, dividing the second document string subset into second document substring, transforming the second document substrings into a synthetic substring set, converting the first document string set into first document embeddings in vector space, converting the synthetic substrings into synthetic embeddings, generate a closest embedding set of one or more first document embeddings, generating, using a respective one of the closest embedding sets and a linking string generator, a linking string defining an association between the respective substring and one or more respective first document portions, and generating a string match matrix data.
1 . A method comprising:
accessing a first document set that includes a first document;
accessing a second document set that includes a second document, wherein each respective document of the second document set has a style different from documents of the first document set and each respective document includes:
respective second document first text content;
respective second document second text content; and
respective second document image content;
for each respective document of the second document set:
extracting a respective extracted string from the respective second document second text content of the respective document;
generating a respective synthetic string based on the respective extracted string and associated portions of one or both of the respective second document first text content and the respective second document image content; and
assessing a respective similarity between the respective document and at least one document of the first document set based on the respective synthetic string;
determining to select the first document from among the first document set based on similarity to the second document from the second document set;
generating a string match data structure describing a relationship between the first document and the second document from the second document set, wherein the string match data structure includes:
first portions based on one or more second document substrings from the second document; and
second portions based on one or more first document strings that correspond to the one or more second document substrings;
and
transmitting a request regarding a duplication error, wherein the request includes the string match data structure, and requests initiating a remediation action selected from a group consisting of: deletion of the first document, hiding the first document, locking the first document, de-indexing the first document from search results, deprioritizing the first document in search results, redirecting links to the first document to the second document, granting privileges associated with the second document, allocating privileges associated with the second document.
2 . The method of claim 1 ,
wherein the first document has a first style;
wherein the second document has a second style; and
wherein the respective synthetic string is generated to mimic the first style rather than the second style.
3 . The method of claim 1 , wherein at least one of the first portions includes image content based on first document image content.
4 . The method of claim 1 , wherein the second document comprises one or more privileged portions relevant to the first document.
5 . The method of claim 1 , wherein accessing the first document set includes accessing sub-domains within a domain, wherein each accessed sub-domain is a document of the first document set.
6 . The method of claim 1 , further comprising;
generating the first document set, wherein the generating includes:
crawling sub-domains within a web domain; and
capturing each respective sub-domain as a respective document of the first document set.
7 . The method of claim 1 , wherein assessing the respective similarity is done based on points in vector space.
8 . A method comprising:
accessing a first document set that includes a plurality of documents and one or more target documents;
accessing a second document set including a second document, wherein each respective document of the second document set has a style different from documents of the first document set and each respective document includes:
respective second document first text content;
respective second document second text content; and
respective second document image content;
for each respective document of the second document set:
extracting a respective extracted string from the respective second document second text content of the respective document;
generating a respective synthetic string based on the respective extracted string and associated portions of one or both of the respective second document first text content and the respective second document image content; and
assessing a respective similarity between the respective document and at least one document of the first document set based on the respective synthetic string;
determining to select the one or more target documents from the first document set based on similarity between the one or more target documents and one or more respective synthetic strings associated with the second document;
generating a string match data structure describing a relationship between the one or more target documents and the second document from the second document set, wherein the string match data structure includes:
first portions based on one or more second document substrings from the second document; and
second portions based on one or more target document strings from the one or more target documents that correspond to the one or more second document substrings;
and
transmitting a request regarding a duplication error, wherein the request includes the string match data structure, and requests initiating a remediation action selected from a group consisting of: deletion of a target document, hiding the target document, locking the target document, de-indexing the target document of the one or more target documents from search results, deprioritizing the target document in search results, redirecting links to the target document to the second document, granting privileges associated with the second document, allocating privileges associated with the second document.
9 . The method of claim 8 ,
wherein documents of the first document set have a first style; and
wherein documents of the second document set have a second style different from the first style.
10 . The method of claim 9 , wherein the respective synthetic string is generated to mimic the first style rather than the second style.
11 . The method of claim 8 , wherein at least one of the first portions includes image content based on the respective second document image content.
12 . The method of claim 8 , wherein the second document comprises one or more privileged portions relevant to a target document of the one or more target documents.
13 . The method of claim 8 , wherein determining to select a target document from the first document set based on similarity includes determining the similarity based on locations within vector space.
14 . The method of claim 8 , wherein the first document set includes one or more web pages.