Data organization and access for mixed media document system
A Mixed Media Reality (MMR) system and associated techniques are disclosed. The MMR system provides mechanisms for forming a mixed media document that includes media of at least two types (e.g., printed paper as a first medium and digital content and/or web link as a second medium). In one particular embodiment, the MMR system includes a content-based retrieval database configured with an index table to represent two-dimensional geometric relationships between objects extracted from a printed document in a way that allows look-up using a text-based index. A ranked set of document, page and location hypotheses can be computed given data from the index table. The techniques effectively transform features detected in an image patch into textual terms (or other searchable features) that represent both the features themselves and the geometric relationship between them. A storage facility can be used to store additional characteristics about each document image patch.
1. A computer-implemented method for accessing information in a mixed media document system, the method comprising:
receiving an image patch of a target document;
determining, with one or more processors, from the received image patch, a query that indicates a two-dimensional geometric relationship between a pair of document features in the target document, the two-dimensional geometric relationship including an indication that the pair of document features in the target document are a horizontally adjacent pair of document features or a vertically adjacent pair of document features;
comparing, with the one or more processors, the query to an index table of document features from mixed media documents to identify candidate regions in the mixed media documents that comprise the query, the index table comprising locations of the document features in the mixed media documents; and
responsive to comparing the query to the document features in the index table, identifying one or more of the mixed media documents comprising the identified candidate regions comprising the query by:
adding a weight to an array of an accumulator for each cell in a zone around each pair of document features based on an inverse document frequency associated with each pair of document features, the inverse document frequency being inversely proportional to a number of document pages that contain the image patch;
searching the array of the accumulator for a cell with a maximum value; and
in response to the maximum value exceeding a threshold, reporting coordinates of the cell as a location of the image patch.
2. The method of claim 1 wherein determining the query comprises generating the horizontally adjacent pair of document features and the vertically adjacent pair of document features extracted from the image patch.
3. The method of claim 1 further comprising:
locating a stored page as a candidate match for the image patch of the target document; and
calculating the location within the stored page as a candidate center of the image patch.
4. The method of claim 3 wherein locating the stored page as the candidate match for the image patch of the target document includes:
adding the inverse document frequency for each pair of document features to the accumulator indexed by the document pages on which that pair of document features appears; and
in response to the maximum value in that accumulator exceeding the threshold, outputting a corresponding document page as a match to the image patch.
5. The method of claim 1 , wherein the weight for each cell is determined by a product of the inverse document frequency of that pair of document features and a normalized geometric distance between that cell and a center of the zone.
6. The method of claim 1 further comprising:
identifying one of the candidate regions that is most consistent with the query; and
in response to determining that the identified candidate region satisfies pre-defined matching criteria, confirming the identified candidate region as a match to the target document.
7. A non-transitory machine-readable medium encoded with instructions, that when executed by one or more processors, cause the one or more processors to carry out a process for accessing information in a mixed media document system, the process comprising:
receiving an image patch of a target document;
determining, from the received image patch, a query that indicates a two-dimensional geometric relationship between document features in the target document, the two-dimensional geometric relationship including an indication that the document features in the target document are a horizontally adjacent pair of document features or a vertically adjacent pair of document features;
comparing the query to an index table of the document features from mixed media documents to identify candidate regions in the mixed media documents that comprise the query, the index table comprising locations of the document features in the mixed media documents; and
responsive to comparing the query to the document features in the index table, identifying one or more of the mixed media documents comprising the identified candidate regions comprising the query by:
adding a weight to an array of an accumulator for each cell in a zone around each pair of document features based on an inverse document frequency associated with each pair of document features, the inverse document frequency being inversely proportional to a number of document pages that contain the image patch;
searching the array of the accumulator for the cell with a maximum value; and
in response to the maximum value exceeding a threshold, reporting coordinates of the cell as a location of the image patch.
8. The machine-readable medium of claim 7 wherein determining the query comprises generating the horizontally adjacent pair of document features and the vertically adjacent pair of document features extracted from the image patch.
9. The machine-readable medium of claim 7 further comprising:
locating a stored page as a candidate match for the image patch of the target document; and
calculating the location within the stored page as a candidate center of the image patch.
10. The machine-readable medium of claim 9 wherein locating the stored page as the candidate match for the image patch of the target document includes:
adding the inverse document frequency for each pair of document features to the accumulator indexed by the document pages on which that pair of document features appears; and
in response to the maximum value in that accumulator exceeding the threshold, outputting a corresponding document page as a match to the image patch.
11. The machine-readable medium of claim 7 , wherein the weight for each cell is determined by a product of the inverse document frequency of that pair of document features and a normalized geometric distance between that cell and a center of the zone.
12. The machine-readable medium of claim 7 further comprising:
identifying one of the candidate regions that is most consistent with the query; and
in response to determining that the identified candidate region satisfies pre-defined matching criteria, confirming the identified candidate region as a match to the target document.
13. A system comprising:
one or more processors; and
a memory, the memory storing instructions, which when executed by the one or more processors, cause the one or more processors to:
receive an image patch of a target document;
determine, from the received image patch, a query that indicates a two-dimensional geometric relationship between document features in the target document, the two-dimensional geometric relationship including an indication that the document features in the target document are a horizontally adjacent pair of document features or a vertically adjacent pair of document features;
compare the query to an index table of the document features from mixed media documents to identify candidate regions in the mixed media documents that comprise the query, the index table comprising locations of the document features in the mixed media documents; and
responsive to comparing the query to the document features in the index table, identify one or more of the mixed media documents comprising the identified candidate regions comprising the query by:
adding a weight to an array of an accumulator for each cell in a zone around each pair of document features based on an inverse document frequency associated with each pair of document features, the inverse document frequency being inversely proportional to a number of document pages that contain the image patch;
searching the array of the accumulator for the cell with a maximum value; and
in response to the maximum value exceeding a threshold, reporting coordinates of the cell as a location of the image patch.
14. The system of claim 13 , wherein to determine the query the instructions cause the one or more processors to generate the horizontally adjacent pair of document features and the vertically adjacent pair of document features extracted from the image patch.
15. The system of claim 13 , wherein the instructions further cause the one or more processors to:
locate a stored page as a candidate match for the image patch of the target document; and
calculate the location within the stored page as a candidate center of the image patch.
16. The system of claim 15 , wherein to locate the stored page as the candidate match for the image patch of the target document, the instructions cause the one or more processors to:
add the inverse document frequency for each pair of document features to the accumulator indexed by the document pages on which that pair of document features appears; and
in response to the maximum value in that accumulator exceeding the threshold, output a corresponding document page as a match to the image patch.
17. The system of claim 13 , wherein the weight for each cell is determined by a product of the inverse document frequency of that pair of document features and a normalized geometric distance between that cell and a center of the zone.
18. The system of claim 13 , wherein the instructions further cause the one or more processors to:
identify one of the candidate regions that is most consistent with the query; and
in response to determining that the identified candidate region satisfies pre-defined matching criteria, confirm the identified candidate region as a match to the target document.