Generating and compressing data blocks for efficient browse structures
The present disclosure is directed toward systems, methods, and non-transitory computer-readable media for determining an in-memory data structure for storing digital images (e.g., newspaper images representing individual pages of digitized newspapers) based on a first level hash and a second level hash that map to nested categories within a browse structure of a genealogical data system. For example, the disclosed systems generate a multilevel data block by implementing one or more compression techniques to reduce overall data size, particularly relating to month data and image/page identification data. In some cases, the disclosed systems greatly reduce the memory and processing requirements of storing, browsing, and searching digital content items (e.g., newspaper images) within a genealogical database.
1. A computer-implemented method comprising:
identifying a newspaper image as part of a browse structure comprising a hierarchy of nested categories for browsing newspaper images;
determining, based on the browse structure, an in-memory data structure for storing the newspaper image within a newspaper image database;
generating, based on the in-memory data structure, a multilevel data block representing the newspaper image, wherein generating the multilevel data block comprises compressing page identification data for the newspaper image using gaps between page identifiers corresponding to newspaper images; and
storing the multilevel data block within the newspaper image database.
2. The computer-implemented method of claim 1 , wherein determining the in-memory data structure comprises determining a multilevel data hash comprising:
a first level hash that includes a first key, a first field, and a first value for mapping a first set of levels within the hierarchy of nested categories within the browse structure; and
a second level hash that includes a second key, a second field, and a second value for mapping a second set of levels within the hierarchy of nested categories within the browse structure.
3. The computer-implemented method of claim 1 , wherein generating the multilevel data block comprises encoding month data for the newspaper image using binary encodings to indicate months corresponding to the newspaper image.
4. The computer-implemented method of claim 1 , wherein generating the multilevel data block comprises:
determining a gap between a numerical identifier for the newspaper image and another numerical identifier for another newspaper image; and
encoding the gap using a binary encoding within the multilevel data block.
5. The computer-implemented method of claim 1 , wherein generating the multilevel data block comprises compressing page identification data for the newspaper image by:
determining gap deltas indicating differences between the gaps between the page identifiers that indicate numerical differences between consecutive page identifiers for newspaper images; and
encoding the gap deltas using zigzag encoding.
6. The computer-implemented method of claim 1 , wherein generating the multilevel data block comprises aligning bytes of the multilevel data block using a byte alignment padding.
7. The computer-implemented method of claim 1 , wherein storing the multilevel data block within the newspaper image database comprises storing the multilevel data block within a repository of multilevel data blocks for a plurality of newspaper images housed at a single in-memory data server.
8. A non-transitory computer readable medium storing instructions which, when executed by at least one processor, cause the at least one processor to:
identify a digital image as part of a browse structure comprising a hierarchy of nested categories for browsing digital images;
determine, based on the browse structure, an in-memory data structure for storing the digital image within a digital image database;
generate, based on the in-memory data structure, a multilevel data block representing the digital image, wherein generating the multilevel data block comprises compressing page identification data for the digital image by encoding the page identification data with gap data indicating gaps between page identifiers corresponding to digital images; and
store the multilevel data block within the digital image database.
9. The non-transitory computer readable medium of claim 8 , further storing instructions which, when executed by the at least one processor, cause the at least one processor to determine the in-memory data structure for storing the digital image by determining a multilevel data hash comprising:
a first level hash that includes a first key, a first field, and a first value for mapping a first set of levels within the hierarchy of nested categories within the browse structure; and
a second level hash that includes a second key, a second field, and a second value for mapping a second set of levels within the hierarchy of nested categories within the browse structure.
10. The non-transitory computer readable medium of claim 9 , further storing instructions which, when executed by the at least one processor, cause the at least one processor to generate the multilevel data block by:
determining a gap between a numerical identifier for the digital image and another numerical identifier for another digital image; and
encoding the gap using a binary encoding within the multilevel data block.
11. The non-transitory computer readable medium of claim 8 , further storing instructions which, when executed by the at least one processor, cause the at least one processor to generate the multilevel data block by:
determining gap deltas indicating differences between the gaps between the page identifiers that indicate numerical differences between consecutive page identifiers for digital images; and
encoding the gap deltas using zigzag encoding to remove negative values within the gap deltas.
12. The non-transitory computer readable medium of claim 8 , further storing instructions which, when executed by the at least one processor, cause the at least one processor to resolve byte boundary misalignment within the multilevel data block using a byte alignment padding.
13. The non-transitory computer readable medium of claim 8 , further storing instructions which, when executed by the at least one processor, cause the at least one processor to generate the multilevel data block by encoding month data for the digital image using binary encodings for months corresponding to the digital image.
14. The non-transitory computer readable medium of claim 8 , further storing instructions which, when executed by the at least one processor, cause the at least one processor to store the multilevel data block within the digital image database by storing the multilevel data block within a repository of multilevel data blocks for a plurality of digital images housed at a single in-memory data server.
15. A system comprising:
at least one processor; and
a non-transitory computer readable medium storing instructions which, when executed by the at least one processor, cause the system to:
identify a newspaper image as part of a browse structure comprising a hierarchy of nested categories for browsing newspaper images;
determine, based on the browse structure, an in-memory data structure for storing the newspaper image within a newspaper image database;
generate, based on the in-memory data structure, a multilevel data block representing the newspaper image by:
compressing page identification data for the newspaper image using gaps between page identifiers corresponding to newspaper images; and
aligning bytes of the multilevel data block using a byte alignment padding; and
store the multilevel data block within the newspaper image database.
16. The system of claim 15 , further storing instructions which, when executed by the at least one processor, cause the system to determine the in-memory data structure by determining a multilevel data hash comprising:
a first level hash that includes a first key, a first field, and a first value for mapping a first set of levels within the hierarchy of nested categories within the browse structure; and
a second level hash that includes a second key, a second field, and a second value for mapping a second set of levels within the hierarchy of nested categories within the browse structure.
17. The system of claim 15 , further storing instructions which, when executed by the at least one processor, cause the system to generate the multilevel data block by encoding month data for the newspaper image using binary encodings to indicate which months correspond to the newspaper image.
18. The system of claim 15 , further storing instructions which, when executed by the at least one processor, cause the system to generate the multilevel data block by:
determining a gap between a numerical identifier for the newspaper image and another numerical identifier for another newspaper image; and
encoding the gap using a binary encoding within the multilevel data block.
19. The system of claim 15 , further storing instructions which, when executed by the at least one processor, cause the system to generate the multilevel data block by compressing page identification data for the newspaper image by:
determining gap deltas indicating differences between the gaps between the page identifiers that indicate numerical differences between consecutive page identifiers for newspaper images; and
encoding the gap deltas using zigzag encoding.
20. The system of claim 15 , further storing instructions which, when executed by the at least one processor, cause the system to store the multilevel data block within the newspaper image database by storing the multilevel data block within a repository of multilevel data blocks for a plurality of newspaper images housed at a single in-memory data server.