Image processing device for displaying moving image and image processing method thereof
View Patent ↗Data of a moving image has a hierarchical structure comprising a 0-th layer, a first layer, a second layer, and a third layer in a z axis direction. Each layer is composed of moving image data of a single moving image expressed in different resolutions. Both the coordinates of a viewpoint at the time of the display of a moving image and a corresponding display area are determined in a virtual three-dimensional space formed by an x axis representing the horizontal direction of the image, a y axis representing the vertical direction of the image, and a z axis representing the resolution. By providing a switching boundary for layers with respect to the z axis, the layers of the moving image data used for frame rendering are switched in accordance with the value of z of the frame coordinates.
1. An image processing device comprising:
a moving image data memory unit configured to store hierarchical moving image data obtained by hierarchizing, in order of resolution, a plurality of image frame sequences that represent image frames constituting a single moving image in different resolutions;
an input information acquisition unit configured to sequentially acquire user operation input information related to a display area; and
a display image processing unit configured to generate a moving image to be displayed on a display device by using the hierarchical moving image data, while changing an area of the moving image to be displayed in accordance with the operation input information acquired by the input information acquisition unit,
wherein the display image processing unit generates the moving image while switching a hierarchical layer, used to generate a moving image in the hierarchical moving image data, in accordance with a change in resolution determined by the operation input information,
wherein the display image processing unit executes a prefetch process in which the display image processing unit sequentially loads, in advance, hierarchical moving image data that is closely related in a spatial and temporal manner to an image frame that is currently displayed, and
wherein the display image processing unit decodes a part of the loaded hierarchical moving image data and stores the decoded data in a memory buffer,
wherein the hierarchical moving image data to be loaded and subsequently decoded is determined by predicting a display area that will become necessary to display based on earlier movement of the display area in accordance with the operation input information acquired by the input information acquisition unit, and
wherein determining the predicted display area includes setting in advance as a prefetch boundary a scale factor of a layer of the hierarchical moving image data, such that when a boundary of the display area related to the acquired user operation input information exceeds the prefetch boundary, the prefetch process is executed.
2. The image processing device according to claim 1 ,
wherein an image frame sequence, constituting each hierarchical layer of the hierarchical moving image data, is constituted by a moving image stream in units of tile images obtained by dividing each image frame by a predetermined size, and
wherein the display image processing unit reads out, from the moving image data memory unit, data of a moving image stream corresponding to an area to be displayed in accordance with the operation input information, so as to generate a moving image.
3. The image processing device according to claim 1 ,
wherein an image frame sequence constituting each hierarchical layer of the hierarchical moving image data is constituted by a moving image stream, which is compression coded in units of a predetermined number of tile images from the predetermined number of image frames, the tile images being obtained by dividing each image frame by a predetermined size, and
wherein the display image processing unit reads out, from the moving image data memory unit, data of a corresponding moving image stream in accordance with the operation input information and with the passage of time, so as to generate a moving image.
4. The image processing device according to claim 2 ,
wherein the hierarchical moving image data includes a hierarchical layer in which the moving image stream corresponding to a part of an area of an image frame constituting the single moving image is eliminated, and
wherein the display image processing unit generates a moving image by changing a scale of a moving image stream of another hierarchical layer instead of the moving image stream eliminated in the hierarchical layer used to generate the moving image.
5. The image processing device according to claim 2 , wherein the hierarchical moving image data includes a hierarchical layer in which the frame rate of the moving image stream corresponding to a part of an area of an image frame constituting the single moving image is different from the frame rate of the moving image stream corresponding to another area.
6. The image processing device according to claim 1 ,
wherein the moving image data memory unit stores a plurality of items of hierarchical moving image data generated in units of a plurality of moving image blocks obtained by temporally dividing the single moving image, and
wherein the display image processing unit sequentially reads out, from the moving image data memory unit, the hierarchical moving image data in accordance with the passage of time so as to generate a moving image.
7. An image processing method comprising:
reading out hierarchical moving image data obtained by hierarchizing, in order of resolution, a plurality of image frame sequences that represent image frames constituting a single moving image in different resolutions, so as to generate a moving image to be displayed on a display device by using the hierarchical moving image data;
acquiring user operation input information related to a display area; and
changing, in accordance with the operation input information, an area of the moving image that is to be displayed,
wherein the changing of the area of the moving image includes generating the moving image while switching a hierarchical layer used to generate a moving image in the hierarchical moving image data in accordance with a change in resolution determined by the operation input information,
wherein the changing of the area of the moving image includes executing a prefetch process in which is sequentially loaded, in advance, hierarchical moving image data that is closely related in a spatial and temporal manner to an image frame that is currently displayed, and
wherein the changing of the area of the moving image includes decoding a part of the loaded hierarchical moving image data and storing the decoded data in a memory buffer,
wherein the hierarchical moving image data to be loaded and subsequently decoded is determined by predicting a display area that will become necessary to display based on earlier movement of the display area in accordance with the acquired operation input information, and
wherein determining the predicted display area includes setting in advance as a prefetch boundary a scale factor of a layer of the hierarchical moving image data, such that when a boundary of the display area related to the acquired user operation input information exceeds the prefetch boundary, the prefetch process is executed.
8. A computer program embedded in a non-transitory computer-readable recording medium, comprising:
a module configured to read out hierarchical moving image data obtained by hierarchizing, in order of resolution, a plurality of image frame sequences that represent image frames constituting a single moving image in different resolutions, so as to generate a moving image to be displayed on a display device by using the hierarchical moving image data;
a module configured to acquire user operation input information related to a display area; and
a module configured to change, in accordance with the operation input information, an area of the moving image that is to be displayed,
wherein the module configured to change the area of the moving image includes a module configured to generate the moving image while switching a hierarchical layer used to generate a moving image in the hierarchical moving image data in accordance with a change in resolution determined by the operation input information,
wherein the module configured to generate the moving image executes a prefetch process in which the module configured to generate the moving image sequentially loads, in advance, hierarchical moving image data that is closely related in a spatial and temporal manner to an image frame that is currently displayed, and
wherein the module configured to generate the moving image decodes a part of the loaded hierarchical moving image data and stores the decoded data in a memory buffer,
wherein the hierarchical moving image data to be loaded and subsequently decoded is determined by predicting a display area that will become necessary to display based on earlier movement of the display area in accordance with the acquired operation input information, and
wherein determining the predicted display area includes setting in advance as a prefetch boundary a scale factor of a layer of the hierarchical moving image data, such that when a boundary of the display area related to the acquired user operation input information exceeds the prefetch boundary, the prefetch process is executed.
9. A non-transitory computer-readable recording medium having embodied thereon a computer program product comprising:
a module configured to read out hierarchical moving image data obtained by hierarchizing, in order of resolution, a plurality of image frame sequences that represent image frames constituting a single moving image in different resolutions, so as to generate a moving image to be displayed on a display device by using the hierarchical moving image data;
a module configured to acquire user operation input information related to a display area; and
a module configured to change, in accordance with the operation input information, an area of the moving image that is to be displayed,
wherein the module configured to change the area of the moving image includes a module configured to generate the moving image while switching a hierarchical layer used to generate a moving image in the hierarchical moving image data in accordance with a change in resolution determined by the operation input information,
wherein the module configured to generate executes a prefetch process in which the module configured to generate sequentially loads, in advance, hierarchical moving image data that is closely related in a spatial and temporal manner to an image frame that is currently displayed, and
wherein the module configured to generate decodes a part of the loaded hierarchical moving image data and stores the decoded data in a memory buffer,
wherein the hierarchical moving image data to be loaded and subsequently decoded is determined by predicting a display area that will become necessary to display based on earlier movement of the display area in accordance with the acquired operation input information, and
wherein determining the predicted display area includes setting in advance as a prefetch boundary a scale factor of a layer of the hierarchical moving image data, such that when a boundary of the display area related to the acquired user operation input information exceeds the prefetch boundary, the prefetch process is executed.
10. A non-transitory computer-readable recording medium having embodied thereon a moving image file for display on a display device, the moving image file having a data structure that maps a resolution determined by user operation input related to a display area to image frame sequences used by switching according to the resolution, the image frame sequences being a plurality of image frame sequences that respectively represent image frames constituting a single moving image in different resolutions, the switching occurring while generating the moving image,
generating the moving image including executing a prefetch process in which is sequentially loaded, in advance, hierarchical moving image data that is closely related in a spatial and temporal manner to an image frame that is currently displayed, and
generating the moving image further including decoding a part of the loaded hierarchical moving image data and storing the decoded data in a memory buffer,
the hierarchical moving image data to be loaded and subsequently decoded being determined by predicting a display area that will become necessary to display based on earlier movement of the display area in accordance with the user operation input,
determining the predicted display area including setting in advance as a prefetch boundary a scale factor of a layer of the hierarchical moving image data, such that when a boundary of the display area related to the user operation input exceeds the prefetch boundary, the prefetch process is executed.
11. The non-transitory computer-readable recording medium according to claim 10 , wherein each of the plurality of image frame sequences is constituted by a moving image stream in units of tile images obtained by dividing each image frame by a predetermined size.