Image data transfer apparatus and image compression
An attention degree estimation section in a compression coding section of a server estimates an attention degree on the basis of contents indicated by a moving image generated by an image forming section, for each of unit regions produced by dividing a frame plane. A compression coding processing section compression-codes the moving image at a compression rate varied in the frame plane according to a distribution of the attention degrees. A communication section transmits compression-coded data to an image processing apparatus.
1. An image data transfer apparatus comprising:
a drawing section that draws a moving image to be displayed;
an image contents acquisition section that acquires information associated with contents indicated by the moving image;
an attention degree estimation section that estimates an attention degree indicating a degree of attention paid by a user, on a basis of the contents indicated by the moving image, for each of equally sized unit regions produced by dividing a frame plane of the moving image at predetermined intervals;
a compression coding section that compression-codes data of the moving image at a compression rate varied in the frame plane according to a distribution of the attention degrees; and a communication section that transmits compression-coded data of the moving image by streaming, wherein:
the attention degree estimation section updates the attention degree each time new gaze point position information is obtained, and
the distribution of the attention degrees is changed during a compression coding process for a frame of the moving image, such that when the attention degree is updated after a start of compression coding processing for the frame, the compression coding section compression-codes a next partial image of the frame at a compression rate based on the updated attention degree.
2. The image data transfer apparatus according to claim 1 , wherein the drawing section dynamically draws a moving image that is to be transmitted and has not been present before.
3. The image data transfer apparatus according to claim 1 , wherein the image contents acquisition section acquires the information associated with the contents indicated by the moving image on a basis of at least either a notification from the drawing section or an analysis result of the drawn moving image.
4. The image data transfer apparatus according to claim 3 , wherein the image contents acquisition section uses, as the analysis result of the moving image, information acquired in a compression coding process by the compression coding section.
5. The image data transfer apparatus according to claim 1 , wherein the image contents acquisition section acquires, as the information associated with the contents, at least any one of a timing of a user operation associated with content that specifies the moving image, an interval of the timing, contents of the user operation, a status of the content, and a status of audio generated according to the content.
6. The image data transfer apparatus according to claim 5 , wherein the image contents acquisition section acquires, as the status of the content, from the drawing section, information for determining at least any one of a) whether a current scene is a scene requiring a user operation and is one influencing processing contents of the content, b) whether the current scene is a movie scene not requiring a user operation, c) whether the current scene is a scene other than a movie scene and is one not requiring a user operation, and d) whether the current scene is a scene requiring a user operation and is one not included in an original story of the content.
7. The image data transfer apparatus according to claim 1 , wherein the attention degree estimation section estimates the attention degree on a basis of at least any one of an amount of an optical flow, an amount of motion estimation, an allocation status of a coding unit, whether or not a current timing is a scene switching timing, a type of image texture expressed in a frame, a feature point, depth information, an amount of objects, a use amount of each level of mipmap texture used for three-dimensional graphics, an LOD (Level Of Detail), a use amount of each level of tessellation, an amount of characters and symbols, a type of an expressed scene, and a type of an expressed object.
8. The image data transfer apparatus according to claim 1 , further comprising:
a communication status acquisition section that obtains a status of communication used for streaming of the moving image, by acquiring at least either a data arrival rate or a delay time period of the moving image at an apparatus corresponding to a transmission destination at predetermined time intervals, wherein
the compression coding section determines a distribution of quantization parameters of the frame plane on a basis of a combination of the attention degree, the communication status, a frame rate, and a resolution.
9. The image data transfer apparatus according to claim 1 , wherein the attention degree estimation section determines scores each indicating the attention degree and reflecting the contents indicated by the moving image, for each of the unit regions under a plurality of rules, and determines a compression rate for each of the unit regions on a basis of a sum of the scores.
10. The image data transfer apparatus according to claim 9 , wherein the attention degree estimation section gives a score higher than those for other unit regions to a unit region that contains an object meeting a condition associated with a degree of importance.
11. The image data transfer apparatus according to claim 9 , wherein, when an input value of a user operation that is associated with content specifying the moving image changes by a predetermined amount or larger in an immediately preceding predetermined period, the attention degree estimation section amplifies a score of a corresponding unit region to which a score of a predetermined value or higher has been given under a different rule.
12. The image data transfer apparatus according to claim 1 , wherein the compression coding section determines the compression rate under a rule optimized by machine learning or deep learning.
13. The image data transfer apparatus according to claim 1 , wherein the compression coding section determines a distribution of the compression rates in the frame plane on a basis of a data size of an entire frame.
14. The image data transfer apparatus according to claim 1 , wherein the compression coding section compression-codes the moving image in units of a partial image smaller than one frame and updates a distribution of the compression rates in units of any one of the partial image, one frame, and a predetermined number of frames or at predetermined time intervals.
15. The image data transfer apparatus according to claim 1 , wherein
the communication section sequentially transmits data compression-coded by the compression coding section for each of compression units produced by dividing the frame plane, and,
in a case where any one of the compression units contains a unit region requiring lowering of the compression rate, the compression coding section lowers the compression rate of the unit region.
16. An image compression method performed by an image data transfer apparatus, the image compression method comprising:
drawing a moving image to be displayed;
acquiring information associated with contents indicated by the moving image;
estimating an attention degree indicating a degree of attention paid by a user, on a basis of the contents indicated by the moving image, for each of equally sized unit regions produced by dividing a frame plane of the moving image at predetermined intervals;
compression-coding data of the moving image at a compression rate varied in the frame plane according to a distribution of the attention degrees; and
transmitting compression-coded data of the moving image by streaming, wherein:
the attention degree is updated each time new gaze point position information is obtained, and
the distribution of the attention degrees is changed during a compression coding process for a frame of the moving image, such that when the attention degree is updated after a start of compression coding processing for the frame, the compression coding section compression-codes a next partial image of the frame at a compression rate based on the updated attention degree.
17. A non-transitory, computer readable storage medium containing a computer program, which when executed by a computer, causes the computer to perform an image compression method performed by an image transfer apparatus, by carrying out actions, comprising:
drawing a moving image to be displayed;
acquiring information associated with contents indicated by the moving image;
estimating an attention degree indicating a degree of attention paid by a user, on a basis of the contents indicated by the moving image, for each of equally sized unit regions produced by dividing a frame plane of the moving image at predetermined intervals;
compression-coding data of the moving image at a compression rate varied in the frame plane according to a distribution of the attention degrees; and transmitting compression-coded data of the moving image by streaming, wherein:
the attention degree is updated each time new gaze point position information is obtained, and
the distribution of the attention degrees is changed during a compression coding process for a frame of the moving image, such that when the attention degree is updated after a start of compression coding processing for the frame, the compression coding section compression-codes a next partial image of the frame at a compression rate based on the updated attention degree.