Image rendering method and apparatus
An image rendering method for a virtual scene includes, for a plurality of IDs, generating a respective mask identifying elements of the scene that are associated with a respective ID; for the resulting plurality of masks, dividing a respective mask into a plurality of tiles; and discarding tiles that do not identify any image elements; for the resulting plurality of remaining tiles, selecting a respective trained machine learning model from among a plurality of machine learning models, the respective machine learning model having been trained to generate data contributing to a render of at least a part of an image, based upon elements of the scene associated with the same respective ID as the elements identified in the mask from which the respective tile was divided; and using the respective trained machine learning model to generate data contributing to a render of at least a part of the image based upon input data at least for the identified elements in the respective tile.
1. An image rendering method for a virtual scene, comprising the steps of:
for a plurality of identifiers, IDs,
generating a respective pixel mask identifying elements of the scene that are associated with a respective ID;
for a resulting plurality of pixel masks,
dividing a respective pixel mask into a plurality of tiles; and
discarding tiles that do not identify any image elements;
for the resulting plurality of remaining tiles,
selecting a respective trained machine learning model from among a plurality of machine learning models, the respective machine learning model having been trained to generate data contributing to a render of at least a part of an image, based upon the same respective ID associated with the elements identified in the pixel mask from which the respective tile was divided; and
using the respective trained machine learning model to generate data contributing to a render of at least a part of the image based upon input data at least for the identified elements in the respective tile.
2. An image rendering method according to claim 1 , in which a size of the tiles is selected according to a capability of a processing hardware.
3. An image rendering method according to claim 1 , in which a batch of tiles is processed in parallel by a plurality of respective trained machine learning models.
4. An image rendering method according to claim 3 , in which a size of the batch is selected according to a capability of a processing hardware.
5. An image rendering method according to claim 1 , in which the respective trained machine learning model generates data contributing to a render of at least a part of the image based upon one of:
input data only for the identified elements in the respective tile; and
input data for the whole respective tile.
6. An image rendering method according to claim 1 , in which whether the respective trained machine learning model generates data contributing to a render of at least a part of the image based upon input data only for the identified elements in the respective tile, or based upon input data for the whole respective tile, depends upon whether the identified elements in the respective tile meet an occupancy criterion.
7. The image rendering method of claim 1 , in which the generated data comprises a factor that, when combined with a distribution function that characterizes an interaction of light with a respective part of the virtual scene, generates a pixel value corresponding to a pixel of a rendered image comprising that respective part of the virtual scene.
8. The image rendering method of claim 7 , in which
a respective trained machine learning system is trained for each of a plurality of contributing components of the image;
a respective distribution function is used for each of the plurality of contributing components of the image; and
the respective generated pixel values are combined to create a final combined pixel value incorporated into the rendered image for display.
9. The image rendering method of claim 7 , in which
material properties of the element can be obtained with reference to the respective ID; and
at least a first respective distribution function corresponding to the respective ID is obtained by one of:
retrieving the distribution function from a storage; and
calculating the distribution function in parallel with the use of at least a first respective trained machine learning model to generate data.
10. The image rendering method of claim 1 , in which:
the respective trained machine learning system is a neural network;
an input to a first portion of the neural network comprises a position of an element within the virtual scene; and
an input a second portion of the neural network comprises an output of the first portion and a direction based on a viewpoint of at least part of the image being rendered.
11. A non-transitory, computer readable storage medium containing a computer program comprising computer executable instructions, which when executed by a computer system, causes the computer system to perform an image rendering method for a virtual scene by carrying out actions, comprising:
for a plurality of identifiers, IDs,
generating a respective pixel mask identifying elements of the scene that are associated with a respective ID;
for a resulting plurality of pixel masks,
dividing a respective pixel mask into a plurality of tiles; and
discarding tiles that do not identify any image elements;
for the resulting plurality of remaining tiles,
selecting a respective trained machine learning model from among a plurality of machine learning models, the respective machine learning model having been trained to generate data contributing to a render of at least a part of an image, based upon the same respective ID associated with the elements identified in the pixel mask from which the respective tile was divided; and
using the respective trained machine learning model to generate data contributing to a render of at least a part of the image based upon input data at least for the identified elements in the respective tile.
12. An entertainment device operable to render an image of a virtual scene, and comprising:
a pixel mask processor adapted, for a plurality of identifiers, IDs, to generate a respective pixel mask identifying elements of the scene that are associated with a respective ID;
a tile processor adapted, for a resulting plurality of pixel masks, to divide a respective pixel mask into a plurality of tiles, and discard tiles that do not identify any image elements;
a selection processor adapted, for the plurality of remaining tiles, to select a respective trained machine learning model from among a plurality of machine learning models, the respective machine learning model having been trained to generate data contributing to a render of at least a part of an image, based upon the same respective ID associated with the elements identified in the pixel mask from which the respective tile was divided; and
a render processor adapted to use the respective trained machine learning model to generate data contributing to a render of at least a part of the image based upon input data at least for the identified elements in the respective tile.
13. An entertainment device according to claim 12 , in which one or more is selected according to a capability of a processing hardware from a list consisting of:
a size of the tiles; and
a size of a batch of tiles processed in parallel by a plurality of respective trained machine learning models.
14. An entertainment device according to claim 12 , in which the respective trained machine learning model generates data contributing to a render of at least a part of the image based upon one of:
input data only for the identified elements in the respective tile; and
input data for the whole respective tile.
15. An entertainment device according to claim 12 , in which the render processor is adapted to select whether the respective trained machine learning model generates data contributing to a render of at least a part of the image based upon input data only for the identified elements in the respective tile, or based upon input data for the whole respective tile, depending upon whether the identified elements in the respective tile meet an occupancy criterion.