Learned image transformation methods and systems in graphics rendering

1 . A method of transforming rendered frames in a graphics processing system to obtain enhanced frames comprising one or more desired characteristics exhibited in a set of target images, the method comprising:

selecting a plurality of trained shaders, each trained shader defined by a parametrized mathematical function arranged to replicate a particular visual characteristic, wherein, for each trained shader of the plurality of trained shaders: parameters of the parametrized mathematical function have been derived in dependence on a set of target images in order that the trained shader is arranged to impose, when applied to a frame, its respective particular visual characteristic in dependence on an extent to which the particular visual characteristic is exhibited in the set of target images, the parameters of the parametrized mathematical function thereby being trained and predetermined parameters;

combining the plurality of trained shaders to form a frame transformation pipeline;

obtaining one or more rendered frames;

applying each trained shader of the frame transformation pipeline to at least a portion of each of the one or more rendered frames to obtain one or more enhanced frames; and

outputting for display each of the one or more enhanced frames, wherein the enhanced frames exhibit the visual characteristics of the set of target images.

2 . The method of claim 1 , wherein the parameters of the parametrized mathematical function have been derived using a neural network based on an adversarial objective function.

3 . The method of claim 2 , wherein the neural network is a generative adversarial network (GAN), comprising an image frame generator configured to produce candidate frames, and a discriminator configured to calculate a probability that the candidate frames produced by the image frame generator belong in the set of target images.

4 . The method of claim 3 , wherein the image frame generator comprises one or more precursor shaders, and wherein the GAN is configured to derive parameters for the one or more precursor shaders, in dependence on the set of target images, wherein the derived parameters represent parameters defining one or more parametrized mathematical functions that define a respective one or more trained shaders that form the frame transformation pipeline.

5 . The method of claim 1 , wherein each parametrized mathematical function represents an image-capture characteristic, and wherein the particular visual characteristic each trained shader is arranged to replicate is a physical phenomenon associated with an image-capture process.

6 . The method of claim 5 , wherein applying each trained shader comprises applying each trained shader in an order corresponding to an order in which the physical phenomena associated with an image-capture process would occur.

7 . The method of claim 1 , wherein each of the plurality of trained shaders are substantially mathematically orthogonal, such that each respective particular visual characteristic imposed by each trained shader in the one or more enhanced frames is substantially not affected by an imposed particular visual characteristic of any other trained shader of the plurality of trained shaders.

8 . The method of claim 1 , wherein the frame transformation pipeline comprises at least one neural network configured to further enhance frames based on a desired visual characteristic.

9 . The method of claim 1 , wherein the frame transformation pipeline comprises a parameter-adapting neural network, the method further comprising, prior to applying the trained shaders of the frame transformation pipeline to at least a portion of each of the one or more rendered frames:

modifying, by the parameter-adapting neural network, one or more parameters of the one or more trained shaders in dependence on the one or more rendered frames.

10 . The method of claim 1 , wherein combining the plurality of trained shaders to form a frame transformation pipeline comprises coalescing one or more of the trained shaders of the plurality of trained shaders into a combined trained shader unit that is arranged to replicate each respective particular visual characteristic of the one or more coalesced trained shaders.

11 . The method of claim 1 , wherein one of the plurality of trained shaders is a trained lens blur shader configured to replicate lens blur.

12 . The method of claim 11 , wherein the parametrized mathematical function comprises at least one kernel comprising an array of values, wherein applying the trained lens blur shader comprises convolving the at least one kernel over at least a portion of an array of values representing pixels of the one or more rendered frames.

13 . The method of claim 1 , wherein one of the plurality of trained shaders is a trained colour mapping shader configured to replicate colour mapping and/or distortion of the set of target images, wherein applying the colour mapping comprises applying a linear transformation and an offset to an array of values representing pixels of the one or more rendered frames.

14 . The method of claim 1 , wherein one of the plurality of trained shaders is a trained bloom shader configured to replicate the effect of light bleeding due to oversaturation in an image-capture system.

15 . The method of claim 14 , wherein applying the trained bloom shader to an input frame of the one or more rendered frames comprises:

downsampling the input frame to obtain a plurality of sub-frames each having a lower image resolution than a resolution of the input frame;

for each sub-frame:

extracting a luma channel,

isolating portions of the extracted luma channel above a brightness threshold, and

applying a blurring function to the isolated portions to obtain a bloomed sub-frame;

rescaling and combining each of the obtained bloomed sub-frames to obtain a bloom mask having the resolution of the input frame; and

combining the bloom mask with the input frame.

16 . The method of claim 1 , wherein one of the plurality of trained shaders is a trained sensor noise shader configured to replicate noise of a digital camera sensor or grain of a film camera, wherein applying the trained sensor noise shader comprises applying randomly distributed perturbations to values representing pixels of the one or more rendered frames.

17 . The method of claim 1 , wherein each trained shader of the plurality of trained shaders is a post-processing shader, and wherein the parametrized mathematical function defining each trained shader operates on pixel values of the portions of each of the one or more rendered frames during the step of applying each trained shader of the frame transformation pipeline.

18 . The method of claim 1 , wherein applying each trained shader of the frame transformation pipeline to the at least a portion of each of the one or more rendered frames comprises, for each portion to which the frame transformation pipeline is applied:

applying all trained shaders that form the frame transformation pipeline to a plurality of contiguous regions of the portion, wherein all trained shaders are applied to each region in turn, to obtain a plurality of enhanced sub-frames; and

combining the enhanced sub-frames to obtain an enhanced frame of the one or more enhanced frames.

19 . A frame transformation pipeline module configured to transform rendered frames in a graphics processing system to obtain enhanced frames comprising one or more desired characteristics exhibited in a set of target images, the frame transformation pipeline module comprising logic configured to:

select a plurality of trained shaders, each trained shader defined by a parametrized mathematical function arranged to replicate a particular visual characteristic, wherein, for each trained shader of the plurality of trained shaders: parameters of the parametrized mathematical function have been derived in dependence on a set of target images in order that the trained shader is arranged to impose, when applied to a frame, its respective particular visual characteristic in dependence on an extent to which the particular visual characteristic is exhibited in the set of target images, the parameters of the parametrized mathematical function thereby being trained and predetermined parameters;

combine the plurality of trained shaders to form a frame transformation pipeline;

obtain one or more rendered frames;

apply each trained shader of the frame transformation pipeline to at least a portion of each of the one or more rendered frames to obtain one or more enhanced frames; and

output for display each of the one or more enhanced frames, wherein the enhanced frames exhibit the visual characteristics of the set of target images.

20 . A non-transitory computer readable storage medium having stored thereon a computer readable dataset description of an integrated circuit definition dataset that, when processed in an integrated circuit manufacturing system, configures the integrated circuit manufacturing system to manufacture a frame transformation pipeline module configured to transform rendered frames in a graphics processing system to obtain enhanced frames comprising one or more desired characteristics exhibited in a set of target images, the frame transformation pipeline module comprising logic configured to:

select a plurality of trained shaders, each trained shader defined by a parametrized mathematical function arranged to replicate a particular visual characteristic, wherein, for each trained shader of the plurality of trained shaders: parameters of the parametrized mathematical function have been derived in dependence on a set of target images in order that the trained shader is arranged to impose, when applied to a frame, its respective particular visual characteristic in dependence on an extent to which the particular visual characteristic is exhibited in the set of target images, the parameters of the parametrized mathematical function thereby being trained and predetermined parameters;

combine the plurality of trained shaders to form a frame transformation pipeline;

obtain one or more rendered frames;

apply each trained shader of the frame transformation pipeline to at least a portion of each of the one or more rendered frames to obtain one or more enhanced frames; and

output for display each of the one or more enhanced frames, wherein the enhanced frames exhibit the visual characteristics of the set of target images.