Calibrated sensitivity model approximating the eye
In one embodiment, a method includes projecting a source image onto a surface using a lens approximation component, where the surface is associated with sampling points approximating photoreceptors of an eye, where each sampling point has a corresponding photoreceptor type, sampling color information from the projected source image at the sampling points, where the color information sampled at each sampling point depends on the corresponding photoreceptor type, accessing pooling units approximating retinal ganglion cells (RGCs) of the eye, where each pooling unit is associated with groups of one or more of the sampling points, calculating weighted aggregations of the sampled color information associated with the groups of one or more sampling points associated with each pooling unit, and computing a perception profile for the source image based on the weighted aggregations associated with each of the pooling units.
1. A method comprising:
projecting a source image onto a surface using a lens approximation component, wherein the surface is associated with sampling points approximating photoreceptors of an eye, wherein each sampling point has a corresponding photoreceptor type, and wherein the lens approximation component projects the source image onto a curved shape surface and maps the source image projected onto the curved shape surface into an image on the surface;
sampling color information from the projected source image at the sampling points, wherein the color information sampled at each sampling point depends on the corresponding photoreceptor type;
accessing pooling units approximating retinal ganglion cells (RGCs) of the eye, wherein each pooling unit is associated with groups of one or more of the sampling points;
calculating, for each of the pooling units, weighted aggregations of the sampled color information associated with the groups of one or more sampling points associated with that pooling unit; and
computing a perception profile for the source image based on the weighted aggregations associated with each of the pooling units.
2. The method of claim 1 , wherein the surface approximates a retina of the eye.
3. The method of claim 2 , wherein the lens approximation component comprises an Optical Transfer Function (OTF) and a warping operator, wherein the OTF convolves a matrix representing pixel information of the source image with convolution templates and multiplies with a coding matrix, wherein the convolution templates and the coding matrix are obtained by a matrix factorization, and wherein the warping operator maps an image on a concave shape of the retina to an image in the surface.
4. The method of claim 1 , wherein a distribution of a type of the sampling points is associated with a corresponding blue noise mask.
5. The method of claim 1 , wherein each pooling unit has a corresponding RGC type.
6. The method of claim 5 , wherein a distribution of a type of the pooling units is associated with a corresponding blue noise mask.
7. The method of claim 1 , wherein a group of one or more of the sampling points associated with a pooling unit is determined based on a field weighting function.
8. The method of claim 7 , wherein the field weighting function is a Gaussian receptive field function.
9. The method of claim 1 , wherein the source image corresponds to a frame of a video stream.
10. The method of claim 1 , wherein the source image can be of any resolution, and wherein a resolution of the surface is pre-determined.
11. The method of claim 1 , wherein the perception profile for the source image is computed using Rectified Linear Unit (ReLU) activation functions.
12. The method of claim 1 , further comprising:
accessing an output image of a machine-learning model as a first source image;
computing a first perception profile for the first source image;
accessing a ground truth image corresponding to the output image of the machine-learning model as a second source image;
computing a second perception profile for the second source image;
measuring differences between the first perception profile and the second perception profile; and
updating trainable variables of the machine-learning model by performing a gradient-descent backpropagation procedure with gradients computed based on the measured differences.
13. The method of claim 12 , wherein the machine-learning model is a foveated-rendering machine-learning model.
14. The method of claim 12 , wherein the machine-learning model is a metamers-generating machine-learning model.
15. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
project a source image onto a surface using a lens approximation component, wherein the surface is associated with sampling points approximating photoreceptors of an eye, wherein each sampling point has a corresponding photoreceptor type, and wherein the lens approximation component projects the source image onto a curved shape surface and maps the source image projected onto the curved shape surface into an image on the surface;
sample color information from the projected source image at the sampling points, wherein the color information sampled at each sampling point depends on the corresponding photoreceptor type;
access pooling units approximating retinal ganglion cells (RGCs) of the eye, wherein each pooling unit is associated with groups of one or more of the sampling points;
calculate, for each of the pooling units, weighted aggregations of the sampled color information associated with the groups of one or more sampling points associated with that pooling unit; and
compute a perception profile for the source image based on the weighted aggregations associated with each of the pooling units.
16. The media of claim 15 , wherein the surface approximates a retina of the eye.
17. The media of claim 16 , wherein the lens approximation component comprises an Optical Transfer Function (OTF) and a warping operator, wherein the OTF convolves a matrix representing pixel information of the source image with convolution templates and multiplies with a coding matrix, wherein the convolution templates and the coding matrix are obtained by a matrix factorization, and wherein the warping operator maps an image on a concave shape of the retina to an image in the surface.
18. The media of claim 15 , wherein a distribution of a type of the sampling points is associated with a corresponding blue noise mask.
19. The media of claim 15 , wherein each pooling unit has a corresponding RGC type.
20. A system comprising: one or more processors; and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to:
project a source image onto a surface using a lens approximation component, wherein the surface is associated with sampling points approximating photoreceptors of an eye, wherein each sampling point has a corresponding photoreceptor type, and wherein the lens approximation component projects the source image onto a curved shape surface and maps the source image projected onto the curved shape surface into an image on the surface;
sample color information from the projected source image at the sampling points, wherein the color information sampled at each sampling point depends on the corresponding photoreceptor type;
access pooling units approximating retinal ganglion cells (RGCs) of the eye, wherein each pooling unit is associated with groups of one or more of the sampling points;
calculate, for each of the pooling units, weighted aggregations of the sampled color information associated with the groups of one or more sampling points associated with that pooling unit; and
compute a perception profile for the source image based on the weighted aggregations associated with each of the pooling units.