Judder detection for dynamic frame rate conversion
One embodiment provides a method comprising generating, via an edge detection algorithm, a first edge map based on a first frame of an input content comprising a sequence of frames. The method further comprises generating, via the edge detection algorithm, a second edge map based on a second frame of the input content. The first frame precedes the second frame in the sequence of frames. The method further comprises determining a difference between the first edge map and the second edge map, determining a metric indicative of an estimated amount of judder present in the input content based on the difference, and dynamically adjusting a frame rate of the input content based on the metric. The input content is displayed on a display device at the adjusted frame rate.
1. A method comprising:
generating, via an edge detection algorithm, a first set of edge maps based on a first set of subsampled images of a plurality of different resolutions, wherein the first set of subsampled images is sampled from a first frame of an input content comprising a sequence of frames;
generating, via the edge detection algorithm, a second set of edge maps based on a second set of subsampled images of the plurality of different resolutions, wherein the second set of subsampled images is sampled from a second frame of the input content, and the first frame precedes the second frame in the sequence of frames;
determining a set of differences between the first set of edge maps and the second set of edge maps;
determining a metric indicative of an estimated amount of judder present in the input content based on the set of differences and a set of different weights corresponding to the plurality of different resolutions; and
dynamically adjusting a frame rate of the input content by selecting, based on the metric, an optimal frame rate from an optimal frame rate table, wherein the optimal frame rate table maps at least one estimated amount of judder to at least one optimal frame rate, the at least one optimal frame rate is based on one or more preferences, and the input content is displayed on a display device at the selected optimal frame rate as output content;
wherein a first amount of judder present in at least a set of the sequence of frames is different than a second amount of judder present in a corresponding set of a sequence of frames of the output content.
2. The method of claim 1 , wherein the input content comprises a portion of a video, the one or more preferences comprises one or more preferences of a content creator of the input content or one or more preferences of a user of the display device, and the second amount of judder present in the corresponding set of the sequence of frames of the output content maintains a creative intent of the content creator of the input content.
3. The method of claim 2 , wherein each portion of each frame comprises one of an entirety of the frame or a portion of the frame that is smaller than the entirety of the frame.
4. The method of claim 1 , further comprising:
generating the first set of subsampled images by sampling a portion of the first frame at the plurality of different resolutions; and
generating the second set of subsampled images by sampling a portion of the second frame at the plurality of different resolutions.
5. The method of claim 4 , wherein the plurality of different resolutions comprises: a first resolution equal to resolution of the input content, and one or more other resolutions that are smaller than the first resolution.
6. The method of claim 1 , wherein determining the metric comprises:
determining weighted differences based on the set of differences and the set of different weights.
7. The method of claim 6 , wherein the metric is a sum of the weighted differences.
8. The method of claim 6 , wherein the metric is a mean of the weighted differences.
9. The method of claim 1 , wherein dynamically adjusting the frame rate of the input content comprises:
triggering an increase in the frame rate if the metric exceeds a pre-determined threshold.
10. The method of claim 1 , wherein dynamically adjusting the frame rate of the input content comprises:
stopping an increase in the frame rate if the metric does not exceed the pre-determined threshold.
11. A system comprising:
at least one processor; and
a non-transitory processor-readable memory device storing instructions that when executed by the at least one processor causes the at least one processor to perform operations including:
generating, via an edge detection algorithm, a first set of edge maps based on a first set of subsampled images of a plurality of different resolutions, wherein the first set of subsampled images is sampled from a first frame of an input content comprising a sequence of frames;
generating, via the edge detection algorithm, a second set of edge maps based on a second set of subsampled images of the plurality of different resolutions, wherein the second set of subsampled images is sampled from a second frame of the input content, and the first frame precedes the second frame in the sequence of frames;
determining a set of differences between the first set of edge maps and the second set of edge maps;
determining a metric indicative of an estimated amount of judder present in the input content based on the set of differences and a set of different weights corresponding to the plurality of different resolutions; and
dynamically adjusting a frame rate of the input content by selecting, based on the metric, an optimal frame rate from an optimal frame rate table, wherein the optimal frame rate table maps at least one estimated amount of judder to at least one optimal frame rate, the at least one optimal frame rate is based on creative intent of a content creator of the input content, and the input content is displayed on a display device at the selected optimal frame rate as output content;
wherein a first amount of judder present in at least a set of the sequence of frames is different than a second amount of judder present in a corresponding set of a sequence of frames of the output content.
12. The system of claim 11 , wherein the input content comprises a portion of a video, and the second amount of judder present in the corresponding set of the sequence of frames of the output content maintains the creative intent of the content creator of the input content.
13. The system of claim 12 , wherein each portion of each frame comprises one of an entirety of the frame or a portion of the frame that is smaller than the entirety of the frame.
14. The system of claim 11 , wherein the operations further comprise:
generating the first set of subsampled images by sampling a portion of the first frame at the plurality of different resolutions; and
generating the second set of subsampled images by sampling a portion of the second frame at the plurality of different resolutions.
15. The system of claim 14 , wherein the plurality of different resolutions comprises: a first resolution equal to resolution of the input content, and one or more other resolutions that are smaller than the first resolution.
16. A non-transitory processor-readable medium that includes a program that when executed by a processor performs a method comprising:
generating, via an edge detection algorithm, a first set of edge maps based on a first set of subsampled images of a plurality of different resolutions, wherein the first set of subsampled images is sampled from a first frame of an input content comprising a sequence of frames;
generating, via the edge detection algorithm, a second set of edge maps based on a second set of subsampled images of the plurality of different resolutions, wherein the second set of subsampled images is sampled from a second frame of the input content, and the first frame precedes the second frame in the sequence of frames;
determining a set of differences between the first set of edge maps and the second set of edge maps;
determining a metric indicative of an estimated amount of judder present in the input content based on the set of differences and a set of different weights corresponding to the plurality of different resolutions; and
dynamically adjusting a frame rate of the input content by selecting, based on the metric, an optimal frame rate from an optimal frame rate table, wherein the optimal frame rate table maps at least one estimated amount of judder to at least one optimal frame rate, the at least one optimal frame rate is based on creative intent of a content creator of the input content, and the input content is displayed on a display device at the selected optimal frame rate as output content;
wherein a first amount of judder present in at least a set of the sequence of frames is different than a second amount of judder present in a corresponding set of a sequence of frames of the output content.
17. The method of claim 16 , wherein the second amount of judder present in the corresponding set of the sequence of frames of the output content maintains the creative intent of the content creator of the input content.