Trim pass metadata prediction in video sequences using neural networks
Methods and systems for generating trim-pass metadata for high dynamic range (HDR) video are described. The trim-pass prediction pipeline includes a feature extraction network followed by a fully connected network which maps extracted features to trim-pass values. In a first architecture, the feature extraction network is based on four cascaded convolutional networks. In a second architecture, the feature extraction network is based on a modified MobileNetV3 neural network. In both architectures, the fully connected network is formed by a set of three linear networks, each set customized to best match its corresponding feature extraction network.
1 . A method for generating trim-pass metadata of pictures in a video sequence, wherein the trim-pass metadata are configured to perform adjustments of a tone-mapping curve that is applied to an input picture when being displayed on a target display, the method comprising:
receiving the input picture;
providing a feature extraction network, the feature extraction network comprising a convolutional neural network for feature extraction that is trained to identify high-level image features of the input picture;
applying the feature extraction network to the input picture to generate the image features;
providing a fully connected network, the fully connected network comprising a plurality of cascaded linear neural networks that are trained to map the image features to output trim-pass metadata values for the input picture; and
applying the fully connected network to the image features to map the image features to the output trim-pass metadata values for the input picture, wherein training the networks comprises:
receiving input training trim-pass parameters corresponding to the input picture;
applying an error loss unit to generate an error metric based on the input training trim-pass parameters and the output trim-pass metadata; and
training the feature extraction network and the fully connected network by minimizing the error metric.
2 . The method of claim 1 , wherein the input picture is a high-dynamic range (HDR) picture coded using PQ encoding in the ICtCp color space.
3 . The method of claim 1 , wherein the feature extraction network comprises four cascaded convolutional networks.
4 . The method of claim 3 , wherein the fully connected network comprises three cascaded linear networks.
5 . The method of claim 1 , wherein the feature extraction network comprises a modified MobileNetV3 neural network accepting inputs with non square aspect ratios.
6 . The method of claim 5 , wherein the fully connected network comprises three cascaded linear networks.
7 . The method of claim 1 , wherein computing the error metric comprises computing a minimum absolute error or a mean square error between the input training trim-pass parameters and the output trim-pass metadata.
8 . The method of claim 1 , wherein computing the error metric comprises:
generating a first tone-mapping function based at least on the input training trim-pass parameters;
generating a second tone-mapping function based at least on the output trim-pass metadata; and
computing the minimum absolute error or the mean square error between values of the first tone-mapping function and the second tone-mapping function.
9 . An apparatus comprising a processor and configured to perform the method recited in claim 1 .
10 . A non-transitory computer-readable storage medium having stored thereon computer-executable instruction for executing a method with one or more processors in accordance with claim 1 .