Method and apparatus for video frame interpolation, and device and storage medium
View Patent ↗A method and apparatus for video frame interpolation, and a device and a storage medium for the same are provided. The method may include: acquiring a target video, and acquiring a (t−1) th frame of image and a t th frame of image in the target video; acquiring motion level information of pixel points of the (t−1) th frame of image and the t th frame of image; acquiring deep frame interpolation features of the (t−1) th frame of image and the t th frame of image, respectively; and performing a frame interpolation operation layer by layer based on the deep frame interpolation features and the motion level information of the (t−1) th frame of image and the t th frame of image to generate an intermediate frame between the (t−1) th frame of image and the t th frame of image, and interpolating the intermediate frame between the (t−1) th frame of image and the t th frame of image.
1. A method for video frame interpolation, comprising:
acquiring a target video, and acquiring a (t−1) th frame of image and a t th frame of image in the target video, t being a positive integer;
acquiring motion level information of pixel points of the (t−1) th frame of image and the t th frame of image;
acquiring a deep frame interpolation feature of the (t−1) th frame of image and a deep frame interpolation feature of the t th frame of image, respectively; and
performing a frame interpolation operation layer by layer based on the deep frame interpolation features and the motion level information of the (t−1) th frame of image and the t th frame of image to generate an intermediate frame between the (t−1) th frame of image and the t th frame of image, and interpolating the intermediate frame between the (t−1) th frame of image and the t th frame of image,
wherein the acquiring motion level information of pixel points of the (t−1) th frame of image and the t th frame of image comprises:
acquiring a motion level deep feature of the (t−1) th frame of image and a motion level deep feature of the t th frame of image, respectively; and
obtaining the motion level information of the pixel points of the (t−1) th frame of image and the t th frame of image based on the motion level deep feature of the (t−1) th frame of image and the motion level deep feature of the t th frame of image.
2. The method according to claim 1 , wherein the obtaining the motion level information of the pixel points of the (t−1) th frame of image and the t th frame of image based on the motion level deep feature of the (t−1) t frame of image and the motion level deep feature of the t th frame of image comprises:
marking pixels having identical motion attributes, objects having identical motion attributes or areas having identical motion attributes as a given level based on the motion level deep feature of the (t−1) th frame of image and the motion level deep feature of the t th frame of image, to obtain the motion level information of the pixel points of the (t−1) th frame of image and the t th frame of image.
3. The method according to claim 2 , wherein motion levels comprise a motion layer and a stationary layer.
4. The method according to claim 1 , wherein the performing a frame interpolation operation layer by layer based on the deep frame interpolation features and the motion level information of the (t−1) th frame of image and the t th frame of image to generate an intermediate frame between the (t−1) th frame of image and the t th frame of image, and interpolating the intermediate frame between the (t−1) th frame of image and the t th frame of image comprises:
inputting the deep frame interpolation features and the motion level information of the (t−1) th frame of image and the t th frame of image into a pre-trained frame interpolation model, the frame interpolation model performing the frame interpolation operation layer by layer based on the motion level information, to obtain intermediate frames of layers; and
combining the intermediate frames of the layers to obtain the intermediate frame between the (t−1) th frame of image and the t th frame of image, and interpolating the intermediate frame between the (t−1) th frame of image and the t th frame of image.
5. The method according to claim 4 , wherein frame interpolation operations for layers are identical.
6. An electronic device, comprising:
at least one processor; and
a storage device, communicated with the at least one processor,
wherein the storage device stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor, to enable the at least one processor to perform operations comprising:
acquiring a target video, and acquiring a (t−1) th frame of image and a t th frame of image in the target video, t being a positive integer;
acquiring motion level information of pixel points of the (t−1) th frame of image and the t th frame of image;
acquiring a deep frame interpolation feature of the (t−1) th frame of image and a deep frame interpolation feature of the t th frame of image, respectively; and
performing a frame interpolation operation layer by layer based on the deep frame interpolation features and the motion level information of the (t−1) th frame of image and the t th frame of image to generate an intermediate frame between the (t−1) th frame of image and the t th frame of image, and interpolating the intermediate frame between the (t−1) th frame of image and the t th frame of image,
wherein the acquiring motion level information of pixel points of the (t−1) th frame of image and the t th frame of image comprises:
acquiring a motion level deep feature of the (t−1) th frame of image and a motion level deep feature of the t th frame of image, respectively; and
obtaining the motion level information of the pixel points of the (t−1) th frame of image and the t th frame of image based on the motion level deep feature of the (t−1) th frame of image and the motion level deep feature of the t th frame of image.
7. The electronic device according to claim 6 , wherein the obtaining the motion level information of the pixel points of the (t−1) th frame of image and the t th frame of image based on the motion level deep feature of the (t−1) th frame of image and the motion level deep feature of the t th frame of image comprises:
marking pixels having identical motion attributes, objects having identical motion attributes or areas having identical motion attributes as a given level based on the motion level deep feature of the (t−1) th frame of image and the motion level deep feature of the t th frame of image, to obtain the motion level information of the pixel points of the (t−1) th frame of image and the t th frame of image.
8. The electronic device according to claim 7 , wherein motion levels comprise a motion layer and a stationary layer.
9. The electronic device according to claim 6 , wherein the performing a frame interpolation operation layer by layer based on the deep frame interpolation features and the motion level information of the (t−1) th frame of image and the t th frame of image to generate an intermediate frame between the (t−1) th frame of image and the t th frame of image, and interpolating the intermediate frame between the (t−1) th frame of image and the t th frame of image comprises:
inputting the deep frame interpolation features and the motion level information of the (t−1) th frame of image and the t th frame of image into a pre-trained frame interpolation model, the frame interpolation model performing the frame interpolation operation layer by layer based on the motion level information, to obtain intermediate frames of layers; and
combining the intermediate frames of the layers to obtain the intermediate frame between the (t−1) th frame of image and the t th frame of image, and interpolating the intermediate frame between the (t−1) th frame of image and the t th frame of image.
10. The electronic device according to claim 9 , wherein frame interpolation operations for layers are identical.
11. A non-transitory computer readable storage medium, storing computer instructions, wherein the computer instructions, when executed by a computer, cause the computer to perform operations comprising:
acquiring a target video, and acquiring a (t−1) th frame of image and a t th frame of image in the target video, t being a positive integer;
acquiring motion level information of pixel points of the (t−1) th frame of image and the t th frame of image;
acquiring a deep frame interpolation feature of the (t−1) th frame of image and a deep frame interpolation feature of the t th frame of image, respectively; and
performing a frame interpolation operation layer by layer based on the deep frame interpolation features and the motion level information of the (t−1) th frame of image and the t th frame of image to generate an intermediate frame between the (t−1) th frame of image and the t th frame of image, and interpolating the intermediate frame between the (t−1) th frame of image and the t th frame of image,
wherein the acquiring motion level information of pixel points of the (t−1) th frame of image and the t th frame of image comprises:
acquiring a motion level deep feature of the (t−1) th frame of image and a motion level deep feature of the t th frame of image, respectively; and
obtaining the motion level information of the pixel points of the (t−1) th frame of image and the t th frame of image based on the motion level deep feature of the (t−1) th frame of image and the motion level deep feature of the t th frame of image.
12. The storage medium according to claim 11 , wherein the obtaining the motion level information of the pixel points of the (t−1) th frame of image and the t th frame of image based on the motion level deep feature of the (t−1) th frame of image and the motion level deep feature of the t th frame of image comprises:
marking pixels having identical motion attributes, objects having identical motion attributes or areas having identical motion attributes as a given level based on the motion level deep feature of the (t−1) th frame of image and the motion level deep feature of the t th frame of image, to obtain the motion level information of the pixel points of the (t−1) th frame of image and the t th frame of image.
13. The storage medium according to claim 12 , wherein motion levels comprise a motion layer and a stationary layer.
14. The storage medium according to claim 11 , wherein the performing a frame interpolation operation layer by layer based on the deep frame interpolation features and the motion level information of the (t−1) th frame of image and the t th frame of image to generate an intermediate frame between the (t−1) th frame of image and the t th frame of image, and interpolating the intermediate frame between the (t−1) th frame of image and the t th frame of image comprises:
inputting the deep frame interpolation features and the motion level information of the (t−1) th frame of image and the t th frame of image into a pre-trained frame interpolation model, the frame interpolation model performing the frame interpolation operation layer by layer based on the motion level information, to obtain intermediate frames of layers; and
combining the intermediate frames of the layers to obtain the intermediate frame between the (t−1) th frame of image and the t th frame of image, and interpolating the intermediate frame between the (t−1) th frame of image and the t th frame of image.
15. The storage medium according to claim 14 , wherein frame interpolation operations for layers are identical.