Film-making using style transfer

1. A method of generating a final video comprising a sequence of final video frames, the method comprising:

receiving an initial video comprising a sequence of initial video frames;

receiving a selection of style images from a library of style images;

for each initial video frame in the sequence of initial video frames, processing the initial video frame to generate a final video frame, the processing comprising:

segmenting the initial video frame to generate a segmented video frame, wherein the segmenting comprises characterizing the initial video frame into multiple segments where each segment characterizes a different object depicted in the initial video frame;

generating a plurality of stylized video frames each according to a respective one of the style images, wherein generating a stylized video frame comprises processing the segmented video frame and the respective style image using a style transfer neural network; and

generating a final video frame comprising, for each segment of the segmented video frame:

selecting one of the stylized video frames,

extracting a respective segment from the selected stylized video frame, and

inserting the extracted segment into the final video frame; and

combining each generated final video frame in sequence to generate the final video.

2. The method of claim 1 , further comprising, for each initial video frame in the sequence of initial video frames, preprocessing the initial video frame.

3. The method of claim 2 , wherein preprocessing an initial video frame comprises one or more of:

applying a filter to the initial video frame,

digitally removing puppet strings from the initial video frame,

inserting one or more CGI objects into the initial video frame, or

dropping the initial frame rate.

4. The method of claim 3 , wherein segmenting the initial video frame comprises segmenting the initial frame using one or more of:

a motion-based segmentation algorithm,

a clustering algorithm,

an edge detection algorithm,

a compression-based algorithm, or

a region-growing algorithm.

5. The method of claim 4 , wherein processing the segmented video frame and the respective style image using a style transfer neural network comprises processing the entire segmented video frame using the style transfer neural network to generate a stylized video frame depicting the entire segmented video frame in a style of the respective style image.

6. The method of claim 4 , wherein processing the segmented video frame and the respective style image using a style transfer neural network comprises processing a particular segment of the segmented video frame using the style transfer neural network to generate a stylized video frame depicting the particular segment in a style of the respective style image.

7. The method of claim 1 , wherein processing the segmented video frame and the respective style image using a style transfer neural network comprises processing the entire segmented video frame using the style transfer neural network to generate a stylized video frame depicting the entire segmented video frame in a style of the respective style image.

8. The method of claim 1 , wherein processing the segmented video frame and the respective style image using a style transfer neural network comprises processing a particular segment of the segmented video frame using the style transfer neural network to generate a stylized video frame depicting the particular segment in a style of the respective style image.

9. The method of claim 1 , wherein segmenting the initial video frame comprises segmenting the initial frame using one or more of:

a motion-based segmentation algorithm,

a clustering algorithm,

an edge detection algorithm,

a compression-based algorithm, or

a region-growing algorithm.

10. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one more computers to perform the operations of the respective method of claim 1 .

11. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one more computers to perform the operations of the respective method of claim 1 .