Live style transfer on a mobile device
Various embodiments of the present invention relate generally to systems and processes for transforming a style of video data. In one embodiment, a neural network is used to interpolate native video data received from a camera system on a mobile device in real-time. The interpolation converts the live native video data into a particular style. For example, the style can be associated with a particular artist or a particular theme. The stylized video data can viewed on a display of the mobile device in a manner similar to which native live video data is output to the display. Thus, the stylized video data, which is viewed on the display, is consistent with a current position and orientation of the camera system on the display.
1 . A method comprising:
receiving, on a mobile device, a plurality of sets of neural network weighting factors from a remote computing device via a network interface, the neural network weighting factors implemented into a first neural network located on the mobile device, wherein the neural network weighting factors are pre-trained on the remote computing device;
receiving a temporal sequence of video data on the mobile device from a camera associated with the mobile device;
determining camera poses for frames in the temporal sequence of video data;
estimating a camera path based on the determined camera poses, wherein the camera path extends outside of a physical trajectory defined by the camera poses associated with two frames in the temporal sequence of video data;
extrapolating a viewpoint on the camera path outside of the physical trajectory of the two frames to synthesize an extrapolated frame for a multi-view interactive digital media representation;
applying the first neural network, including the neural network weighting factors, to frames in the video data and to the extrapolated frame to apply a style transformation to the video data and the extrapolated frame;
outputting the transformed video data and the extrapolated frame to a display of the mobile device; and
recording the transformed video data and the extrapolated frame to a memory module.
2 . The method of claim 1 , wherein the neural network weighting factors are pre-trained to convert video images to a first style.
3 . The method of claim 1 , wherein the video data is a live feed.
4 . The method of claim 1 , wherein outputting the transformed video data occurs in real time.
5 . The method of claim 1 , receiving a second plurality of sets of neural network weighting factors.
6 . The method of claim 1 , further comprising:
generating, based upon a second style, a second neural network including second weighting factors;
applying the second neural network to second frames in the video data to transform the second frames to the second style associated with the second neural network; and
outputting the second frames to the display in real-time.
7 . The method of claim 1 , further comprising, after the transformed video data is output to the display, receiving a request to record the video data from the live feed.
8 . A mobile device comprising:
a processor; and
memory, the memory storing instructions for the processor to execute a method, the method comprising:
receiving, on a mobile device, a plurality of sets of neural network weighting factors from a remote computing device via a network interface, the neural network weighting factors implemented into a first neural network located on the mobile device, wherein the neural network weighting factors are pre-trained on the remote computing device;
receiving a temporal sequence of video data on the mobile device from a camera associated with the mobile device;
determining camera poses for frames in the temporal sequence of video data;
estimating a camera path based on the determined camera poses, wherein the camera path extends outside of a physical trajectory defined by the camera poses associated with two frames in the temporal sequence of video data, wherein the trajectory is defined by camera poses associated with the two frames;
extrapolating a viewpoint on the camera path outside of the physical trajectory of the two frames in the video data to synthesize an extrapolated frame for a multi-view interactive digital media representation;
applying the first neural network, including the neural network weighting factors, to frames in the video data and to transform the extrapolated frame to apply a style transformation to the video data and the extrapolated frame;
outputting the transformed video data and the extrapolated frame to a display of the mobile device; and
recording the transformed video data and the extrapolated frame to a memory module.
9 . The mobile device of claim 8 , wherein the neural network weighting factors are pre-trained to convert video images to a first style.
10 . The mobile device of claim 8 , wherein the video data is a live feed.
11 . The mobile device of claim 8 , wherein outputting the transformed video data occurs in real time.
12 . The mobile device of claim 8 , receiving a second plurality of sets of neural network weighting factors.
13 . The mobile device of claim 8 , wherein the method further comprises:
generating, based upon a second style, a second neural network including second weighting factors;
applying the second neural network to second frames in the video data to transform the second frames to the second style associated with the second neural network; and
outputting the second frames to the display in real-time.
14 . The mobile device of claim 8 , wherein the method further comprises, after the transformed video data is output to the display, receiving a request to record the video data from the live feed.
15 . A non-transitory computer readable medium storing instructions to cause a processor to execute a method, the method comprising:
receiving, on a mobile device, a plurality of sets of neural network weighting factors from a remote computing device via a network interface, the neural network weighting factors implemented into a first neural network located on the mobile device, wherein the neural network weighting factors are pre-trained on the remote computing device;
receiving a temporal sequence of video data on the mobile device from a camera associated with the mobile device;
determining camera poses for frames in the temporal sequence of video data;
estimating a camera path based on the determined camera poses, wherein the camera path extends outside of a physical trajectory defined by the camera poses associated with two frames in the temporal sequence of video data, wherein the trajectory is defined by camera poses associated with the two frames;
extrapolating a viewpoint on the camera path outside of the physical trajectory of the two frames in the video data to synthesize an extrapolated frame for a multi-view interactive digital media representation;
applying the first neural network, including the neural network weighting factors, to frames in the video data and to transform the extrapolated frame to apply a style transformation to the video data and the extrapolated frame;
outputting the transformed video data and the extrapolated frame to a display of the mobile device; and
recording the transformed video data and the extrapolated frame to a memory module.
16 . The non-transitory computer readable medium of claim 15 , wherein the neural network weighting factors are pre-trained to convert video images to a first style.
17 . The non-transitory computer readable medium of claim 15 , wherein the video data is a live feed.
18 . The non-transitory computer readable medium of claim 15 , wherein outputting the transformed video data occurs in real time.
19 . The method of claim 1 , wherein:
the physical trajectory is defined by the camera poses associated with the at least two frames; and
estimating the camera path comprises generating a continuous camera path by:
interpolating a smooth curve between the camera poses to model the physical trajectory, and
extrapolating the smooth curve beyond the physical trajectory to define the camera path.
20 . The method of claim 19 , wherein the camera poses are determined based on sensor data from one or more sensors onboard the mobile device, the sensor data comprising inertial measurement unit (IMU) data, GPS data, or a combination thereof.