IP Library › Granted Patent US 12,229,972
Granted Patent B2
US 12,229,972 · App. 17/721,288 · Granted Feb 18, 2025

Unsupervised training of optical flow estimation neural networks

Inventors: Daniel Rudolf Maurer (Mountain View, CA); Austin Charles Stone (San Francisco, CA); Alper Ayvaci (Santa Clara, CA); Anelia Angelova (Sunnyvale, CA); Rico Jonschkowski (Berlin, DE)
Assignee: Waymo LLC
G06T7/248G06N3/08G06T3/18G06T5/77G06T7/174G06T2207/20081G06T2207/20084G06T2207/20132
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,229,972
App. No.
17/721,288
Granted
Feb 18, 2025
Kind
B2
Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a neural network to predict optical flow. One of the methods includes obtaining a batch of one or more training image pairs; for each of the pairs: processing the first training image and the second training image using the neural network to generate a final optical flow estimate; generating a cropped final optical flow estimate from the final optical flow estimate; and training the neural network using the cropped optical flow estimate.

Claims (69)

1. A method performed by one or more computers and for training a neural network that has a plurality of network parameters and that is configured to receive as input a first image and a second image and to generate as output an optical flow estimate of optical flow between the first image and the second image, the method comprising:

obtaining a batch of one or more training image pairs, each training image pair comprising a respective first training image and a respective second training image;

for each of the one or more training image pairs:

processing the first training image and the second training image using the neural network to generate a final optical flow estimate from the first training image to the second training image;

generating a cropped final optical flow estimate from the first training image to the second training image, comprising cropping the final optical flow estimate from the first training image to the second training image; and

training the neural network on the one or more training image pairs, the training comprising, for each training image pair, using the cropped final optical flow estimate for the training image pair as a target output for the neural network.

2. The method of claim 1 , further comprising, for each of the one or more training image pairs:

generating a modified first training image, comprising cropping the first training image in the training image pair;

generating a modified second training image, comprising cropping the second training image in the training image pair; and

processing the modified first training image and the modified second training image using the neural network to generate one or more modified optical flow estimates, wherein training the neural network on the one or more training image pairs comprises:

computing a gradient with respect to the network parameters of a loss function that comprises a first term that measures, for each training image pair, an error between (i) the one or more modified optical flow estimates for the training image pair and (ii) the cropped final optical flow estimate for the training image pair; and

updating the network parameters using the gradient.

3. The method of claim 2 , wherein the neural network is configured to generate the optical flow estimate of optical flow between the first image and the second image by:

initializing the optical flow estimate, and

at each of a plurality of update iterations, updating the optical flow estimate using features of the first image and the second image, and

wherein the one or more modified optical flow estimates include a respective modified optical flow estimate for each of the update iterations.

4. The method of claim 3 , wherein the first term measures, for each training image pair and for each of the plurality of update iterations, an error between (i) the respective modified optical flow estimate for the update iteration for the training image pair and (ii) the cropped final optical flow estimate for the training image pair.

5. The method of claim 3 , further comprising, for each training image pair and for each of the plurality of update iterations:

generating a warped second training image by warping the second training image in the training image pair using the respective modified optical flow estimate for the update iteration, wherein:

the loss function further comprises a second term that measures, for each training image pair and for each of the plurality of update iterations, a photometric difference between the warped second training image for the training image pair and for the update iteration and the first training image in the training image pair.

6. The method of claim 5 , wherein the photometric difference is measured using an occlusion mask that masks out occluded pixels from contributing to the photometric difference.

7. The method of claim 3 , wherein the loss function comprises a third term that measures, for each training image pair and for each of the plurality of update iterations, an edge-aware smoothness of the respective modified optical flow estimate for the training image pair and for the update iteration.

8. The method of claim 1 , further comprising:

obtaining one or more sequences of training images, each sequence comprising (i) a current training image, (ii) a preceding training image that precedes the current training image in the sequence, and (iii) a following training image that follows the current training image in the sequence;

for each sequence:

processing the current training image and the preceding training image in the sequence using the neural network to generate a backward optical flow estimate from the current training image to the preceding training image;

processing the current training image and the following training image in the sequence using the neural network to generate a forward optical flow estimate from the current training image to the forward training image;

generating, from the backward optical flow estimate, a prediction of the forward optical flow estimate; and

generating an in-painted forward flow estimate from the prediction and from the forward optical flow estimate;

computing a gradient with respect to the network parameters of a second loss function that includes a third term that measures, for each sequence, an error between the in-painted forward flow estimate and the forward optical flow estimate; and

updating the network parameters using the gradient.

9. The method of claim 8 , wherein generating, from the backward optical flow estimate, a prediction of the forward optical flow estimate comprises:

processing an input comprising at least the backward optical flow estimate using a learned inversion machine learning model that has been trained to generate the prediction.

10. The method of claim 9 , wherein the input comprises normalized image coordinates of the pixels in the current image.

11. The method of claim 9 , wherein the learned inversion machine learning model is trained specifically for the preceding image—following image pair.

12. The method of claim 8 , wherein generating an in-painted forward flow estimate from the prediction and from the forward optical flow estimate comprises:

in-painting one or more occluded regions of the forward optical flow estimate using the prediction.

13. The method of claim 1 , wherein the optical flow estimate of optical flow between the first image and the second image includes, for each pixel in the first image, a respective offset of a corresponding pixel in the second image.

14. The method of claim 1 , wherein:

generating a modified first training image further comprises applying one or more data augmentations to the first training image in the pair; and

generating a modified second training image further comprises applying one or more data augmentations to the second training image in the pair.

15. One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training a neural network that has a plurality of network parameters and that is configured to receive as input a first image and a second image and to generate as output an optical flow estimate of optical flow between the first image and the second image, the operations comprising:

obtaining a batch of one or more training image pairs, each training image pair comprising a respective first training image and a respective second training image;

for each of the one or more training image pairs:

processing the first training image and the second training image using the neural network to generate a final optical flow estimate from the first training image to the second training image;

generating a cropped final optical flow estimate from the first training image to the second training image, comprising cropping the final optical flow estimate from the first training image to the second training image; and

training the neural network on the one or more training image pairs, the training comprising, for each training image pair, using the cropped final optical flow estimate for the training image pair as a target output for the neural network.

16. A system comprising:

one or more computers; and

one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations training a neural network that has a plurality of network parameters and that is configured to receive as input a first image and a second image and to generate as output an optical flow estimate of optical flow between the first image and the second image, the operations comprising:

obtaining a batch of one or more training image pairs, each training image pair comprising a respective first training image and a respective second training image;

for each of the one or more training image pairs:

processing the first training image and the second training image using the neural network to generate a final optical flow estimate from the first training image to the second training image;

generating a cropped final optical flow estimate from the first training image to the second training image, comprising cropping the final optical flow estimate from the first training image to the second training image; and

training the neural network on the one or more training image pairs, the training comprising, for each training image pair, using the cropped final optical flow estimate for the training image pair as a target output for the neural network.

17. The system of claim 16 , the operations further comprising, for each of the one or more training image pairs:

generating a modified first training image, comprising cropping the first training image in the training image pair;

generating a modified second training image, comprising cropping the second training image in the training image pair; and

processing the modified first training image and the modified second training image using the neural network to generate one or more modified optical flow estimates, wherein training the neural network on the one or more training image pairs comprises:

computing a gradient with respect to the network parameters of a loss function that comprises a first term that measures, for each training image pair, an error between (i) the one or more modified optical flow estimates for the training image pair and (ii) the cropped final optical flow estimate for the training image pair; and

updating the network parameters using the gradient.

18. The system of claim 17 , wherein the neural network is configured to generate the optical flow estimate of optical flow between the first image and the second image by:

initializing the optical flow estimate, and

at each of a plurality of update iterations, updating the optical flow estimate using features of the first image and the second image, and

wherein the one or more modified optical flow estimates include a respective modified optical flow estimate for each of the update iterations.

19. The system of claim 18 , wherein the first term measures, for each training image pair and for each of the plurality of update iterations, an error between (i) the respective modified optical flow estimate for the update iteration for the training image pair and (ii) the cropped final optical flow estimate for the training image pair.

20. The system of claim 18 , further comprising, for each training image pair and for each of the plurality of update iterations:

generating a warped second training image by warping the second training image in the training image pair using the respective modified optical flow estimate for the update iteration, wherein:

the loss function further comprises a second term that measures, for each training image pair and for each of the plurality of update iterations, a photometric difference between the warped second training image for the training image pair and for the update iteration and the first training image in the training image pair.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 30, 2022
From: MAURER, DANIEL RUDOLF; STONE, AUSTIN CHARLES; AYVACI, ALPER; ANGELOVA, ANELIA; JONSCHKOWSKI, RICO
To: WAYMO LLC
Reel/Frame 060942/0560 →
Continuity (2)
Provisional Application 63175498 · Apr 15, 2021
Related Publication 20220335624A1 · Oct 20, 2022
References Cited (52)
US 10289912B1 · Vijayanarasimhan · 2019 [cited by examiner]
US 20110286635A1 · Nishigaki · 2011 [cited by examiner]
US 20170186176A1 · Paluri · 2017 [cited by examiner]
US 20190392591A1 · Lee · 2019 [cited by examiner]
US 20200160065A1 · Weinzaepfel · 2020 [cited by examiner]
US 20200210708A1 · Jia · 2020 [cited by examiner]
US 20200357099A1 · Long · 2020 [cited by examiner]
US 20210035268A1 · Xu · 2021 [cited by examiner]
US 20210097715A1 · Li · 2021 [cited by examiner]
US 20210183079A1 · Cook · 2021 [cited by examiner]
US 20210232933A1 · Gu · 2021 [cited by examiner]
US 20220129667A1 · Afrasiabi · 2022 [cited by examiner]
US 20220262012A1 · Li · 2022 [cited by examiner]
US 20220319054A1 · Abdo · 2022 [cited by examiner]
Brox et al., “High accuracy optical flow estimation based on a theory for warping,” European Conference on Computer Vision, 2004, pp. 25-36. [cited by applicant]
Butler et al., “A naturalistic open source movie for optical flow evaluation,” European Conference on Computer Vision, 2012, pp. 611-625. [cited by applicant]
Chen et al., “Full flow: Optical flow estimation by global optimization over regular grids,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 4706-4714. [cited by applicant]
Dosovitskiy et al., “Flownet: Learning optical flow with convolutional networks,” Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 2758-2766. [cited by applicant]
Extended European Search Report in European Appln. No. 22168578.7, dated Sep. 22, 2022, 10 pages. [cited by applicant]
Horn et al., “Determining optical flow,” Artificial Intelligence, Aug. 1981, 17(1-3):185-203. [cited by applicant]
Ilg et al., “Flownet 2.0: Evolution of optical flow estimation with deep networks,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 2462-2470. [cited by applicant]
Janai et al., “Unsupervised learning of multi-frame optical flow with occlusions,” Proceedings of the European Conference on Computer Vision, 2018, pp. 690-706. [cited by applicant]
Jonschkowski et al., “What matters in unsupervised optical flow,” European Conference on Computer Vision, Aug. 2020, pp. 557-572. [cited by applicant]
Kingma et al., “Adam: A method for stochastic optimization,” CoRR, Dec. 2014, https://arxiv.org/abs/1412.6980, 15 pages. [cited by applicant]
Liu et al., “A survey on deep learning methods for scene flow estimation,” Pattern Recognition, May 2020, 106, 106:107378. [cited by applicant]
Liu et al., “DDFlow: Learning optical flow with unlabeled data distillation,” AAAI Techincal Track: Vision, Jul. 17, 2019, 33(1):IAAI-19-EAAI-20. [cited by applicant]
Liu et al., “Selflow:Self-supervised learning of optical flow,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 4571-4580. [cited by applicant]
Lucas et al., “An iterative image registration technique with an application to stereo vision,” Proceedings DARPA Image Understanding Workshop, Apr. 1981, pp. 121-130. [cited by applicant]
Maurer et al., “Proflow: Learning to predict optical flow,” BMVC, 2018, 16 pages. [cited by applicant]
Mayer, et al., “A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 40… [cited by applicant]
Meister et al., “Unflow: Unsupervised learning of optical flow with a bidirectional census loss,” Proceedings of the AAAI Conference on Artificial Intelligence, Apr. 27, 2018, 32(1):7251-7259. [cited by applicant]
Menze et al., “Joint 3D estimation of vehicles and scene flow,” ISPRS Workshop on Image Sequence Analysis, Sep. 28, 2015, II-3/W5:427-434. [cited by applicant]
Ranjan et al., “Competitive collaboration: Joint unsupervised learning of depth, camera motion, optical flow and motion segmentation,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (C… [cited by applicant]
Ranjan et al., “Optical flow estimation using a spatial pyramid network,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 4161-4170. [cited by applicant]
Ren et al., “Unsupervised deep learning for optical flow estimation,” Proceedings of the AAAI Conference on Artificial Intelligence, Feb. 12, 2017, pp. 1495-1501. [cited by applicant]
Steinbrucker et al., “Large displacement optical flow computation without warping,” 2009 IEEE 12th International Conference on Computer Vision, Sep. 29, 2009, 6 pages. [cited by applicant]
Stone et al., “SMURF: Self-Teaching Multi-Frame Unsupervised RAFT with Full-Image Warping,” 2021 IEEE/CVF Conference on Computer Vison and Pattern Recognition (CVPR), Jun. 2021, pp. 3886-3895. [cited by applicant]
Sun et al., “Secrets of optical flow estimation and their principles,” 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Jun. 13-18, 2010, 8 pages. [cited by applicant]
Sun et al., “PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 8934-8943. [cited by applicant]
Teed et al., “Raft: Recurrent all pairs field transforms for optical flow,” ECCV 2020: Computer Vision—ECCV 2020 , Nov. 3, 2020, pp. 402-419. [cited by applicant]
Tomasi et al., “Bilateral filtering for gray and color images,” Sixth International Conference on Computer Vision (IEEE Cat. No. 98CH36271), Aug. 6, 2002, 8 pages. [cited by applicant]
Ulyanov et al., “Instance normalization: The missing ingredient for fast stylization,” CoRR, Jul. 27, 2016, arxiv.org/abs/1607.08022, 6 pages. [cited by applicant]
Vedula et al., “Three-dimensional scene flow,” Proceedings of the Seventh IEEE International Conference on Computer Vision, Sep. 20-27, 1999, 8 pages. [cited by applicant]
Wang et al., “Occlusion aware unsupervised learning of optical flow,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 4884-4893. [cited by applicant]
Wang et al., “Unos: Unified unsupervised optical-flow and stereo-depth estimation by watching videos,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 8071-8081. [cited by applicant]
Xu et al., “Accurate optical flow via direct cost vol. processing,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 1289-1297. [cited by applicant]
Yang et al., “Volumetric correspondence networks for optical flow,” Advances in Neural Information Processing Systems 32 (NeurIPS 2019), 2019, 12 pages. [cited by applicant]
Yin et al., “Geonet: Unsupervised learning of dense depth, optical flow and camera pose,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 1983-1992. [cited by applicant]
Yu et al., “Back to basics: Unsupervised learning of optical flow via brightness constancy and motion smoothness,” European Conference on Computer Vision, Nov. 24, 2016, 4 pages. [cited by applicant]
Zabih et al., “Non-parametric local transforms for computing visual correspondence,” European Conference on Computer Vision, May 1994, pp. 151-158. [cited by applicant]
Zhong et al., “Unsupervised deep epipolar flow for stationary or dynamic scenes,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 12095-12104. [cited by applicant]
Zou et al., “DF-Net: Unsupervised joint learning of depth and flow using cross-task consistency,” Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 36-53. [cited by applicant]