IP Library › Granted Patent US 12,315,083
Granted Patent B2
US 12,315,083 · App. 18/120,989 · Granted May 27, 2025

Performing point cloud tasks using multi-scale features generated through self-attention

Inventors: Pei Sun (Palo Alto, CA); Mingxing Tan (Newark, CA); Weiyue Wang (Sunnyvale, CA); Fei Xia (Sunnyvale, CA); Zhaoqi Leng (Milpitas, CA); Dragomir Anguelov (San Francisco, CA); Chenxi Liu (Santa Clara, CA)
Assignee: Waymo LLC
G06T17/20G06F30/27G06T5/60G06T15/20G06T17/05G06T2210/56
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,315,083
App. No.
18/120,989
Granted
May 27, 2025
Kind
B2
Abstract

Methods, systems, and apparatus for processing point clouds using neural networks to perform a machine learning task. In one aspect, a system comprises one or more computers configured to obtain a set of point clouds captured by one or more sensors. Each point cloud includes a respective plurality of three-dimensional points. The one or more computers assign the three-dimensional points to respective voxels in a voxel grid, where the grid of voxels includes non-empty voxels to which one or more points are assigned and empty voxels to which no points are assigned. For each non-empty voxel, the one or more computers generate initial features based on the points that are assigned to the non-empty voxel. The one or more computers generate multi-scale features of the voxel grid, and the one or more computers generate an output for a point cloud processing task using the multi-scale features of the voxel grid.

Claims (67)

1. A method performed by one or more computers, the method comprising:

obtaining a set of point clouds captured by one or more sensors, wherein each point cloud comprises a respective plurality of three-dimensional points;

assigning the three-dimensional points to respective voxels in a voxel grid of voxels;

generating multi-scale features of the voxel grid, the multi-scale features comprising, for each of a plurality of scales, respective features for each non-empty voxel in a scaled voxel grid corresponding to the scale, the generating comprising:

processing respective features for each non-empty voxel in the voxel grid through a hierarchical sequence of self-attention neural network blocks, the processing comprising, for each scale:

obtaining initial features for each non-empty voxel in the scaled voxel grid corresponding to the scale; and

processing the initial features for the non-empty voxels in the scaled voxel grid corresponding to the scale using a self-attention neural network to generate the respective features for the non-empty voxels in the scaled voxel grid corresponding to the scale; and

generating an output for a point cloud processing task using the multi-scale features of the voxel grid.

2. The method of claim 1 , wherein the non-empty voxels are respective voxels to which one or more points are assigned and empty voxels are respective voxels to which no points are assigned.

3. The method of claim 2 , further comprising:

for each non-empty voxel, generating initial features for the non-empty voxel based on the points that are assigned to the non-empty voxel.

4. The method of claim 3 , wherein processing the initial features for the non-empty voxels in the scaled voxel grid corresponding to the scale using a self-attention neural network block to generate the respective features for the non-empty voxels in the scaled voxel grid corresponding to the scale comprises:

partitioning the scaled voxel grid into a plurality of windows;

for each window that has non-empty voxels;

processing the initial features of the non-empty voxels in the window using one or more Transformers layers that each apply self-attention only over the non-empty voxels in the window to generate respective updated features for the non-empty voxels in the window;

partitioning the scaled voxel grid into a plurality of shifted windows that are different from the plurality of windows; and

for each shifted window that has non-empty voxels:

processing the updated features of the non-empty voxels in the shifted window using one or more Transformers layers that each apply self-attention only over the non-empty voxels in the shifted window to generate the respective features for the non-empty voxels in the shifted window.

5. The method of claim 3 , wherein, for each scale, each voxel in the scaled voxel grid corresponding to the scale corresponds to a respective region in the voxel grid having a size corresponding to the scale.

6. The method of claim 5 , wherein:

the plurality of scales comprise a first scale that is equal to a scale of the voxel,

the corresponding scaled voxel grid for the first scale is the voxel grid, and

the initial features for the non-empty voxels in the corresponding scaled voxel for the first scale are the initial features for the non-empty voxels in the voxel grid.

7. The method of claim 3 , wherein obtaining initial features for each non-empty voxel in the scaled voxel grid corresponding to the scale comprises:

for each non-empty voxel in the scaled voxel grid corresponding to the scale, identifying a non-empty voxel in the voxel grid that is closest to a center of the respective region in the voxel grid corresponding to the voxel; and

generating the initial features for the voxel based on the identified non-empty voxel.

8. The method of claim 7 , wherein generating the initial features for the voxel based on the identified non-empty voxel comprises:

for each scale other than the first scale,

identifying a preceding non-empty voxel in a scaled voxel grid corresponding to a smaller scale having a corresponding region that includes the identified non-empty voxel; and

setting the initial features for the voxel equal to the features for the identified preceding non-empty voxel.

9. The method of claim 1 , wherein generating an output for a point cloud processing task using the multi-scale features of the voxel grid comprises:

for each scale, generating respective fused features for each non-empty voxel in the scaled voxel grid for the scale from the multi-scale features; and

generating the output for the point cloud processing task from the fused features.

10. The method of claim 9 , wherein the respective fused features for a largest scale are the respective features for the largest scale.

11. The method of claim 10 , wherein for each scale other than the largest scale, generating respective fused features for each non-empty voxel in the scaled voxel grid for the scale comprises:

generating a concatenated feature for each non-empty voxel in the scaled voxel grid for the scale from the features for the non-empty voxel and fused features for a corresponding voxel in an upscaled voxel grid generated from a scaled voxel grid having a higher scale; and

processing the concatenated features for the non-empty voxels in the scaled voxel grid using a self-attention neural network block.

12. The method of claim 9 , wherein:

generating the output for the point cloud processing task comprises:

applying voxel diffusion to at least the fused features for the smallest scale to generate a diffused feature for each voxel in the voxel grid; and

processing the diffused features using a decoder neural network to generate the output for the point cloud processing task.

13. The method of claim 1 , wherein the point cloud processing task is object detection.

14. The method of claim 1 , wherein the point cloud processing task is semantic segmentation.

15. A system comprising:

one or more computers; and

one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:

obtaining a set of point clouds captured by one or more sensors, wherein each point cloud comprises a respective plurality of three-dimensional points;

assigning the three-dimensional points to respective voxels in a voxel grid of voxels;

generating multi-scale features of the voxel grid, the multi-scale features comprising, for each of a plurality of scales, respective features for each non-empty voxel in a scaled voxel grid corresponding to the scale, the generating comprising:

processing respective features for each non-empty voxel in the voxel grid through a hierarchical sequence of self-attention neural network blocks, the processing comprising, for each scale:

obtaining initial features for each non-empty voxel in the scaled voxel grid corresponding to the scale; and

processing the initial features for the non-empty voxels in the scaled voxel grid corresponding to the scale using a self-attention neural network to generate the respective features for the non-empty voxels in the scaled voxel grid corresponding to the scale; and

generating an output for a point cloud processing task using the multi-scale features of the voxel grid.

16. The system of claim 15 , wherein the non-empty voxels are respective voxels to which one or more points are assigned and empty voxels are respective voxels to which no points are assigned.

17. The system of claim 16 , further comprising:

for each non-empty voxel, generating initial features for the non-empty voxel based on the points that are assigned to the non-empty voxel.

18. One or more non-transitory computer-readable storage media storing encoded with computer program instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

obtaining a set of point clouds captured by one or more sensors, wherein each point cloud comprises a respective plurality of three-dimensional points;

assigning the three-dimensional points to respective voxels in a voxel grid of voxels;

generating multi-scale features of the voxel grid, the multi-scale features comprising, for each of a plurality of scales, respective features for each non-empty voxel in a scaled voxel grid corresponding to the scale, the generating comprising:

processing respective features for each non-empty voxel in the voxel grid through a hierarchical sequence of self-attention neural network blocks, the processing comprising, for each scale:

obtaining initial features for each non-empty voxel in the scaled voxel grid corresponding to the scale; and

processing the initial features for the non-empty voxels in the scaled voxel grid corresponding to the scale using a self-attention neural network to generate the respective features for the non-empty voxels in the scaled voxel grid corresponding to the scale; and

generating an output for a point cloud processing task using the multi-scale features of the voxel grid.

19. The non-transitory computer-readable storage media of claim 18 , wherein the non-empty voxels are respective voxels to which one or more points are assigned and empty voxels are respective voxels to which no points are assigned.

20. The non-transitory computer-readable storage media of claim 18 , the operations further comprising:

for each non-empty voxel, generating initial features for the non-empty voxel based on the points that are assigned to the non-empty voxel.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 19, 2023
From: SUN, PEI; TAN, MINGXING; WANG, WEIYUE; XIA, FEI; LENG, ZHAOQI; ANGUELOV, DRAGOMIR; LIU, CHENXI
To: WAYMO LLC
Reel/Frame 064309/0065 →
Continuity (3)
Provisional Application 63323914 · Mar 25, 2022
Provisional Application 63319228 · Mar 11, 2022
Related Publication 20230351691A1 · Nov 2, 2023
References Cited (53)
US 10970518B1 · Zhou · 2021 [cited by examiner]
US 11100669B1 · Zhou · 2021 [cited by examiner]
US 20210335031A1 · Hamilton · 2021 [cited by examiner]
US 20210350620A1 · Bronstein · 2021 [cited by examiner]
US 20230316642A1 · Mory · 2023 [cited by examiner]
US 20240282051A1 · Roimela · 2024 [cited by examiner]
Bello et al., “Attention augmented convolutional networks,” Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 2019, pp. 3286-3295. [cited by applicant]
Bewley et al., “Range conditioned dilated convolutions for scale invariant 3D object detection,” CoRR, May 20, 2020, arxiv.org/pdf/2005.09927, 16 pages. [cited by applicant]
Carion et al., “End-to-end object detection with transformers,” European conference on computer vision, Nov. 3, 2020, pp. 213-229. [cited by applicant]
Chai et al., “To the point: Efficient 3Dobject detection in the range image with graph convolution kernels,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 16000-16009. [cited by applicant]
Cheng et al., “Improving 3D object detection through progressive population based augmentation,” European Conference on Computer Vision, Nov. 12, 2020, pp. 279-294. [cited by applicant]
Dai et al., “CoAtNnet: Marrying convolution and attention for all data sizes,” Advances in Neural Information Processing Systems 34, 2021, 13 pages. [cited by applicant]
Devlin et al., “BERT: Pre-training of deep bidirectional transformers for language understanding,” CoRR, Oct. 11, 2018, arXiv:1810.04805, 16 pages. [cited by applicant]
Dosovitskiy et al., “An image is worth 16×16 words: Transformers for image recognition at scale,” CoRR, Oct. 22, 2020, arXiv:2010.11929, 22 pages. [cited by applicant]
Engel et al., “Point transformer,” IEEE Access, Sep. 29, 2021, 9:134826-134840. [cited by applicant]
Fan et al., “Embracing single stride 3D object detector with sparse transformer,” CoRR, Dec. 13, 2021, arXiv:2112.06375, 14 pages. [cited by applicant]
Fan et al., “Rangedet: In defense of range view for lidar-based 3D object detection,” Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 2918-2927. [cited by applicant]
Ge et al., “AFDet: Anchor free one stage 3d object detection,” CoRR, Jun. 23, 2020, arXiv:2006.12671, 10 pages. [cited by applicant]
Graham et al., “Submanifold sparse convolutional networks,” CoRR, Jun. 5, 2017, arXiv:1706.01307, 10 pages. [cited by applicant]
Guizilini et al., “3D packing for self supervised monocular depth estimation,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 2485-2494. [cited by applicant]
Huang et al., “Deep networks with stochastic depth,” European Conference on Computer Vision, Sep. 17, 2016, pp. 646-661. [cited by applicant]
Kingma et al., “Adam: A method for stochastic optimization,” CoRR, Dec. 22, 2014, arXiv:1412.6980, 15 pages. [cited by applicant]
Lang et al., “Pointpillars: Fast encoders for object detection from point clouds,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 12697-12705. [cited by applicant]
Law et al., “Cornernet: Detecting objects as paired keypoints,” Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 734-750. [cited by applicant]
Lin et al., “Feature pyramid networks for object detection,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 2117-2125. [cited by applicant]
Lin et al., “Focal loss for dense object detection,” Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017, pp. 2980-2988. [cited by applicant]
Liu et al., “Swin transformer: Hierarchical vision transformer using shifted windows,” Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 10012-10022. [cited by applicant]
Mao et al., “Voxel transformer for 3D object detection,” Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 3164-3173. [cited by applicant]
Meyer et al., “Lasernet: An efficient probabilistic 3D object detector for autonomous driving,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 12677-12686. [cited by applicant]
Misra et al., “An end-to-end transformer model for 3D object detection,” Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 2906-2917. [cited by applicant]
Ngiam et al., “Starnet: Targeted computation for object detection in point clouds,” CoRR, Aug. 29, 2019, arXiv:1908.11069, 12 pages. [cited by applicant]
Pan et al., “3D object detection with pointformer,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 7463-7472. [cited by applicant]
Qi et al., “PointNet: Deep learning on point sets for 3D classification and segmentation,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 652-660. [cited by applicant]
Qi et al., “Onboard 3D object detection from point cloud sequences,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 6134-6144. [cited by applicant]
Qi et al., “Pointnet++: Deep hierarchical feature learning on point sets in a metric space,” Advances in Neural Information Processing Systems 30, 2017, 10 pages. [cited by applicant]
Ramachandran et al., “Stand-alone self-attention in vision models,” 33rd Conference on Neural Information Processing Systems, 2019, 13 pages. [cited by applicant]
Ren et al., “Faster R-CNN: Towards real-time object detection with region proposal networks,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Jun. 2017, 39(6):1137-1149. [cited by applicant]
Sabne, “XLA: Compiling machine learning for peak performance,” Google Research, 2020, 1 pages (abstract only). [cited by applicant]
Shi et al., “PV-RCNN: Point-voxel feature set abstraction for 3D object detection,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 10529-10538. [cited by applicant]
Shi et al., “PointRCNN: 3D object proposal generation and detection from point cloud,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 770-779. [cited by applicant]
Sun et al., “Scalability in perception for autonomous driving: Waymo open dataset,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 2446-2454. [cited by applicant]
Sun et al., “RSN: Range sparse net for efficient, accurate LiDAR 3D object detection,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 5725-5734. [cited by applicant]
Vaswani et al., “Attention is all you need,” IAdvances in Neural Information Processing Systems 30, 2017, 11 pages. [cited by applicant]
Wang et al., “Non-local neural networks,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 7794-7803. [cited by applicant]
Wang et al., “Pillar-based object detection for autonomous driving,” European Conference on Computer Vision, Nov. 17, 2020, 18 pages. [cited by applicant]
Waymo.com [online], “Introducing the 5th-generation Waymo Driver: Informed by experience, designed for scale, engineered to tackle more environments,” Mar. 4, 2020, retrieved on Dec. 5, 2023, retrieved from URL<https://… [cited by applicant]
Yan et al., “Second: Sparsely embedded convolutional detection,” Sensors, Oct. 6, 2018, 18(10):3337. [cited by applicant]
Yin et al., “Center-based 3D object detection and tracking,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 11784-11793. [cited by applicant]
Zhao et al., “Point transformer,” Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 16259-16268. [cited by applicant]
Zhou et al., “IoU loss for 2D/3D object detection,” 2019 International Conference on 3D Vision, Sep. 16-19, 2019, 10 pages. [cited by applicant]
Zhou et al., “Objects as points,” CoRR, Apr. 16, 2019, arXiv:1904.07850, 12 pages. [cited by applicant]
Zhou et al., “End-to-end multi-view fusion for 3D object detection in LiDAR point clouds,” Proceedings of the Conference on Robot Learning, 2020, 100:923-932. [cited by applicant]
Zhou et al., “Voxelnet: End-to-end learning for point cloud based 3D object detection,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 4490-4499. [cited by applicant]