IP Library › Granted Patent US 12,412,082
Granted Patent B2
US 12,412,082 · App. 17/086,114 · Granted Sep 9, 2025

Fine-grained per-vector scaling for neural network quantization

Inventors: Brucek Kurdo Khailany (Austin, TX); Steve Haihang Dai (Union City, CA); Rangharajan Venkatesan (San Jose, CA); Haoxing Ren (Austin, TX)
Assignee: NVIDIA Corporation
G06N3/08G06F17/16G06N3/04
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,412,082
App. No.
17/086,114
Filed
Oct 30, 2020
Granted
Sep 9, 2025
Kind
B2
Art Unit
2128
USPC
706/25
Abstract

Today neural networks are used to enable autonomous vehicles and improve the quality of speech recognition, real-time language translation, and online search optimizations. However, operation of the neural networks for these applications consumes energy. Quantization of parameters used by the neural networks reduces the amount of memory needed to store the parameters while also reducing the power consumed during operation of the neural network. Matrix operations performed by the neural networks require many multiplication calculations, so reducing the number of bits that are multiplied reduces the energy that is consumed. Quantizing smaller sets of the parameters using a shared scale factor improves accuracy compared with quantizing larger sets of the parameters. Accuracy of the calculations may be maintained by quantizing and scaling the parameters using fine-grained per-vector scale factors. A vector includes one or more elements within a single dimension of a multi-dimensional matrix.

Claims (46)

1. A computer-implemented method for quantizing parameters of a neural network model, the method comprising:

computing, for each vector of parameters within a single dimension of a multi-dimensional parameter tensor, a per-vector scale factor to produce a plurality of per-vector scale factors for the multi-dimensional parameter tensor;

computing a coarse scale factor for the multi-dimensional parameter tensor that is shared by at least two of the vectors of parameters;

reducing, for each vector of parameters, a bitwidth of each parameter in the vector of parameters according to a corresponding per-vector scale factor in the plurality of per-vector scale factors to generate a reduced bitwidth vector of parameters, wherein the reduced bitwidth of each of the at least two of the vectors of parameters is also reduced according to the coarse scale factor; and

processing the reduced bitwidth vectors of parameters using a set of multiplier circuits that operate at the reduced bitwidth in a layer of the neural network model to produce each element of a multi-dimensional output tensor.

2. The computer-implemented method of claim 1 , wherein the parameters are at least one of weights or activations.

3. The computer-implemented method of claim 1 , wherein the parameters are weights and the scale factors are precalculated based on the weights.

4. The computer-implemented method of claim 1 , wherein the parameters are calculated by one of post-training calibration or quantization-aware retraining.

5. The computer-implemented method of claim 1 , wherein the parameters are activations and the per-vector scale factors are dynamically calculated at each layer of the neural network model during inferencing.

6. The computer-implemented method of claim 1 , wherein the processing comprises:

computing vector dot-products for each vector of the parameters;

modifying the vector dot-products according to the per-vector scale factor corresponding to each vector of the parameters to produce scaled vector dot-products for each vector of the parameters; and

accumulating the scaled vector dot-products for each vector of the parameters to produce the elements of the multi-dimensional output tensor.

7. The computer-implemented method of claim 6 , wherein the parameters comprise weights and a second multi-dimensional parameter tensor comprises activations, and wherein modifying the vector dot-products comprises multiplying each vector dot-product by a corresponding per-vector scale factor for the weights and a corresponding per-vector scale factor for the activations used to compute the vector dot-product.

8. The computer-implemented method of claim 6 , wherein the parameters comprise weights and a second multi-dimensional parameter tensor comprises activations, and wherein modifying the vector dot-products comprises:

multiplying a corresponding per-vector scale factor for the weights and a corresponding per-vector scale factor for the activations used to compute the vector dot-product to produce a scale product factor;

rounding the scale product factor; and

multiplying each vector dot-product by the rounded scale factor to produce the scaled vector dot-products.

9. The computer-implemented method of claim 1 , wherein the coarse scale factor is one of a per-layer, per-tensor, or per-channel scale factor.

10. The computer-implemented method of claim 9 , further comprising modifying elements of the multi-dimensional output tensor computed using the at least two of the vectors of parameters by multiplying each element by the coarse scale factor.

11. The computer-implemented method of claim 9 , wherein the per-vector scale factor is an integer format number and the coarse scale factor is a floating-point format number.

12. The computer-implemented method of claim 1 , wherein at least one of the steps of computing the per-vector scale factor, computing the coarse scale factor, reducing, or processing are performed on a server or in a data center to generate data, and the data are streamed to a user device.

13. The computer-implemented method of claim 1 , wherein at least one of the steps of computing the per-vector scale factor, computing the coarse scale factor, reducing, or processing are performed within a cloud computing environment.

14. The computer-implemented method of claim 1 , wherein at least one of the steps of computing the per-vector scale factor, computing the coarse scale factor, reducing, or processing generate data that are used for training, testing, or certifying the neural network model or another neural network model that is employed in a machine, robot, or autonomous vehicle.

15. The computer-implemented method of claim 1 , wherein at least one of the steps of computing the per-vector scale factor, computing the coarse scale factor, reducing, or processing are performed on a virtual machine comprising a portion of a graphics processing unit or multi-core processor.

16. A system, comprising:

a processor configured to implement a neural network model that is configured to quantize parameters by:

computing, for each vector of parameters within a single dimension of a multi-dimensional parameter tensor, a per-vector scale factor to produce a plurality of scale factors for the multi-dimensional parameter tensor;

computing a coarse scale factor for the multi-dimensional parameter tensor that is shared by at least two of the vectors of parameters;

reducing, for each vector of parameters, bitwidth of each parameter in the vector of parameters according to a corresponding per-vector scale factor in the plurality of per-vector scale factors to generate a reduced bitwidth vector of parameters, wherein the reduced bitwidth of each of the at least two of the vectors of parameters is also reduced according to the coarse scale factor; and

processing the reduced bitwidth vectors of parameters using a set of multiplier circuits that operate at the reduced bitwidth in a layer of the neural network model to produce each element of a multi-dimensional output tensor.

17. The system of claim 16 , wherein the parameters are activations and the per-vector scale factors are dynamically calculated at each layer of the neural network model during inferencing.

18. The system of claim 16 , wherein the processing comprises:

computing vector dot-products for each vector of the parameters;

modifying the vector dot-products according to the per-vector scale factor corresponding to each vector of the parameters to produce scaled vector dot-products for each vector of the parameters; and

accumulating the scaled vector dot-products for each vector of the parameters to produce the elements of the multi-dimensional output tensor.

19. A non-transitory computer-readable media storing computer instructions for quantizing parameters that, when executed by one or more processors, cause the one or more processors to perform the steps of:

computing, for each vector of parameters within a single dimension of a multi-dimensional parameter tensor, a scale factor to produce a plurality of scale factors for the multi-dimensional parameter tensor;

computing a coarse scale factor for the multi-dimensional parameter tensor that is shared by at least two of the vectors of parameters;

reducing, for each vector of parameters, a bitwidth of each parameter in the vector of parameters according to a corresponding per-vector scale factor in the plurality of per-vector scale factors to generate a reduced bitwidth vector of parameters, wherein the reduced bitwidth of each of the at least two of the vectors of parameters is also reduced according to the coarse scale factor; and

processing the reduced bitwidth vectors of parameters using a set of multiplier circuits that operate at the reduced bitwidth in a layer of the neural network model to produce each element of a multi-dimensional output tensor.

20. The non-transitory computer-readable media of claim 19 , wherein the parameters are activations and the per-vector scale factors are dynamically calculated at each layer of the neural network model during inferencing.

21. The non-transitory computer-readable media of claim 19 , wherein the processing comprises:

computing vector dot-products for each vector of the parameters;

modifying the vector dot-products according to the per-vector scale factor corresponding to each vector of the parameters to produce scaled vector dot-products for each vector of the parameters; and

accumulating the scaled vector dot-products for each vector of the parameters to produce the elements of the multi-dimensional output tensor.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 2, 2020
From: KHAILANY, BRUCEK KURDO; DAI, STEVE HAIHANG; VENKATESAN, RANGHARAJAN; REN, HAOXING
To: NVIDIA CORPORATION
Reel/Frame 054236/0642 →
Continuity (3)
Provisional Application 63089889 · Oct 9, 2020
Provisional Application 63071949 · Aug 28, 2020
Related Publication 20220067512A1 · Mar 3, 2022
References Cited (82)
US 11520561B1 · Afzal · 2022 [cited by examiner]
US 20160149745A1 · Singh · 2016 [cited by examiner]
US 20180314940A1 · Kundu et al. · 2018 [cited by applicant]
US 20180322607A1 · Mellempudi et al. · 2018 [cited by applicant]
US 20190171927A1 · Diril · 2019 [cited by examiner]
US 20200193273A1 · Chung · 2020 [cited by examiner]
US 20210110508A1 · Mellempudi et al. · 2021 [cited by applicant]
US 20210149670A1 · Stevens · 2021 [cited by examiner]
US 20210350210A1 · Gong · 2021 [cited by examiner]
US 20210406682A1 · Meeyakhan Rawther · 2021 [cited by examiner]
US 20220366238A1 · Liu · 2022 [cited by examiner]
CN 106909649A · 2017 [cited by applicant]
CN 115730653A · 2023 [cited by applicant]
WO 2020024093A1 · 2020 [cited by applicant]
Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey (Year: 2020). [cited by examiner]
Quantizing Oriented Object Detection Network via Outlier-Aware Quantization and IoU Approximation (Year: 2020). [cited by examiner]
Fine Grained Urban Flow Inference (Year: 2020). [cited by examiner]
Compressing Deep Convolutional Networks Using Vector Quantization (Year: 2014). [cited by examiner]
Bhandare, A., et al., “Efficient 8-bit quantization of transformer neural machine language translation model,” arXiv preprint arXiv:1906:00532, 2019. [cited by applicant]
Choi, J., et al., “PACT: Parameterized clipping activation for quantized neural networks,” arXiv preprint arXiv:1805.06085, 2018. [cited by applicant]
Courbariaux, M., et al., “Binary Connect: Training deep neural networks with binary weights during propagations,” In Advances in neural information processing systems, pp. 3123-3131, 2015. [cited by applicant]
Fang, J., et al., “Near-lossless post-training quantization of deep neural networks via a piecewise linear approximation,” arXiv preprint arXiv:2002.00104, 2020. [cited by applicant]
Gong, Y., et al., “Compressing deep convolutional networks using vector quantization,” arXiv preprint arXiv:1412.6115, 2014. [cited by applicant]
Gray, R., “Vector quantization,” IEEE Assp Magazine, 1(2) 4-29, 1984. [cited by applicant]
Hubara, I., et al., “Binarized neural networks,” In Advances in neural information processing systems, pp. 4107-4115, 2016. [cited by applicant]
Jain, S., et al., “Quantizing long-tailed datastructures with two scale factors for deep neural networks,” In Proc. DAC, pp. 1-6, 2019. [cited by applicant]
Khoram, S., et al., “Adaptive quantization of neural networks,” In International Conference on Learning Representations, 2018. [cited by applicant]
Koster, U., et al., “An adaptive numerical format for efficient training of deep neural networks,” In Advances in neural information processing systems, pp. 1742-1752, 2017. [cited by applicant]
Krishnamoorthi, R., “Quantizing deep convolutional networks for efficient inference: A Whitepaper,” arXiv preprint arXiv:1806.08342, 2018. [cited by applicant]
Lecun, Y., et al., “Deep Learning,” nature, 521(7553):436-444, 2015. [cited by applicant]
Lee, J., et al., “Quantization for rapid deployment of deep neural networks,” arXiv preprint arXiv:1810.05488, 2018. [cited by applicant]
McKinstry, J., et al., “Discovering low-precision networks close to full-precision networks for efficient embedded inference,” arXiv preprint arXiv:1809.04191, 2018. [cited by applicant]
Mishra, A., et al., “Apprentice: Using knowledge distillation techniques to improve low-precision network accuracy,” arXiv preprint arXiv:1711.05852, 2017. [cited by applicant]
Miyashita, D., et al., “Convolutional neural networks using logarithmic data representation,” arXiv preprint arXiv: 1603.01025, 2016. [cited by applicant]
Moons, B., et al., “Minimum energy quantized neural networks,” In 2017 51 [cited by applicant]
Nagel, M., et al., “Data-free quantization through weight equalization and bias correction,” In Proceedings of the IEEE International Conference on Computer Vision, pp. 1325-1334, 2019. [cited by applicant]
Prato, G., et al., “Fully quantized transformer for improved translation,” arXiv preprint arXiv:1910.10485, 2019. [cited by applicant]
Shen, S., et al., “Q-bert: Hessian based ultra low precision quantization of bert,” In AAAI, pp. 8815-8821, 2020. [cited by applicant]
Sijstermans, F., “The NVIDIA deep learning accelerator.” [cited by applicant]
Stock, P., et al., “And the bit goes down: Revisiting the quantization of neural networks,” In Eighth International Conference on Learning Representations, 2020. [cited by applicant]
Sze, V., et al., “Efficient processing of deep neural networks,” Synthesis Lectures on Computer Architecture, 15(2):1-341, 2020. [cited by applicant]
Tambe, T., et al., “Algorithm-hardware co-design of adaptive floating-point encodings for resilient deep learning inference,” Design Automation Conference (DAC), 2020. [cited by applicant]
Venkatesan, R., et al., “MAGNet: A modular accelerator generator for neural networks,” In ICCAD, pp. 1-8, 2019. [cited by applicant]
Wu, B., et al., “Mixed precision quantization of convnets via differentiable neural architecture search,” arXiv preprint arXiv:1812.00090, 2018. [cited by applicant]
Wu, H., “Low precision inference on gpus,” GPU Technology Conference (GTC), 2019. [cited by applicant]
Wu, H., et al., “Integer quantization for deep learning inference: Principles and empirical evaluation,” arXiv preprint arXiv:2004.09602, 2020. [cited by applicant]
Wu, J., et al., “Quantized convolutional neural networks for mobile devices,” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4820-4828, 2016. [cited by applicant]
Wu, Y., et al., “Google's neural machine translation system: Bridging the gap between human machine translation.” arXiv preprint arXiv:1609.08144, 2016. [cited by applicant]
Zafrir, O., et al., “Q8bert: Quantized 8bit bert,” arXiv preprint arXiv:1910.06188, 2019. [cited by applicant]
Zhao, R., et al., “Improving neural network quantization without retraining using outlier channel splitting,” In International Conference on Machine Learning, pp. 7543-7552, 2019. [cited by applicant]
Zhu, C., et al., “Trained ternary quantization,” arXiv preprint arXiv:1612.01064, 2016. [cited by applicant]
Dai, S., et al., “VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inference,” arXiv:2102.04503, 2021. [cited by applicant]
Mellempudi, N., et al., “Ternary neural networks with fine-grained quantization,” arXiv preprint arXiv:1705.01462, 2017. [cited by applicant]
Wikipedia.org, “Artificial neural network,” retrieved from https://en.wikipedia.org/wiki/Artificial_neural_network, 2024. [cited by applicant]
Bubek, S., et al., “Sparks of artificial general intelligence: Early experiments with GPT-4,” arXiv preprint arXiv:2303.12712, 2023. [cited by applicant]
Lin, J., et al., “AWQ: Activation-aware weight quantization for LLM compression and acceleration,” 2023. [cited by applicant]
Frantar, E., et al., “GPTQ: Accurate quantization for generative pre-trained transformers,” in The Eleventh International Conference on Learning Representations, 2023. [cited by applicant]
Kim, S., et al., “SqueezeLLM: Dense-and-sparse quantization,” 2023. [cited by applicant]
Park, E., et al., “Energy-efficient neural network accelerator based on outlier-aware low-precision computation,” in 2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA), pp. 688-698, 2018. [cited by applicant]
Xiao, G., et al., “SmoothQuant: Accurate and efficient post-training quantization for large language models,” 2023. [cited by applicant]
Choi, J., et al., “PACT: Parameterized clipping activation for quantized neural networks,” 2018. [cited by applicant]
Sakr, C., et al., “Optimal clipping and magnitude-aware differentiation for improved quantization-aware training,” 2022. [cited by applicant]
Dettmers, T., et al., “The case for 4-bit precision: k-bit inference scaling laws,” 2023. [cited by applicant]
Dai, S., et al., “VS-Quant: Per-vector scaled quantization for accurate low-precision neural network inference,” In Proceedings of Machine Learning and Systems, vol. 3, pp. 873-884, 2021. [cited by applicant]
Rouhani, B., et al., “With shared microexponents, a little shifting goes a long way,” 2023. [cited by applicant]
Rouhani, B., et al., “Pushing the limits of narrow precision inferencing at cloud scale with microsoft floating point,” in Advances in Neural Information Processing Systems, vol. 33, pp. 10271-10281, Curran Associates, … [cited by applicant]
Wu, X.,, et al., “ZeroQuant-FP: A leap forward in 1lms post-training W4A8 quantization using floating-point formats,” 2023. [cited by applicant]
Lo, Y.C., et al., “Block and subword-scaling floating-point (BSFP): An efficient non-uniform quantization for low precision inference,” In the Eleventh International Conference on Learning Representations, 2023. [cited by applicant]
Lloyd, S., et al., “Least squares quantization in PCM,” IEEE Transactions on Information Theory, vol. 28, No. 2, pp. 129-137, 1982. [cited by applicant]
Han, S., et al., “Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding,” in 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, … [cited by applicant]
Bondarenko, Y., et al., “Understanding and overcoming the challenges of efficient transformer quantization,” 2021. [cited by applicant]
Tambe, T., et al., “Algorithm-hardware co-design of adaptive floating-point encodings for resilient deep learning inference,” in 2020 57th ACM/IEEE Design Automation Conference (DAC), pp. 1-6, 2020. [cited by applicant]
Bai, H., et al., “Towards effiecient post-training quantization of pre-trained language models,” 2021. [cited by applicant]
Yao, Z., et al., “ZeroQuant: Efficient and affordable post-training quantization for large-scale transformers,” in Advances in Neural Information Processing Systems, 2022. [cited by applicant]
Dettmers, T., et al., “LLM.int8(): 8-bit matrix multiplication for transformers at scale,” 2022. [cited by applicant]
Wei, X., et al., “Outlier suppression: Pushing the limit of low-bit transformer language models,” 2023. [cited by applicant]
Guo, C., et al., “OliVe: Accelerating large language models via hardware-friendly outlier-victim pair quantization,” ISCA '23 (New York, NY), Association for Computing Machinery, 2023. [cited by applicant]
Chee, J., et al., “QuIP: 2-bit quantization of large language models with guarantees,” 2023. [cited by applicant]
Kim, Y.J., et al., “FineQuant: Unlocking efficiency with fine-grained weight-only quantization for LLMs,” 2023. [cited by applicant]
Yao, Z., et al., “ZeroQuant-V2: Exploring post-training quantization in LLMs from comprehensive study to low rank compensation,” 2023. [cited by applicant]
Zhang, Y., et al., “Integer or floating point? New outlooks for low-bit quantization on large language models,” 2023. [cited by applicant]
Koster, U., et al., “Flexpoint: An adaptive numerical format for efficient training of deep neural networks,” 2017. [cited by applicant]