Granted Patent B2

US 12,711,385 · App. 18/003,821 · Granted Aug 18, 2026

Inventors: Yufeng Gao (Xi'an, CN); Shibing Zhu (Xi'an, CN); Shaoli Liu (Xi'an, CN); Xishan Zhang (Xi'an, CN); Deyuan He (Xi'an, CN)

G06N3/084G06N3/082

Abstract

Claims (70)

1 . A method for sparsification training of a neural network model by a data processing apparatus, comprising:

performing sparsification processing on a neural network parameter based on a mask tensor to compute a value of a loss function in forward propagation;

computing a neuron gradient and a neural network parameter gradient based on the loss function in back propagation; and

updating the neural network parameter based on the neural network parameter gradient.

2 . The method of claim 1 , further comprising:

computing the neuron gradient and the neural network parameter gradient based on an unsparsified neural network parameter in the back propagation; and

updating the neural network parameter based on the neural network parameter gradient.

3 . The method of claim 2 , further comprising:

performing anti-sparsification processing on a sparsified neural network parameter to obtain the unsparsified neural network parameter.

4 . The method of claim 1 , further comprising:

computing the neuron gradient and the neural network parameter gradient based on the neural network parameter sparsified in the back propagation; and

updating the neural network parameter based on the neuron gradient.

5 . The method of claim 4 , further comprising:

performing sparsification processing on the neural network parameter based on a reverse mask tensor to obtain the neural network parameter sparsified in the back propagation.

6 . The method of claim 5 , wherein the mask tensor is a two-dimensional tensor, wherein the two-dimensional tensor performs sparsification processing on an input channel dimension of the neural network parameter and an output channel dimension of the neural network parameter, and wherein the reverse mask tensor is generated by performing a dimension conversion on the mask tensor.

7 . The method of claim 1 , wherein the mask tensor is a one-dimensional tensor, and wherein the one-dimensional tensor performs sparsification processing on an input channel dimension of the neural network parameter.

8 . The method of claim 1 , wherein the updating the neural network parameter comprises updating an unsparsified neural network parameter;

wherein the method further comprises generating the mask tensor based on the updated unsparsified neural network parameter;

wherein when the mask tensor is a one-dimensional tensor, the method generates the mask tensor by:

selecting n data elements with relatively large absolute values as valid data elements from every m data elements of a specified dimension of the neural network parameter, wherein m>n; and

determining the mask tensor based on positions of n valid data elements in the m data elements;

and wherein when the mask tensor is a two-dimensional tensor, the method generates the mask tensor by:

presetting a specific number of two-dimensional mask tensors, wherein each dimension of the two-dimensional mask tensors comprises m elements, of which n elements are 1, m-n elements are 0, and m>n;

masking two specified dimensions of the neural network parameter respectively based on each preset two-dimensional mask tensor to obtain a masked parameter tensor;

performing a product sum operation on training data of a neural network based on each masked parameter tensor to obtain a parameter evaluation value; and

selecting a two-dimensional mask tensor that generates the largest of all parameter evaluation values as the mask tensor.

9 . The method of claim 1 , wherein the method is performed in multiple iterations in a mask adjustment phase of sparsification training; wherein the mask adjustment phase further comprises:

obtaining a percentage of element values of the mask tensor that are not changed in all element values of the mask tensor, and determining whether the percentage reaches a threshold in multiple consecutive iterative training; and

ending the mask adjustment phase if the percentage that all element values of the mask tensor are not changed reaches the threshold in the multiple consecutive iterative training;

and wherein the threshold is one of 80%, 90%, and 100%.

10 . The method of claim 1 , wherein the method is performed in multiple iterations in a mask fixing phase of sparsification training, and the mask tensor is fixed as a mask tensor that is finally determined in a previous phase;

wherein updating the neural network parameter comprises:

using the mask tensor to perform sparsification processing on the neuron gradient; and

updating the sparsified neural network parameter based on the sparsified neuron gradient;

and wherein during the mask fixing phase, fixed mask tensors and the sparsified neural network parameter are stored, or fixed mask tensors and unsparsified neural network parameter are stored.

11 . A data processing apparatus, comprising a control circuit, a storage circuit, and an operation circuit, wherein

the control circuit is configured to control the storage circuit and the operation circuit to perform sparsification training on a neural network model;

the storage circuit is configured to store information, wherein the information comprises a neural network parameter and a mask tensor; and

the operation circuit is configured to perform following operations under the control of the control circuit:

performing sparsification processing on the neural network parameter based on the mask tensor to compute a value of a loss function in forward propagation;

computing a neuron gradient and a neural network parameter gradient based on the loss function in back propagation; and

updating the neural network parameter based on the neural network parameter gradient.

12 . The apparatus of claim 11 , wherein the operation circuit is further configured to:

compute the neuron gradient and the neural network parameter gradient based on an unsparsified neural network parameter in the back propagation; and

update the neural network parameter based on the neural network parameter gradient.

13 . The apparatus of claim 12 , wherein the operation circuit is further configured to:

perform anti-sparsification processing on a sparsified neural network parameter to obtain the unsparsified neural network parameter.

14 . The apparatus of claim 11 , wherein the operation circuit is further configured to:

compute the neuron gradient and the neural network parameter gradient based on the neural network parameter sparsified in the back propagation; and

update the neural network parameter based on the neuron gradient.

15 . The apparatus of claim 14 , wherein the operation circuit is further configured to:

perform sparsification processing on the neural network parameter based on a reverse mask tensor to obtain the neural network parameter sparsified in the back propagation.

16 . The apparatus of claim 15 , wherein the mask tensor is a two-dimensional tensor, wherein the two-dimensional tensor performs sparsification processing on an input channel dimension of the neural network parameter and an output channel dimension of the neural network parameter, and wherein when the mask tensor is a two-dimensional tensor, the reverse mask tensor is generated by performing a dimension conversion on the mask tensor by the operational circuit.

17 . The apparatus of claim 11 , wherein the mask tensor is a one-dimensional tensor and wherein the one-dimensional tensor performs sparsification processing on an input channel dimension of the neural network parameter.

18 . The apparatus of claim 11 , wherein the operation circuit is further configured to:

update an unsparsified neural network parameter and generate the mask tensor based on the updated unsparsified neural network parameter;

wherein when the mask tensor is a one-dimensional tensor, the operation circuit is configured to generate the mask tensor by:

selecting n data elements with relatively large absolute values as valid data elements from every m data elements of a specified dimension of the neural network parameter, wherein m>n; and

determining the mask tensor based on positions of n valid data elements in the m data elements;

and wherein when the mask tensor is a two-dimensional tensor, the operation circuit is configured to generate the mask tensor by:

presetting a specific number of two-dimensional mask tensors, wherein each dimension of the two-dimensional mask tensors comprises m elements, of which n elements are 1, m-n elements are 0, and m>n;

masking two specified dimensions of the neural network parameter respectively based on each preset two-dimensional mask tensor to obtain a masked parameter tensor;

performing a product sum operation on training data of a neural network based on each masked parameter tensor to obtain a parameter evaluation value; and

selecting a two-dimensional mask tensor that generates the largest of all parameter evaluation values as the mask tensor.

19 . The apparatus of claim 11 , wherein the operation circuit is configured to perform operations in multiple iterations in a mask adjustment phase of sparsification training;

wherein the operation circuit is further configured to:

in the mask adjustment phase, obtaining a percentage of element values of the mask tensor are not changed in all element values of the mask tensor, and determining whether the percentage reaches a threshold in multiple consecutive iterative training; and

end the mask adjustment phase if the percentage that all element values of the mask tensor are not changed reaches the threshold in the multiple consecutive iterative training;

and wherein the threshold is one of 80%, 90%, and 100%.

20 . A chip, comprising the data processing apparatus of claim 11 .

Priority Claims (2)

Continuity (1)

References Cited (254)