Granted Patent B2

US 12,530,560 · App. 17/072,485 · Granted Jan 20, 2026

Inventors: Pan Zhou (Singapore, SG); Chu Hong Hoi (Singapore, SG)

Assignee: Salesforce, Inc.

G06N3/04G06N3/08

Abstract

Claims (76)

1 . A method for providing a multi-layer neural network, comprising:

initializing the multi-layer neural network including a plurality of cells, each cell corresponding to a graph including a plurality of nodes, each node corresponding to a latent representation of a dataset;

generating a plurality of gates for a first cell, wherein each gate independently determines whether an operation between two nodes in the first cell is used, wherein the first cell includes:

two input nodes corresponding to outputs of two previous cells respectively,

one or more intermediate nodes connecting with previous nodes in the cell, and

an output node provided by concatenating the one or more intermediate nodes;

performing a first regularization using the plurality of gates, wherein the first regularization includes a group-structured sparsity regularization and a path-depth-wise regularization,

wherein the group-structured sparsity regularization includes:

assigning a first regularization constant to a skip connection group; and

assigning a second regularization constant to a non-skip connection group;

wherein the first regularization constant is larger than the second regularization constant; and

wherein the path-depth-wise regularization is configured for correcting cell-selection bias caused by the plurality of gates in group-structured sparsity regularization, wherein the cell-selection bias is generated after intermediate nodes are connected with input nodes in the multi-layer neural network; and

performing an optimization on the multi-layer neural network by adjusting its network parameters and gate parameters based on the first regularization using the first regularization constant and the second regularization constant.

2 . The method of claim 1 , wherein the performing the group-structured sparsity regularization includes:

generating two or more groups from the plurality of gates based on a corresponding operation type of each gate; and

performing a regularization of sparsity of each of the two or more groups.

3 . The method of claim 2 , wherein the performing the group-structured sparsity regularization includes:

generating a first group loss based on activation probability of a first group of the two or more groups; and

generating a second group loss based on activation probability of a second group of the two or more groups;

wherein the optimization is performed based on the first group loss and the second group loss.

4 . The method of claim 1 , wherein the performing the path-depth-wise regularization includes:

generating a path-depth-wise loss using probabilities based on a depth of a path of the multi-layer neural network;

wherein the optimization is performed based on the path-depth-wise loss.

5 . The method of claim 1 , wherein the operation is one of skip connection, zero operation, and pooling operation, separable convolutions, dilated separable convolutions, average pooling and max pooling.

6 . The method of claim 1 , wherein the optimization uses gradient descent.

7 . A non-transitory machine-readable medium comprising a plurality of machine-readable instructions which, when executed by one or more processors, are adapted to cause the one or more processors to perform a method comprising:

initializing a multi-layer neural network including a plurality of cells, each cell corresponding to a graph including a plurality of nodes, each node corresponding to a latent representation of a dataset;

generating a plurality of gates for a first cell, wherein each gate independently determines whether an operation between two nodes in the first cell is used, wherein the first cell includes:

two input nodes corresponding to outputs of two previous cells respectively,

one or more intermediate nodes connecting with previous nodes in the cell, and

an output node provided by concatenating the one or more intermediate nodes;

performing a first regularization using the plurality of gates, wherein the first regularization includes a group-structured sparsity regularization and a path-depth-wise regularization,

wherein performing the group-structured sparsity regularization includes:

assigning a first regularization constant to a skip connection group; and

assigning a second regularization constant to a non-skip connection group;

wherein the first regularization constant is larger than the second regularization constant; and

wherein the path-depth-wise regularization is configured for correcting cell-selection bias caused by the plurality of gates in group-structured sparsity regularization, wherein the cell-selection bias is generated after intermediate nodes are connected with input nodes in the multi-layer neural network via skip connections and after connections between neighboring nodes are cut via zero operation; and

performing an optimization on the multi-layer neural network by adjusting its network parameters and gate parameters based on the first regularization using the first regularization constant and the second regularization constant.

8 . The non-transitory machine-readable medium of claim 7 , wherein the performing the group-structured sparsity regularization includes:

generating two or more groups from the plurality of gates based on a corresponding operation type of each gate; and

performing a regularization of sparsity of each of the two or more groups.

9 . The non-transitory machine-readable medium of claim 8 , wherein the performing the group-structured sparsity regularization includes:

generating a first group loss based on activation probability of a first group of the two or more groups; and

generating a second group loss based on activation probability of a second group of the two or more groups;

wherein the optimization is performed based on the first group loss and the second group loss.

10 . The non-transitory machine-readable medium of claim 7 , wherein the performing the path-depth-wise regularization includes:

generating a path-depth-wise loss using probabilities based on a depth of a path of the multi-layer neural network;

wherein the optimization is performed based on the path-depth-wise loss.

11 . The non-transitory machine-readable medium of claim 7 , wherein the operation is one of skip connection, zero operation, and pooling operation, separable convolutions, dilated separable convolutions, average pooling and max pooling.

12 . The non-transitory machine-readable medium of claim 7 , wherein the optimization uses gradient descent.

13 . A system, comprising:

a non-transitory memory; and

one or more hardware processors coupled to the non-transitory memory and configured to read instructions from the non-transitory memory to cause the system to perform a method comprising:

initializing a multi-layer neural network including a plurality of cells, each cell corresponding to a graph including a plurality of nodes, each node corresponding to a latent representation of a dataset;

generating a plurality of gates for a first cell, wherein each gate independently determines whether an operation between two nodes in the first cell is used, wherein the first cell includes:

two input nodes corresponding to outputs of two previous cells respectively,

one or more intermediate nodes connecting with previous nodes in the cell, and

an output node provided by concatenating the one or more intermediate nodes;

performing a first regularization using the plurality of gates, wherein the first regularization includes a group-structured sparsity regularization and a path-depth-wise regularization,

wherein performing the group-structured sparsity regularization includes:

assigning a first regularization constant to a skip connection group; and

assigning a second regularization constant to a non-skip connection group;

wherein the first regularization constant is larger than the second regularization constant; and

wherein the path-depth-wise regularization is configured for correcting cell-selection bias caused by the plurality of gates in group-structured sparsity regularization, wherein the cell-selection bias is generated after intermediate nodes are connected with input nodes in the multi-layer neural network via skip connections and after connections between neighboring nodes are cut via zero operation; and

performing an optimization on the multi-layer neural network by adjusting its network parameters and gate parameters based on the first regularization using the first regularization constant and the second regularization constant.

14 . The system of claim 13 , wherein the performing the group-structured sparsity regularization includes:

generating two or more groups from the plurality of gates based on a corresponding operation type of each gate; and

performing a regularization of sparsity of each of the two or more groups.

15 . The system of claim 14 , wherein the performing the group-structured sparsity regularization includes:

generating a first group loss based on activation probability of a first group of the two or more groups; and

generating a second group loss based on activation probability of a second group of the two or more groups;

wherein the optimization is performed based on the first group loss and the second group loss.

16 . The system of claim 13 , wherein the performing the path-depth-wise regularization includes:

generating a path-depth-wise loss using probabilities based on a depth of a path of the multi-layer neural network;

wherein the optimization is performed based on the path-depth-wise loss.

17 . The system of claim 13 , wherein the operation is one of skip connection, zero operation, and pooling operation, separable convolutions, dilated separable convolutions, average pooling and max pooling.

Assignments (2)

Continuity (2)

References Cited (135)