IP Library › Granted Patent US 11,727,267
Granted Patent B2
US 11,727,267 · App. 16/556,937 · Granted Aug 15, 2023

Artificial neural network with trainable activation functions and fractional derivative values

Inventors: Julio Cesar Zamora Esquivel (Zapopan, MX); Jose Rodrigo Camacho Perez (Guadalajara, MX); Paulo Lopez Meyer (Zapopan, MX); Hector Cordourier Maruri (Guadalajara, MX); Jesus Cruz Vargas (Zapopan, MX)
Assignee: Intel Corporation
G06N3/08G06N3/047
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,727,267
App. No.
16/556,937
Granted
Aug 15, 2023
Kind
B2
Abstract

Systems, apparatuses and methods may provide for technology that adjusts a plurality of weights in a neural network model and adjusts a plurality of activation functions in the neural network model. The technology may also output the neural network model in response to one or more conditions being satisfied by the plurality of weights and the plurality of activation functions. In one example, two or more of the activation functions are different from one another and the activation functions are adjusted on a per neuron basis.

Claims (53)

1. A computing system comprising:

a network controller;

a processor coupled to the network controller; and

a memory coupled to the processor, the memory including a set of instructions, which when executed by the processor, cause the computing system to:

adjust a plurality of weights in a neural network model,

generate partial derivatives of a plurality of activation functions in the neural network model with respect to fractional derivative values, wherein the fractional derivative values are part of the plurality of activation functions and control whether the plurality of activation functions morph during training,

generate a plurality of difference values based on the partial derivatives,

adjust the fractional derivative values in the plurality of activation functions based on the plurality of difference values and on a per neuron basis to modify the plurality of activation functions in the neural network model to morph the plurality of activation functions, and

output the neural network model in response to one or more conditions being satisfied by the plurality of weights and the plurality of activation functions.

2. The computing system of claim 1 , wherein two or more of the plurality of activation functions are to be different from one another.

3. The computing system of claim 1 , wherein to modify the plurality of activation functions, the instructions, when executed, cause the computing system to:

apply a first difference value from the plurality of difference values to a first activation function of the plurality of activation functions that is associated with a first neuron; and

apply a second difference value from the plurality of difference values to a second activation function of the plurality of activation functions that is associated with a second neuron.

4. The computing system of claim 1 , wherein the instructions, when executed, cause the computing system to select the plurality of activation functions from a library of trainable activation functions.

5. The computing system of claim 1 , wherein the one or more conditions include an accuracy condition.

6. A semiconductor apparatus comprising:

one or more substrates; and

logic coupled to the one or more substrates, wherein the logic is implemented at least partly in one or more of configurable logic or fixed-functionality hardware logic, the logic coupled to the one or more substrates to:

adjust a plurality of weights in a neural network model,

generate partial derivatives of a plurality of activation functions in the neural network model with respect to fractional derivative values, wherein the fractional derivative values are part of the plurality of activation functions and control whether the plurality of activation functions morph during training,

generate a plurality of difference values based on the partial derivatives,

adjust the fractional derivative values in the plurality of activation functions based on the plurality of difference values and on a per neuron basis to modify the plurality of activation functions in the neural network model to morph the plurality of activation functions, and

output the neural network model in response to one or more conditions being satisfied by the plurality of weights and the plurality of activation functions.

7. The semiconductor apparatus of claim 6 , wherein two or more of the plurality of activation functions are to be different from one another.

8. The semiconductor apparatus of claim 6 , wherein to modify the plurality of activation functions, the logic coupled to the one or more substrates is to:

apply a first difference value from the plurality of difference values to a first activation function of the plurality of activation functions that is associated with a first neuron; and

apply a second difference value from the plurality of difference values to a second activation function of the plurality of activation functions that is associated with a second neuron.

9. The semiconductor apparatus of claim 6 , wherein the logic coupled to the one or more substrates is to select the plurality of activation functions from a library of trainable activation functions.

10. The semiconductor apparatus of claim 6 , wherein the one or more conditions include an accuracy condition.

11. At least one non-transitory computer readable storage medium comprising a set of instructions, which when executed by a computing system, cause the computing system to:

adjust a plurality of weights in a neural network model;

generate partial derivatives of a plurality of activation functions in the neural network model with respect to fractional derivative values, wherein the fractional derivative values are part of the plurality of activation functions and control whether the plurality of activation functions morph during training;

generate a plurality of difference values based on the partial derivatives;

adjust the fractional derivative values in the plurality of activation functions based on the plurality of difference values and on a per neuron basis to modify the plurality of activation functions in the neural network model; and

output the neural network model in response to one or more conditions being satisfied by the plurality of weights and the plurality of activation functions.

12. The at least one non-transitory computer readable storage medium of claim 11 , wherein two or more of the plurality of activation functions are to be different from one another.

13. The at least one non-transitory computer readable storage medium of claim 11 , wherein to adjust the plurality of activation functions, the instructions, when executed, cause the computing system to:

apply a first difference value from the plurality of difference values to a first activation function of the plurality of activation functions that is associated with a first neuron; and

apply a second difference value from the plurality of difference values to a second activation function of the plurality of activation functions that is associated with a second neuron.

14. The at least one non-transitory computer readable storage medium of claim 11 , wherein the instructions, when executed, cause the computing system to select the plurality of activation functions from a library of trainable activation functions.

15. The at least one non-transitory computer readable storage medium of claim 11 , wherein the one or more conditions include an accuracy condition.

16. A method comprising:

adjusting a plurality of weights in a neural network model;

generating partial derivatives of a plurality of activation functions in the neural network model with respect to fractional derivative values, wherein the fractional derivative values are part of the plurality of activation functions and control whether the plurality of activation functions morph during training;

generating a plurality of difference values based on the partial derivatives;

adjusting the fractional derivative values in the plurality of activation functions based on the plurality of difference values and on a per neuron basis to modify the plurality of activation functions in the neural network model to morph the plurality of activation functions; and

outputting the neural network model in response one or more conditions being satisfied by the plurality of weights and the plurality of activation functions.

17. The method of claim 16 , wherein two or more of the plurality of activation functions are different from one another.

18. The method of claim 16 , wherein modifying the plurality of activation functions includes:

applying a first difference value from the plurality of difference values to a first activation function of the plurality of activation functions that is associated with a first neuron; and

applying a second difference value from the plurality of difference values to a second activation function of the plurality of activation functions that is associated with a second neuron.

19. The method of claim 16 , further including selecting the plurality of activation functions from a library of trainable activation functions.

20. The method of claim 16 , wherein the one or more conditions include an accuracy condition.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 21, 2021
From: ZAMORA ESQUIVEL, JULIO CESAR; CAMACHO PEREZ, JOSE RODRIGO; LOPEZ MEYER, PAULO; CORDOURIER MARURI, HECTOR; CRUZ VARGUS, JESUS
To: INTEL CORPORATION
Reel/Frame 056603/0421 →
Continuity (1)
Related Publication 20200005143A1 · Jan 2, 2020