Replacement of neural network gradient values in transmission data
Apparatuses, systems, and techniques to replace one or more corrupt neural network gradient values with one or more corresponding surrogate neural network gradient values. In at least one embodiment, faulty gradient values are identified in a transmitted data packet and are replaced according to a replacement policy with a suitable surrogate value.
1 . One or more processors, comprising:
circuitry to replace one or more neural network gradient values with one or more corresponding surrogate neural network gradient values, wherein the circuitry is to train a neural network using the one or more surrogate neural network gradient values, and wherein the one or more corresponding surrogate neural network gradient values are all zero values or are based, at least in part, on at least one of an average of recently received gradient values or a gradient value most recently received.
2 . The one or more processors of claim 1 , wherein the one or more neural network gradient values are transmitted in a packet, and wherein the circuitry is to perform a cyclic redundancy check on a first checksum value of a packet header portion and a second checksum value of a packet data portion.
3 . The one or more processors of claim 1 , wherein the circuitry is to store the one or more corresponding surrogate neural network gradient values in memory.
4 . The one or more processors of claim 1 , wherein the average of the recently received gradient values is computed based on a mean and standard deviation of the recently received gradient values.
5 . The one or more processors of claim 1 , wherein the circuitry is to perform forward error correction (FEC) on the one or more neural network gradient values prior to replacement of the one or more neural network gradient values with the one or more corresponding surrogate neural network gradient values.
6 . The one or more processors of claim 1 , wherein the circuitry is to generate a neural network loss function using stochastic gradient descent based, at least in part, on the one or more surrogate neural network gradient values.
7 . The one or more processors of claim 1 , wherein the one or more neural network gradient values have been corrupted during transmission within a communication network.
8 . A system comprising:
one or more processors to replace one or more neural network gradient values with one or more corresponding surrogate neural network gradient values, wherein the one or more processors are to train a neural network using the one or more surrogate neural network gradient values, and wherein the one or more corresponding surrogate neural network gradient values are all zero values or are based, at least in part, on at least one of an average of recently received gradient values or a gradient value most recently received.
9 . The system of claim 8 , wherein the one or more neural network gradient values are transmitted in a packet, and wherein the one or more processors are to perform a cyclic redundancy check on a first checksum value of a packet header portion and a second checksum value of a packet data portion.
10 . The system of claim 8 , wherein the one or more processors are to store the one or more corresponding surrogate neural network gradient values in memory.
11 . The system of claim 8 , wherein the average of the recently received gradient values is computed based on a mean and standard deviation of the recently received gradient values.
12 . The system of claim 8 , wherein the one or more processors are to perform forward error correction (FEC) on the one or more neural network gradient values prior to replacement of the one or more neural network gradient values with the one or more corresponding surrogate neural network gradient values.
13 . The system of claim 8 , wherein the one or more processors are to generate a neural network loss function using stochastic gradient descent based, at least in part, on the one or more surrogate neural network gradient values.
14 . The system of claim 8 , wherein the one or more neural network gradient values have been corrupted during transmission within a communication network.
15 . A method comprising:
replacing one or more neural network gradient values with one or more corresponding surrogate neural network gradient values; and
training a neural network using the one or more surrogate neural network gradient values, wherein the one or more corresponding surrogate neural network gradient values are all zero values or are based, at least in part, on at least one of an average of recently received gradient values or a gradient value most recently received.
16 . The method of claim 15 , further comprising:
transmitting the one or more neural network gradient values in a packet; and
performing a cyclic redundancy check on a first checksum value of a packet header portion and a second checksum value of a packet data portion.
17 . The method of claim 15 , further comprising storing the one or more corresponding surrogate neural network gradient values in memory.
18 . The method of claim 15 , further comprising computing the average of the recently received gradient values based on a mean and standard deviation of the recently received gradient values.
19 . The method of claim 15 , further comprising performing forward error correction (FEC) on the one or more neural network gradient values prior to replacement of the one or more neural network gradient values with the one or more corresponding surrogate neural network gradient values.
20 . The method of claim 15 , further comprising generating a neural network loss function using stochastic gradient descent based, at least in part, on the one or more surrogate neural network gradient values.