Pipelining spikes during memory access in spiking neural networks
The present disclosure is directed to pipelining operations of a spiking neural network (SNN) that performs in-memory operations. To model a computer-implemented SNN after a biological neural network, the architecture in the present disclosure involves different memory sections for storing inbound spike messages, synaptic connection data, and synaptic connection parameters (e.g., states). The section of memory containing synaptic connection data to identify matching inbound spike messages. In parallel, the section of memory containing synaptic connection parameters may be accessed to perform various neuromorphic calculations, synaptic plasticity and outbound spike message generation.
1 . A system comprising:
a first memory section configured to store at least one spike message in at least one group, wherein the first memory section is organized by a number of bucket elements which each correspond to a different time step in which spike messages are received;
a second memory section configured to store data indicating a plurality of synaptic connections, each synaptic connection being associated with a source neuron and at least one target neuron, wherein the second memory section is organized by a number of bucket elements associated with each neuron;
a third memory section configured to store a synaptic connection state for each synaptic connection and a neuronal state for each neuron, wherein the third memory section is organized by a number of bucket elements associated with each neuron;
logic configured to search a bucket element associated each neuron of the second memory section in a first clock cycle with respect to a first bucket of the first memory section for a first match between at least a portion of a first spike message and a first synaptic connection, wherein the logic is configured to search for the first match by activating a group of word lines of the second memory section in parallel and activating a group of bit lines of the second memory section in parallel, and wherein the logic is configured to perform the search in response to prioritizing the first spike message according to a delay value of the first spike message;
a sense amplifier array configured to generate a bitmask that indicates the first match, wherein the bitmask signifies a source neuron identifier stored in the group of word lines that matches a first source neuron identifier of the first spike message, and wherein each bit of the bitmask indicates whether there is a match of a bit of the first source neuron identifier against a respective bit retrieved from the stored source neuron identifier; and
the logic configured to perform a neuromorphic operation in a second clock cycle with respect to the first match in parallel with searching additional bucket elements of the second memory section for an additional match between at least a portion of the first spike message and an additional synaptic connection, wherein the neuromorphic operation is performed using at least one of the synaptic connection state of the first synaptic connection or the neuronal state of a first neuron stored in a first bucket element of the third memory section;
wherein the search for the first match comprises identification of a bit line that stores at least a portion of a first synaptic connection identifier of the first synaptic connection, and wherein the bit line maps to a corresponding portion of the third memory section that is configured to store the synaptic connection state for the first synaptic connection;
wherein the logic is configured to calculate the neuronal state of the first neuron and synaptic connection state of the first synaptic connection by accessing the third memory section in parallel with searching for the additional match between the first spike message and the additional synaptic connection;
wherein the neuromorphic operation comprises determining a synaptic current associated with the first synaptic connection, determining a change in the first synaptic connection due to Long-Time Depression (LTD), and determining a change for the first synaptic connection due to Long-Time Potentiation (LTP), wherein the synaptic connection state is updated in a single time step and wherein a single time step includes processing all bucket elements by repeating the searching, matching, and neuromorphic operation for each respective bucket element.
2 . The system of claim 1 , wherein the at least one spike message is assigned to a corresponding spike group according to at least one of a time delay value, a synaptic connection, or a neuron identifier.
3 . The system of claim 1 , wherein the logic is configured to search for the first match by comparing a bit pattern of the first source neuron identifier of the first spike message to bit patterns of synaptic connection identifiers to identify the first synaptic connection.
4 . The system of claim 1 , wherein the logic is configured to calculate at least a portion of the neuronal state of the first neuron in response to performing an integration operation across synaptic connections of the first neuron.
5 . The system of claim 4 , wherein the logic is configured to store the neuronal state of the first neuron in the third memory section.
6 . The system of claim 1 , wherein the logic, the first memory section, the second memory section, and the third memory section are integrated into a memory device that is configured as a node interfacing with a plurality of other nodes that implement a spiking neural network.
7 . A system comprising:
an input filter configured to receive a spike message, the spike message comprising a first source neuron identifier;
a spike cache memory section configured to store at least one spike message in at least one group, wherein the spike cache memory section is organized by a number of bucket elements which each correspond to a different time step in which spike messages are received;
a first memory section configured to store data indicating a plurality of synaptic connections, each synaptic connection referencing at least one of a source neuron identifier of a source neuron or a target neuron identifier of a target neuron, wherein the first memory section is organized by a number of bucket elements associated with each neuron;
a second memory section configured to store a synaptic connection state for each synaptic connection, wherein the second memory section is organized by a number of bucket elements associated with each neuron;
logic configured to search a first bucket element of each neuron of the first memory section in a first clock cycle with respect to a first bucket of the spike cache memory section to identify at least one first match of the first source neuron identifier and a first synaptic connection, wherein the logic is configured to perform the search by activating a group of word lines of the first memory section in parallel and activating a group of bit lines of the first memory section in parallel, wherein the logic is configured to perform the search in response to prioritizing the spike message according to a delay value of the spike message;
the logic being configured to perform a neuromorphic operation in a second clock cycle with respect to the first match using the synaptic connection state of the first synaptic connection stored in a first bucket element of the second memory section in parallel with performing a search in the second clock cycle; and
a sense amplifier array configured to generate a bitmask that indicates the first match, wherein the bitmask signifies a source neuron identifier stored in the group of word lines that matches the first source neuron identifier of the spike message, and wherein each bit of the bitmask indicates whether there is a match of a bit of the first source neuron identifier against a respective bit retrieved from the stored source neuron identifier;
wherein the first match is identified by identifying a bit line corresponding to a matching synaptic connection, and wherein the bit line maps to a corresponding portion of the second memory section that is configured to store the synaptic connection state and a neuronal state of a neuron associated with the matching synaptic connection;
wherein the logic is configured to calculate a first neuronal state of a first neuron and synaptic connection state of the first synaptic connection in a second clock cycle with respect to the first match by accessing the second memory section in parallel with searching additional bucket elements of the first memory section for an additional match between a first spike message and an additional synaptic connection;
wherein the neuromorphic operation comprises determining a synaptic current associated with the first synaptic connection, determining a change in the first synaptic connection due to Long-Time Depression (LTD) and determining a change for the first synaptic connection due to Long-Time Potentiation (LTP), wherein the synaptic connection state is updated in a single time step and wherein a single time step includes processing all bucket elements by repeating the searching, matching, and neuromorphic operation for each respective bucket element.
8 . The system of claim 7 , wherein the logic is configured to search for the first match by contemporaneously comparing the first source neuron identifier to a set of patterns stored in the first memory section.
9 . The system of claim 7 , wherein the logic is configured to calculate a second neuronal state in response to performing the neuromorphic operation.
10 . The system of claim 9 , wherein the logic is configured to store the second neuronal state in the second memory section.
11 . The system of claim 9 , wherein the logic is configured to calculate the second neuronal state by accessing the second memory section in parallel with performing the search for the first match.
12 . The system of claim 7 , wherein the logic, the input filter, the first memory section, and the second memory section are integrated into a memory device that is configured as a node interfacing with a plurality of other nodes that implement a spiking neural network.
13 . A method comprising:
receiving a spike message, the spike message comprising a first source neuron identifier;
storing the spike message in at least one group in a spike cache memory section, wherein the spike cache memory section is organized by a number of buckets which each correspond to a different time step in which spike messages are received;
storing data indicating a plurality of synaptic connections in a first memory section, each synaptic connection referencing at least one of a source neuron identifier of a source neuron or a target neuron identifier of a target neuron, wherein the first memory section is organized by a number of bucket elements associated with each neuron;
storing a synaptic connection state for each synaptic connection in a second memory section, wherein the second memory section is organized by a number of bucket elements associated with each neuron;
searching a first bucket element of the first memory section in a first clock cycle with respect to a first bucket of the spike cache memory section to identify at least one first match of the first source neuron identifier and a first synaptic connection, wherein the first match is identified by identifying a bit line corresponding to a matching synaptic connection, wherein the searching is performed in response to prioritizing the spike message according to a delay value of the spike message, wherein a sense amplifier array generates a bitmask that indicates the first match, wherein the bitmask signifies a source neuron identifier stored in a group of word lines that matches the first source neuron identifier of the spike message, and wherein each bit of the bitmask indicates whether there is a match of a bit of the first source neuron identifier against a respective bit retrieved from the stored source neuron identifier;
performing a neuromorphic operation in a second clock cycle with respect to the first match using the synaptic connection state of the first synaptic connection stored in a first bucket element of the second memory section in parallel with performing a search in the second clock cycle; and
searching for an additional match between a first spike message and an additional synaptic connection, wherein the second memory section is configured to store the synaptic connection state for the first synaptic connection, and a neuronal state of a first neuron and synaptic connection state of the first synaptic connection are calculated in a second clock cycle with respect to the first match by accessing the second memory section in parallel with searching additional bucket elements of the first memory section for the additional match;
wherein the identified bit line stores at least a portion of a first synaptic connection identifier of the first synaptic connection, and wherein the bit line maps to a corresponding portion of the memory section;
wherein the neuromorphic operation comprises determining a synaptic current associated with the first synaptic connection, determining a change in the first synaptic connection due to Long-Time Depression (LTD), and determining a change for the first synaptic connection due to Long-Time Potentiation (LTP), wherein the synaptic connection state is updated in a single time step and wherein a single time step includes processing all bucket elements by repeating the searching, matching, and neuromorphic operation for each respective bucket element.
14 . The method of claim 13 , wherein the search for the first match is performed by contemporaneously comparing the first source neuron identifier to a set of stored patterns.