Implementing a neural associative memory based on non-linear learning of discrete synapses

1. A neural associative memory system for maintaining associations between memory address patterns and memory content patterns, the memory system comprising:

a neural network comprising a set of synapses and a set of neurons, the synapses connecting neurons and storing the associations,

accepting means for accepting an input query pattern,

processing means for

computing a matrix of synaptic potentials as a function of the memory address patterns and the memory content patterns,

transforming the matrix of synaptic potentials into a matrix of synaptic weights, wherein each weight is chosen from a finite set of synaptic strengths,

optimizing the neural network with respect to a target architecture chosen for implementation, and

computing for each neuron an optimal output function for minimizing output noise, and

output means for returning an output pattern to the input query pattern equal or similar to the memory content pattern associated with the memory address pattern equal or similar to the input query pattern.

2. The neural associative memory system of claim 1 , wherein

said accepting means for accepting an input query pattern is at least one of a sensor, a hardware interface and a software interface.

3. The neural associative memory system of claim 1 , wherein

the processing means is a processor and/or a signal processing unit formed of hardware and/or software.

4. The neural associative memory system of claim 1 , wherein

the processing means is adapted for parallel processing.

5. The neural associative memory system of claim 1 , wherein

the output means is at least one of a hardware interface, or a software interface.

6. The neural associative memory system of claim 1 , wherein

the neural network is optimized by application of compression mechanisms.

7. The neural associative memory system of claim 1 , wherein

the neural network is optimized by skipping synapses with a predefined or zero weight in the optimization.

8. The neural associative memory system of claim 1 , wherein

the output noise is defined the distance between the original content memory pattern and the output generated by the memory system, and wherein

parameters are determined through the matrix of synaptic weights and a noise distribution describing how a query input pattern used for retrieval deviates from an original memory address pattern.

9. The neural associative memory system of claim 1 , wherein

the synaptic potentials are computed as a linear sum of learning increments.

10. The neural associative memory system of claim 1 , wherein

the synaptic potentials are computed corresponding to the weights of a Bayesian associative memory model.

11. The neural associative memory system of claim 1 , wherein

the synaptic potentials are transformed into synaptic weights by applying synaptic thresholds yielding discrete synaptic strengths, wherein

the synaptic thresholds are computed to minimize the expected number of component errors in the retrieval output.

12. The neural associative memory system of claim 11 , wherein

the synaptic thresholds are chosen such that each content neuron has the same set of matrix loads defined as an expected fraction of synapses with specified weights.

13. The neural associative memory system of claim 11 , wherein

the synaptic thresholds are adapted homeostatically such that, at any time, each neuron has the same number of synapses with a predetermined weight.

14. The neural associative memory system of claim 11 , wherein

discretization parameters are chosen to maximize the signal-to-noise ratio and to maximize the number of storable patterns and/or stored information per synapse.

15. The neural associative memory system of claim 14 , wherein

the discretization parameters are obtained with respect to the synaptic strength and memory load.

16. The neural associative memory system of claim 14 , wherein

the discretization parameters are chosen to maximize the information capacity of the neural network, wherein

the capacity is defined as the stored information per computer bit.

17. The neural associative memory system of claim 14 , wherein

the discretization parameters are obtained with respect to the synaptic strength, the memory load, and information content of a discrete synaptic weight.

18. The neural associative memory system of claim 14 , wherein

the discretization parameters are chosen to maximize the synaptic capacity, wherein

the synaptic capacity is defined as the stored information per non-silent synapse.

19. The neural associative memory system of claim 14 , wherein

the discretization parameters are obtained with respect to the synaptic strength, the memory load and a total matrix load, wherein

the total matrix load is defined as the fraction of non-silent synapses.

20. The neural associative memory system of claim 14 , wherein

the neural network is structurally static.

21. The neural associative memory system of claim 14 , wherein

the neural network is a structurally plastic network.

22. The neural associative memory system of claim 14 , wherein

for a set of synapses a discrete synaptic weight matrix is implemented by a set of binary matrices.

23. The neural associative memory system of claim 22 , wherein

each entry of the binary matrices is defined as a binary weight, which is active if the discrete weight is a linear combination of predetermined factors, wherein the factor corresponding to the binary matrix considered is active.

24. The neural associative memory system of claim 14 , wherein

synapses have discrete weights, and wherein

each discrete weight is calculated as a binary linear combination of values described by a binary bit vector.

25. A method for retrieving a memory content pattern from a neural associative memory system, the method comprising the steps of:

accepting an input query pattern by an accepting means,

using a neural network comprising a set of synapses and a set of neurons, the synapses connecting neurons and storing the associations between memory address patterns and memory content patterns for retrieval of an output pattern,

computing dendritic potentials from the input query pattern with a synaptic weight matrix by a processing means,

computing an output pattern by a non-linear function of the dendritic potentials, parametrized by firing thresholds by the processing means, and

returning the output pattern to the input query pattern equal or similar to the memory content pattern associated with the memory address pattern equal or similar to the input query pattern through output means.

26. The method of claim 25 , wherein

the accepting means for accepting an input query pattern is at least one of a sensor, a hardware interface and a software interface.

27. The method of claim 25 , wherein

the processing means is a processor and/or a signal processing unit formed of hardware and/or software.

28. The method of claim 25 , wherein

the processing means is adapted for parallel processing.

29. The method of claim 25 , wherein

the output means is at least one of a hardware interface, or a software interface.

30. The method of claim 25 , wherein

the processing means is used for

computing a matrix of synaptic potentials as a function of the memory address patterns and the memory content patterns,

transforming the matrix of synaptic potentials into a matrix of discrete synaptic weights, wherein each weight is chosen from a finite set of synaptic strength,

optimizing the neural network with respect to a target architecture chosen for implementation, and

computing for each neuron an optimal output function for minimizing the output noise.

31. The method of claim 25 , wherein

the dendritic potentials are computed by a matrix multiplication of the query input pattern with the synaptic weight matrix, and wherein

the output pattern is computed by applying firing thresholds to the dendritic potentials, where the firing thresholds are optimized for minimizing the expected number of component errors in the output pattern if a binary input query pattern is provided.

32. The method of claim 31 , wherein

the matrix multiplications are performed bit-parallel.

33. The method of claim 31 , wherein

the matrix multiplications are parallelized, such that in each computation step several synapses are processed in parallel.

34. The method of claim 25 , wherein

the dendritic potentials are computed by a matrix multiplication of the input query pattern with the synaptic weight matrix, wherein

a vector of firing thresholds is subtracted, if parameters are such that not all resulting firing thresholds are zero or near zero, wherein

the dendritic potentials are normalized, and wherein

the neurons with the largest dendritic potentials are activated.

35. The method of claim 25 , wherein

the dendritic potentials are computed for discrete synapses from the synaptic strength, matrices and the input query pattern.

36. A neural network system used in a neural associative memory system of maintaining associations between memory address patterns and memory content patterns, the memory system comprising:

a neural network comprising a set of synapses and a set of neurons, the synapses connecting neurons and storing the associations,

accepting means for accepting an input query pattern,

processing means for

computing a matrix of synaptic potentials as a function of the memory address patterns and the memory content patterns,

transforming the matrix of synaptic potentials into a matrix of synaptic weights, wherein each weight is chosen from a finite set of synaptic strengths,

optimizing the neural network with respect to a target architecture chosen for implementation,

computing for each neuron an optimal output function for minimizing output noise,

output means for returning an output pattern to the input query pattern equal or similar to the memory content pattern associated with the memory address pattern equal or similar to the input query pattern, and

further comprising in the method of claim 25 , wherein

the neural network comprising a set of synapses and a set of neurons, the synapses connecting neurons and storing the associations between memory address patterns and memory content patterns is implemented on a parallel computational architecture with at least two processors, the processors being connected by a common bus.

37. The neural network of claim 36 , wherein

each processor stores a column of a discrete synaptic weight matrix in its local memory, wherein

one processor per neuron is provided, and wherein during retrieval, active components of the input query patterns accepted by an accepting means are presented one after the other on the common bus to neurons implemented by the processors, and wherein

the processors add up corresponding synaptic weights in parallel, such that retrieval requires only as many tune steps as the number of neurons active for the input query pattern.

38. The neural network of claim 36 , wherein

the architecture is a matrix like architecture, and wherein

the neurons instantaneously add synaptic inputs such that a retrieval takes only a single time step.

39. The neural network of claim 36 , wherein

the accepting means for accepting an input query pattern is at least one of a sensor, a hardware interface and a software interface.

40. A neural associative memory method for maintaining associations between memory address patterns and memory content patterns, the method comprising:

connecting neurons and storing the associations by synapses in a neural network comprising a set of synapses and a set of neurons,

accepting an input query pattern,

computing a matrix of synaptic potentials as a function of the memory address patterns and the memory content patterns,

transforming the matrix of synaptic potentials into a matrix of synaptic weights, wherein each weight is chosen from a finite set of synaptic strengths,

optimizing the neural network with respect to a target architecture chosen for implementation, and

computing for each neuron an optimal output function for minimizing output noise, and

returning an output pattern to the input query pattern equal or similar to the memory content pattern associated with the memory address pattern equal or similar to the input query pattern.