Information capacity of binary weights associative memories

We study the amount of information stored in the fixed points of random instances of two binary weights associative memory models: the Willshaw model (WM) and the inverted neural network (INN). For these models, we show divergences between the information capacity (IC) as defined by Abu-Mostafa and Jacques, and information calculated from the standard notion of storaqe capacity by Palm and Grossman, respectively. We prove that the WM has asymptotically optimal IC for nearly the full range of threshold values, the INN likewise for constant threshold values, and both over all degrees of sparseness of the stored vectors. This is contrasted with the result by Palm, which required stored random vectors to be logarithmically sparse to achieve good storage capacity for the WM, and with that of Grossman, which showed that the INN has poor storage capacity for random vectors. We propose Q- state versions of the WM and the INN, and show that they retain asymptotically optimal IC while guaranteeing stable storage. By contrast, we show that the Q-state INN has poor storage capacity for random vectors. Our results indicate that it might be useful to ask analogous questions for other associative memory models. Our techniques are not limited to working with binary weights memories.