1: \documentclass[showpacs,preprintnumbers,amsmath,amssymb,superscriptaddress]{revtex4}
2: %\documentclass[preprint,showpacs,preprintnumbers,amsmath,amssymb,superscriptaddress]{revtex4}
3: %\documentclass[twocolumn,showpacs,preprintnumbers,amsmath,amssymb,superscriptaddress]{revtex4}
4:
5: \usepackage{graphicx}% Include figure files
6: \usepackage{dcolumn}% Align table columns on decimal point
7: \usepackage{bm}% bold math
8: \newcommand{\bra}[1]{\left(#1\right)}
9: \newcommand{\tb}[1]{\textbf{#1}}
10: \newcommand{\avgg}[1]{\ensuremath{\Bigg\langle\!\!\!\Bigg\langle
11: #1
12: \Bigg\rangle\!\!\!\Bigg\rangle
13: }}
14: \newcommand{\distnorm}{\ensuremath{{\cal N}}}
15:
16: \begin{document}
17:
18: \preprint{Physica A \textbf{387}, (2008) 4411 - 4416}
19:
20: \title{Topology and Dynamics of Attractor Neural Networks: The Role of Loopiness}
21: \author{Pan Zhang}
22: %{\email{july.lzu@gmail.com}
23: \affiliation{Institute of Theoretical Physics, Lanzhou University, Lanzhou 730000, China}
24:
25: \author{Yong Chen}
26: \altaffiliation{Author to whom correspondence should be addressed. Email: ychen@lzu.edu.cn}
27: \affiliation{Institute of Theoretical Physics, Lanzhou University, Lanzhou 730000, China}
28:
29: \date{\today}
30:
31: \begin{abstract}
32: We derive an exact representation of the topological effect on the
33: dynamics of sequence processing neural networks within
34: signal-to-noise analysis. A new network structure parameter,
35: loopiness coefficient, is introduced to quantitatively study the
36: loop effect on network dynamics. The large loopiness coefficient
37: means the large probability of finding loops in the networks. We
38: develop the recursive equations for the overlap parameters of neural
39: networks in the term of the loopiness. It was found that the large
40: loopiness increases the correlations among the network states at
41: different times, and eventually it reduces the performance of neural
42: networks. The theory is applied to several network topological
43: structures, including fully-connected, densely-connected random,
44: densely-connected regular, and densely-connected small-world, where
45: encouraging results are obtained.
46:
47: \end{abstract}
48:
49: \pacs{
50: 87.10.+e, %General theory and mathematical aspects
51: 89.75.Fb, %Structures and organization in complex systems
52: 87.18.Sn, %Neural networks
53: 02.50.-r %Probability theory, stochastic processes, and statistics
54: }
55:
56: \maketitle
57:
58: Starting from pioneering works of modelling complex networks \cite{Watts98,Ba99}, research area related with complex networks have been growing very fast. In parallel to the studies of structural properties of complex networks, there has also been a growing interest in dynamic systems defined on networks. For example, synchronization and collective dynamics, epidemic spreading, cascading failures, opinion formation as well as various strategic games, and some physical model like Ising model (see \cite{boccalettia06} and refs. therein).
59:
60: Neural assemblies (i.e. local networks of neurons transiently linked by selective interactions) are considered to be largely distributed and linked to form a web-like structure of the brain. Many researches suggested that neural connectivity is far more complex than the random graph. The cortical neural networks of Caenorhabditis elegans and cat were reported to be small-world (SW) and scale-free (SF), respectively \cite{Watts98,nncomplex1}. It is very important to understand how the complex neural wiring architecture is related to brain functions. With same average connections, Hopfield network with random topology was reported to be more efficient for storage and recognition of patterns than both SW network and regular network \cite{Hopfield1982,Patrick03}. For the SF connection, with the same number of synapses, Torres et al. found that the capacity is larger than the storage of the highly diluted random Hopfield networks \cite{Torres03}. Using a Mente-Carlo method to lower the clustering coefficient smoothly with the degree of each neuron kept unchanged, Kim found that the networks with the lower clustering exhibit much better performance \cite{Kim04}.
61:
62: It is well known that the equilibrium properties of fully-connected Hopfield networks had been extensively studied using spin-glass theory, especially the replica method \cite{replicas,Amit85a,Amit85b}. And the dynamics of fully-connected Hopfield model with static patterns and sequence patterns were widely studied using generating functional analysis \cite{Coolen2000,Coolen98}. As a simple relaxation of biological unrealistical fully-connected model, various random diluted model were studied, include extreme diluted model \cite{Derrida87,Watkin91}, finite diluted model \cite{zhang07,Theumann03}, and finite connection model \cite{WC03}. However, in the case of complex network topology, as far as we know, there are hardly any theoretical studies for dynamics or statics.
63:
64: In this paper, we use signal-to-noise analysis to study the effect of topology on transient dynamics of sequence processing neural networks. Considering the mathematical convenience, here we only focus on the sequence processing models and the relationship between our results and Hopfield models will be discussed.
65:
66: The topological effect on neural networks mainly come from loops of
67: topology \cite{Barkai90}. In the case of so-called extremely diluted
68: structure ($\lim_{N\to \infty}\bar k^{-1}=\lim_{N\to \infty}\bar
69: k/N=0$), average loop length is very big, like $\log_{k-1}N$, so
70: number of short loops (like triangles, quadrangles) in networks will
71: be very small and effect caused by short loops can be neglected.
72: Dynamics of networks in this case is easy to study because each spin
73: at different time steps is uncorrelated \cite{Derrida87}. In
74: contrast, if such small loops exist, the correlations and feedbacks
75: in network will lead to more complicated dynamics.
76:
77: \begin{figure}[h]
78: \includegraphics[width=0.7\textwidth]{fig1.EPS}
79: \caption{Left: the first-order loopiness coefficient, $L_1$, means the probability of connectivity between $k$ and $i$ when $j$ is connected to $i$ and $k$ is connected to $j$. Middle: the second-order loopiness coefficient $L_2$. Right: the third-order loopiness coefficient $L_3$.} \label{fig1}
80: \end{figure}
81:
82: In order to quantitatively present the effect from loops, we define a new parameter, loopiness coefficient, to present the probability of finding loops in the network. The definition is shown in Fig. \ref{fig1}. In the left triangle, spin $j$ is connected to spin $i$, and spin $k$ is connected to $j$. We define the probability that $k$ is connected to $i$ as the first-order loopiness coefficient, $L_1$. Similarly, the $n$th-order loopiness coefficient, $L_n$, denotes the linking probability between two vertices to form a loop with $n+2$ edges.
83:
84: We now study a sequence processing model consisting of $N \rightarrow \infty$ Ising-type neurons $s_i\bra{t} \in \{ +1,-1 \}$. The neurons update their states simultaneously, with the following probabilities,
85: \begin{equation}
86: \mathrm{Prob} [s_i \left(t+1\right) | h_i\left(t\right) ] = \frac{e^{\beta s_i\bra{t+1}h_i\bra{t}}}{2\cosh\bra{\beta h_i\bra{t}}} ,
87: \label{eq01}
88: \end{equation}
89: where the local field $h_i\bra{t} = \sum_{j=1}^N J_{ij} s_j\bra{t}$, and $\beta$ is the inverse temperature. For the transfer function $g \bra{\cdot}$, we denote by $s_i \bra{t+1} = g(h_i\bra{t})$.
90:
91: Let us store $p = \alpha N$ random patterns $\xi^\mu =
92: \bra{\xi_1^\mu,\ldots,\xi_N^\mu}$ in network, where $\alpha$ is the
93: loading ratio. So, the interaction matrix $J_{ij} =
94: \frac{c_{ij}}{Nc} \sum_{\mu=1}^p \xi_i^{\mu+1} \xi_j^\mu$ is chosen
95: to retrieve the patterns as $\xi^1 \rightarrow \xi^2 \rightarrow
96: \cdots \rightarrow \xi^p \rightarrow \xi^1$ (note that $\xi^{p+1} =
97: \xi^1$), where $c_{ij}$ is the adjacency matrix ($c_{ij}=1$ if $j$
98: is connected to $i$, $c_{ij}=0$ otherwise) \cite{Sompolinsky86}.
99: Consequently, the degree of spin $i$ is $k_i = \sum_{j \in T_i}
100: c_{ij} = \sum_{j} c_{ij}$, where $T_i$ is the set of spin $j$
101: connected with $i$. In this work, for studying the role of
102: loopiness, we only consider $k_i \approx \bar{k} = Nc$. We assume
103: that this property holds for dense connected random network, dense
104: connected
105: SW network, and dense connected regular network.
106:
107: For any pattern $\xi ^{\nu}$, the order parameter is $m^{\nu} \bra{t} = \frac{1}{N} \sum \xi _{i}^{\nu} s_{i}\bra{t}$ which represent the overlap between $\mathbf{s} \bra{t}$ and the condensed pattern $\xi ^{\nu}$. The local field in neuron $i$ is described by
108: \begin{equation}
109: h_{i}\bra{t} =\frac{1}{Nc} \sum_{j \in T_i} \xi_{i}^{\nu+1} \xi_j^{\nu} s_j\left(t\right) + Z_{i} \left(t\right),
110: \label{eq02}
111: \end{equation}
112: where the first term of the left-hand side is the signal term and the second one the crosstalk noise from uncondensed patterns,
113: \begin{equation}
114: Z_i\left(t\right) = \frac{1}{Nc} \sum_{\mu \neq \nu} \sum_{j\in T_i} \xi_i^{\mu+1} \xi_j^\mu s_j\left(t\right).
115: \label{eq03}
116: \end{equation}
117: The state of the spins $s_i(t)$ is determined by the sign of local field $h_i(t)$, that is,
118: \begin{eqnarray}
119: s_j \left( t \right) &=& g\left(h_j\bra{t-1}\right) = g \left( \frac{1}{Nc} \sum_{k\in T_j} \xi_j^{\nu} \xi_k^{\nu-1} s_k \bra{t-1} \right. \nonumber\\
120: &+& \left . \frac{1}{Nc} \sum_{k\in T_j} \xi_j^{\mu} \xi_k^{\mu-1}
121: s_k\bra{t-1} + \frac{1}{Nc} \sum_{\chi\neq \nu , \mu} \sum_{k\in
122: T_j} \xi_j^{\chi} \xi_k^{\chi-1} s_k\bra{t-1} \right ). \label{eq04}
123: \end{eqnarray}
124: Expanding $s_j\left(t\right)$ into first order, we obtain
125: \begin{eqnarray}
126: s_j\left(t\right) &=& \hat{s}_j^\mu\left(t\right) + \frac{1}{Nc} \left . \frac{\partial g}{\partial h} \right |_{h_j\left(t-1\right)} \sum_{k\in T_j} \xi_j^\mu\xi_k^{\mu-1} s_k\left(t-1\right) \nonumber\\
127: &+& \mathcal{O}\left(N^{-1}\right),
128: \label{eq05}
129: \end{eqnarray}
130: where
131: \begin{eqnarray}
132: \hat{s}_j^\mu\left(t\right)&=& g \left( \frac{1}{Nc} \sum_{k\in T_j}
133: \xi_j^{\nu} \xi_k^{\nu-1} s_k \bra{t-1}+ \frac{1}{Nc} \sum_{\chi\neq
134: \nu , \mu} \sum_{k\in T_j} \xi_j^{\chi} \xi_k^{\chi-1} s_k\bra{t-1}
135: \right )\nonumber\\ &=& g\left(h_j\bra{t-1} - \frac{1}{Nc}
136: \sum_{k\in T_j} \xi_j^{\mu} \xi_k^{\mu-1}
137: s_k\left(t-1\right)\right).
138: \end{eqnarray}
139: Note that, in here, $\hat{s}_j^\mu\left(t\right)$ is independent of
140: $\xi_j^\mu$. Then the crosstalk noise can be presented by
141: \begin{eqnarray}
142: &&Z_{i}\left( t\right) = \sum_{\mu \neq \nu} \frac{1}{Nc} \sum_{j\in T_{i}} \xi _{i}^{\mu +1} \xi _{j}^{\mu } \hat{s}_{j}^{\mu }\left( t\right) \nonumber \\
143: &+& U\left( t-1\right) \frac{1}{Nc} \sum_{j\in T_{i}} \frac{1}{Nc} \sum_{\mu \neq \nu} \sum_{k\in T_{j}} \xi_{i}^{\mu +1} \xi_{k}^{\mu -1} s_{k}\left(t-1\right),
144: \label{eq06}
145: \end{eqnarray}
146: where $U(t-1) = \langle\!\langle \langle \frac{\partial g} {\partial h} \vert _{h(t-1)} \rangle_{\boldsymbol{h}} \rangle \!\rangle$. $\langle .\rangle _{{\boldsymbol{h}}}$ denotes the average over the distribution of the local field, and $\left\langle \!\left\langle \cdot \right\rangle \!\right\rangle $ means the average over the initial conditions and the condensed pattern.
147:
148: Using the central limit theorem, the first term of Eq. (\ref{eq06}) converge to a zero mean Gaussian form $\mathcal{N} (0,\alpha / c )$. Since $\xi_{i}^{\mu +1} \xi _{k}^{\mu-1} s_{k}\left( t-1\right)$ is not independent, we cannot apply this theorem directly into the second term of Eq. (\ref{eq05}), we have to expand $Z_i\bra{t}$ into time $t-2$,
149: \begin{widetext}
150: \begin{eqnarray}
151: Z_{i}\left( t\right) &=&\mathcal{N}(0,\alpha / c ) + U\left( t-1\right) \frac{1}{Nc} \sum_{j\in T_{i}} \frac{1}{Nc} \sum_{\mu \neq \nu} \sum_{k\in T_{j}} \xi _{i}^{\mu +1} \xi _{k}^{\mu -1} \hat{s}^{\mu -1}_{k}\left( t-1\right) \nonumber\\
152: &&+ U\left( t-1\right) U\bra{t-2} \frac{1}{Nc} \sum_{j\in T_{i}} \frac{1}{Nc} \sum_{k\in T_{j}} \frac{1}{Nc} \sum_{\mu \neq \nu} \sum_{l\in T_{k}} \xi _{i}^{\mu +1} \xi _{l}^{\mu -2} s_{l} \left( t-2\right),
153: \label{eq07}
154: \end{eqnarray}
155: \end{widetext}
156: with $U(t-2) = \langle\!\langle \langle \frac{\partial g} {\partial h}\vert _{h(t-2)}\rangle_{\boldsymbol{h}} \rangle\! \rangle$. Even though averaging over $j\in T_{i}$, not all $k\in T_{j}$ are residual spins. Note that only $k$ belonging to both $T_{i}$ and $T_{j}$ will survive the averaging operations. The probability of k is described by the first order loopiness coefficient $L_1$ (See Fig. \ref {fig1}). Similarly, from the central limit theorem, it was found that the second term of Eq. (\ref{eq06}) converged to $\mathcal{N}(0,L_{1}U^{2}\left( t-1\right) \alpha / c )$. Then we expand $Z_i\bra{t}$ to time $t-3$,
157: \begin{widetext}
158: \begin{eqnarray}
159: Z_{i}\left( t\right) &=& \mathcal{N} (0,\alpha / c ) + \mathcal{N}(0,L_1 U^2\bra{t-1} \alpha / c ) \nonumber\\
160: && + U\left( t-1\right) U\bra{t-2} \frac{1}{Nc} \sum_{j\in T_{i}} \frac{1}{Nc} \sum_{k\in T_{j}} \frac{1}{Nc} \sum_{\mu \neq \nu} \sum_{l\in T_{k}} \xi_{i}^{\mu +1} \xi_{l}^{\mu -2} \hat{s}^{\mu-2}_{l}\left( t-2\right) \nonumber\\
161: && + U\left( t-1\right) U\bra{t-2} U\bra{t-3} \frac{1}{Nc} \sum_{j\in T_{i}} \frac{1}{Nc} \sum_{k\in T_{j}} \frac{1}{Nc} \sum_{l\in T_{k}} \frac{1}{Nc} \sum_{n\in T_l} \sum_{\mu \neq \nu} \xi _{i}^{\mu+1} \xi _{n}^{\mu -3} s_{n}\left( t-3\right).
162: \label{eq08}
163: \end{eqnarray}
164: \end{widetext}
165: Here, it is easy to show that the third term of Eq. (\ref{eq07}) converge to $\mathcal{N}(0,L_{2}U^{2}\left( t-1\right)U^2\bra{t-2} \alpha / c )$.
166:
167: Repeating the above expansion up to time $0$, we collect all the correlations caused by the loops and derive the final expression of crosstalk noise at time $t$,
168: \begin{equation}
169: Z_i\bra{t} = \mathcal{N}\bra{0,\alpha /c} + \sum_{t_i=0}^{t-1} \mathcal{N}\bra{0,\bra{\prod_{t_i'=t_i}^{t-1}U^2(t_i')} L_{t-t_i} \frac{\alpha}{c}}.
170: \label{eq09}
171: \end{equation}
172: Because each sum in the above equation is independent, one can calculate the variances directly (for proof of independency of each sum, see \cite{Kawamura02,Kitano98}),
173: \begin{equation}
174: \sigma^2\bra{t} = \frac{\alpha}{c} + \sum_{t_i=0}^{t-1} \bra{\prod_{t_i'=t_i}^{t-1}U^2(t_i')}L_{t-t_i} \frac{\alpha}{c}.
175: \label{eq10}
176: \end{equation}
177:
178: And we write down explicitly the following closed equations of the order parameter $m(t)$,
179: \begin{equation}\label{eq:m}
180: m\bra{t+1} = \int Dz\left\langle \xi g\bra{\xi m \bra{t} + \sigma \bra{t} z}\right\rangle_\xi,
181: \end{equation}
182: \begin{equation}\label{eq:U}
183: U\bra{t+1} = \frac{1}{\sigma\bra{t}} \int Dzz \left\langle g \bra{\xi m\bra{t} + \sigma\bra{t} z} \right\rangle_\xi,
184: \end{equation}
185: where $\langle\cdot\rangle_\xi$ stands for the average over the retrieval patterns $\xi$, and $Dz = \frac{1}{\sqrt{2\pi}} \exp\bra{-z^2/2}dz$.
186:
187: We are now ready to present the signal-to-noise treatment of the sequence processing neural networks with narrow-degree-distributed topology at zero temperature. The above equations form a recursive scheme to calculate the dynamical properties of the systems with an arbitrary time step. However, the following practical trouble is how to calculate the loopiness coefficients up to arbitrary order for a given topology.
188:
189: In the special case of fully-connected networks, $c=1$, all the loopiness coefficient are $L_n = 1$. And substituting into Eq. (\ref{eq10}), we find $\sigma^2$ under the form,
190: \begin{equation}
191: \sigma^2\bra{t} = \alpha + U^2(t-1) \sigma^2\bra{t-1}.
192: \label{eq:fully:sigma}
193: \end{equation}
194: This equation, together with Eqs. (\ref{eq:m}-\ref{eq:U}), is coincident with the dynamical equations in \cite{Kawamura02,Amari88}. The stationary equations and simulation results can be found in \cite{Kawamura02}.
195:
196: \begin{figure}
197: \includegraphics[width=0.8\textwidth]{fig2.EPS}
198: \caption{(Color online) (a) Temporal evolution of the overlap
199: parameters of network with random topology, the initial overlap
200: ranges from $1.0$ to $0.1$ (top to bottom). (b) basin of attraction
201: of network with random topology. the lower part means the lowest
202: initial overlap to retrieve successfully. The upper part is the final overlap for successful retrieval. Temperature is zero. The parameters are (a) $N=5000$, $c=0.16$, and $\alpha N=190$; (b) $N=3000$ and $c=0.1$.}
203: \label{fig2}
204: \end{figure}
205:
206: \begin{figure}
207: \includegraphics[width=0.8\textwidth]{fig3.EPS}
208: \caption{(Color online) Temporal evolution of the overlap parameters with (a) regular topology and (b) SW topology. the initial overlap ranges from $1.0$ to $0.1$ (up to down). Temperature is zero. The parameters are $N=5000$, $\bar{k}=800$, and $\alpha N=190$. Rewiring probability of SW network is $0.5$. In our paper, both regular topology and SW topology is in one-dimensional.}
209: \label{fig3}
210: \end{figure}
211:
212: \begin{figure}
213: \includegraphics[width=0.8\textwidth]{fig4.EPS}
214: \caption{(Color online) (a) The first-, second-, and third- order
215: loopiness coefficients and (b) the theoretical results of $4$ time step dynamics for regular networks, SW networks($p$ is rewiring probability), and random networks. The initial overlaps are $0.2$. The parameters of networks are $N=5000$, $\bar{k}=800$, and $\alpha N=190$. }
216: \label{fig4}
217: \end{figure}
218:
219: For random networks, each node has a probability $c$ to connect with others. So the loopiness coefficients are also $L_n = c$. The temporal evolutions of overlap up to $20$ time steps obtained by both theory and numerical simulations are plotted in Fig. \ref{fig2}. The initial overlap ranges from $0.1$ to $1.0$. Fig. \ref{fig2} shows the basin of attraction from theory and simulations.
220:
221: However, for regular networks or SW networks, it is impossible to get the exact analytical expressions of the loopiness coefficients. so the only solution is programming. Due to algorithm complexity in large size networks, we only present the first, second, and third order loopiness coefficients. As a result, one only derives the $4$ time step dynamics from Eqs. (\ref{eq10}-\ref{eq:U}). In Fig. \ref{fig3}, we show the theoretical results and simulations of $4$ time step dynamics for regular topology and SW topology with rewiring probability $p=0.5$.
222:
223: In practice, it was found that the loopiness coefficients decrease with increasing the rewiring probability of SW networks, and the overlaps are increaseed with decreasing the loopiness coefficients (see Fig. \ref{fig4}). From Eq. (\ref{eq10}), the larger loopiness coefficients induce larger variances of the crosstalk noise, and eventually result in a negative effect on performances of neural networks.
224:
225: Comparing to the sequence processing model, besides of a noise part and a Gaussian part, the local field of Hopfield model has a discrete self-interaction part, and crosstalk noise is no longer normally distributed, which produce studied intractable. But similarly to sequence processing model, the large loopiness coefficient also increases correlations of spin states at different time and leads to large variance of the Gaussian part of crosstalk noise. So it is easy to qualitatively understand why small amount of randomly rewired connections greatly improve the performance of Hopfield model with regular architecture \cite{Patrick03,Torres03,sw1,Morelli04}, and why Hopfield model with random networks is better than SW networks \cite{Patrick03}.
226:
227: Note that the value of the first-order loopiness coefficient in undirected graph is the number of triples divided by the number of connected triples (See Fig. \ref {fig1}), which is the same with well-known network parameter \textit{clustering coefficient} \cite{Watts98}. Considering Eq. (\ref{eq10}), it shows that $L_1$ plays a much more important role than the higher order loopiness coefficients, and the lower $L_1$ or clustering coefficient make better performance for neural networks. This is indeed confirmed by Kim with Monte-Carlo method to tune the clustering coefficient while retaining the same degree distributions \cite{Kim04}.
228:
229: We also have to point out that all topologies studied in both theory and simulations are densely-connected networks, our results can not be extended to finite-connectivity networks directly, the reason is: in Eq.(7), using central limit theorem, we assume that the term $\sum_{\mu \neq \nu} \frac{1}{Nc} \sum_{j\in T_{i}} \xi _{i}^{\mu +1} \xi _{j}^{\mu } \hat{s}_{j}^{\mu }\left( t\right)$ converges to a zero mean Gaussian form $\mathcal{N} (0,\alpha / c )$. This assumption is correct in densely-connected networks where $\bar{k}\to\infty$, but incorrect in finite-connectivity networks. However, we find that if $\bar k$ is big($\bar k>=50$), above assumption is a good approximation and our theory gives a good approximation to finite-connectivity network dynamics.
230:
231:
232: At last, it should be very careful that this theory is not suitable for scale-free networks since the power-law degree distribution is out of action of signal-to-noise analysis and its loopiness coefficients are degree-correlated, which cannot be expressed simply as $L_n$. Attempts of studying dynamics of sequence processing neural networks with power-law and even arbitrary degree distributions have been done by neglecting the loopiness effect \cite{chen07}. Some numerical studies of properties of neural networks with scale-free topology can be found in \cite{stauffer03}.
233:
234: \section*{Acknowledgments}
235:
236: This work was supported by the National Natural Science Foundation of China under Grant No. $10305005$ and by the Special Fund for Doctor Programs at Lanzhou University.
237:
238: \begin{thebibliography}{99}
239:
240: %Collective dynamics of 'small-world' networks.
241: \bibitem{Watts98} D. J. Watts and S. H. Strogatz, Nature(London) \textbf{393}, 440(1998).
242:
243: %Emergence ofscaling in random networks" "Mean-field theory for scale-free random networks
244: \bibitem{Ba99} A. L. Barab\'{a}si and R. Albert, Science \textbf{286}, 509(1999); A. L. Barab\'{a}si, R. Albert, and H. Jeong, Physica A \textbf{272}, 173(1999).
245:
246: %Complex networks: Structure and dynamics
247: \bibitem{boccalettia06} S. Boccalettia \textit{et al.}, Phys. Rep. \textbf{424}, 175(2006).
248:
249: \bibitem{nncomplex1} K. E. Stephan, L. Kamper, A. Bozkurt, G. A. P. C. Burns, M. P. Young, and R. Koter, Phil. Trans. R. Soc. Lond. B Biol. Sci. \textbf{356}1159(2001); C. Cherniak, J. Neurosci. \textbf{14}, 2418(1994); J. W. Scannell, G. A. P. C. Burns, C. C. Hilgetag, M. A. OˇŻNeill, and M. P. Young, Cerebral Cortex \textbf{9}, 277(1999).
250: %"Advanced database methodology for the collation of connectivity data on the Macaque brain (CoCoMac). "
251: %"Component placement optimization in the brain"
252: %"The Connectional Organization of the Cortico-thalamic System of the Cat"
253:
254: %Neural Networks and Physical Systems with Emergent Collective Computational Abilities
255: \bibitem{Hopfield1982} J. J. Hopfield, Proc. Nat. Acad. Sci. USA \textbf{79}, 2554(1982).
256:
257: %Topology and computational performance of attractor neural networks
258: \bibitem{Patrick03} P. N. McGraw and M. Menzinger, Phys. Rev. E \textbf{68}, 047102(2003).
259:
260: %Influence of topology on the performance of a neural network
261: \bibitem{Torres03} J. J. Torres, M. A. Munoz, J. Marro, and P. L. Garrido, Neurocomputing \textbf{58}, 229(2004).
262:
263: %Performance of networks of artificial neurons: The role of clustering
264: \bibitem{Kim04} B. J. Kim, Phys. Rev. E \textbf{69}, 045101(2004).
265:
266: \bibitem{replicas} M. Mezard, G. Parisi, and M. A. Virasoro, \textit{Spin Glass Theory and Beyond} (World Scientific, Singapore, 1987).
267:
268: %Spin-glass models of neural networks
269: \bibitem{Amit85a} D. J. Amit, H. Gutfreund, and H. Sompolinsky, Phys. Rev. A \textbf{32}, 1007(1985).
270:
271: %Storing Infinite Numbers of Patterns in a Spin-Glass Model of Neural Networks
272: \bibitem{Amit85b} D. J. Amit, H. Gutfreund, and H. Sompolinsky, Phys. Rev. Lett. \textbf{55}, 1530(1985).
273:
274: %Statistical Mechanics of Recurrent Neural Networks II. Dynamics
275: \bibitem{Coolen2000} A. C. C. Coolen, arXiv:cond-mat/0006011.
276:
277: %Phase diagram and storage capacity of sequence processing neural networks
278: \bibitem{Coolen98} A. D\"{u}ring, A. C. C. Coolen, and D. Sherrington, J. Phys. A \textbf{31}, 8607(1998).
279:
280: %An exactly solvable asymmetric neural network model
281: \bibitem{Derrida87} B. Derrida, E. Gardner, and A. Zippelius, Europhys. Lett. \textbf{4}, 167(1987).
282:
283: %The parallel dynamics of a dilute symmetric Hebb-rule network
284: \bibitem{Watkin91} T. L. H. Watkin and D. Sherrington, J. Phys. A \textbf{24}, 5427(1991).
285:
286: %Statistical Neurodynamics for Sequence Processing Neural Networks with Finite Dilution
287: \bibitem{zhang07} P. Zhang and Y. Chen, Lect. Notes Comput. Sci. \textbf{4491}, 1144(2007). (cond-mat/0611313)
288:
289: \bibitem{Theumann03} W. K. Theumann, Physica A \textbf{328}, 1(2003).
290: %"Mean-field dynamics of sequence processing neural networks with finite connectivity
291:
292: \bibitem{WC03}B. Wemmenhove1 and A. C. C. Coolen, J. Phys. A \textbf{36}, 9617(2003).
293: %"Finite connectivity attractor neural networks"
294:
295: %Properties of sparsely connected excitatory neural networks
296: \bibitem{Barkai90} E. Barkai, I. Kanter, and H. Sompolinsky, Phys. Rev. A \textbf{41},590(1990).
297:
298: %Temporal Association in Asymmetric Neural Networks
299: \bibitem{Sompolinsky86} H. Sompolinsky and I. Kanter, Phys. Rev. Lett. \textbf{57}, 2861(1986).
300:
301: %Transient dynamics for sequence processing neural networks
302: \bibitem{Kawamura02} M. Kawamura and M. Okada, J. Phys. A \textbf{35}, 253(2002).
303:
304: %Retrieval dynamics of neural networks for sparsely coded sequential patterns
305: \bibitem{Kitano98} K. Kitano and T. Aoyagi, J. Phys. A \textbf{31}, L613(1998).
306:
307: %Statistical Neurodynamics of Various Versions of Correlation Associative Memory
308: \bibitem{Amari88} S. Amari, Proc. IEEE Conference on Neural Networks \textbf{1}, 633(1988).
309:
310: %parallel dynamics of fully connected q-ising neural networks
311: \bibitem{Bolle98} D. Boll\'{e}, G. Jongen, and G. M. Shim, J. Stat. Phys. \textbf{91}, 125(1998).
312:
313: %Systematic rewiring in associative neural networks with small-world architecture
314: % Neural Networks, 2005. IJCNN '05. Proceedings. 2005 IEEE International Joint Conference on
315: \bibitem{sw1} O. K. Dekhtyarenko, Proc. IJCNN'05 \textbf{2}, 1178(2005).
316:
317: %Associative memory on a small-world neural network
318: \bibitem{Morelli04} L. G. Morelli, G. Abramson, and M. N. Kuperman, Eur. Phys. J. B \textbf{38},495(2004).
319:
320: %Transient Dynamics of Sparsely Connected Hopfield Neural Networks with Arbitrary Degree Distributions
321: %Transient dynamics of sequence processing neural networks: effect of degree distributions (PDF, 478k)
322: \bibitem{chen07} P. Zhang and Y. Chen, Physica A \textbf{387}, 1009(2008); Y. Chen, P. Zhang, L.C. Yu, and S.L. Zhang, arXiv:0705.3679v1, (2007).
323:
324: %"Efficient Hopfield pattern recognition on a scale-free neural network"
325: \bibitem{stauffer03} D. Stauffer, A. Aharony, L. da F. Costa, and J. Adler, Euro. Phys. J. B \textbf{32}, 395(2003).
326:
327: \end{thebibliography}
328:
329: \end{document}
330: