1:
2:
3: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
4: %%
5: %% Increase of Synapse Efficiency
6: %% due to Pruning following Over-Growth
7: %%
8: %% Kazushi Mimura, Tomoyuki Kimoto, Masato Okada
9: %%
10:
11:
12:
13: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
14: \documentclass[twocolumn,showpacs,preprintnumbers,amsmath,amssymb]{revtex4}
15: %\documentclass[preprint,showpacs,preprintnumbers,amsmath,amssymb]{revtex4}
16: %%%%%%% Some other (several out of many) possibilities
17: %%%%%%%\documentclass[preprint,aps]{revtex4}
18: %%%%%%%\documentclass[preprint,aps,draft]{revtex4}
19: %%%%%%%\documentclass[prb]{revtex4}% Physical Review B
20: \usepackage{graphicx}
21: \usepackage{dcolumn}
22: \usepackage{bm}
23: \newcommand{\mbx}[1]{\mbox{\boldmath $#1$}}
24: %\nofiles
25:
26: %%double space style
27: %\documentclass[12pt]{article}
28: %\usepackage{graphics}
29: %\setlength{\oddsidemargin}{0.2cm}
30: %\setlength{\textwidth}{15.6cm}
31: %\setlength{\topmargin}{-0.5cm}
32: %\setlength{\textheight}{24.0 cm}
33: %\renewcommand{\baselinestretch}{1.8}
34: %\pagestyle{empty}
35:
36:
37:
38: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
39: \renewcommand{\vec}[1]{\mbox{\boldmath $#1$}}
40: \newcommand{\sgn}{\mbox{sgn}}
41: \newcommand{\erf}{\mbox{erf}}
42: \newcommand{\tabtopsp}[1]{\vbox{\vbox to#1{}\vbox to1zw{}}}
43: \newcommand{\epsfig}[2]{
44: \begin{center}
45: \scalebox{0.37}{\includegraphics{#1}} % <original size = x0.27>
46: \end{center}
47: }
48:
49:
50:
51: \begin{document}
52: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
53: \preprint{APS/123-QED}
54: \title{Synapse efficiency diverges due to synaptic pruning following
55: over-growth}
56: %%%%%%%%%%%%%%%%%%%%%%%
57: \author{Kazushi Mimura${}^{\dagger}$}
58: \email{mimura@kobe-kosen.ac.jp}
59: %%%%%%%%%%%%%%%%%%%%%%%
60: \author{Tomoyuki Kimoto${}^{\ddagger}$}
61: %%%%%%%%%%%%%%%%%%%%%%%
62: \author{Masato Okada${}^{\S}$}
63: \affiliation{
64: ${}^{\dagger}$
65: Department of Electrical Engineering,
66: Kobe City College of Technology,
67: Gakuenhigashi-machi 8-3, Nishi-ku, Kobe, Hyogo, 651-2194 Japan\\
68: ${}^{\ddagger}$
69: Oita National College of Technology,
70: Maki 1666, Oita-shi, Oita, 870-0152 Japan\\
71: ${}^{\S}$
72: Laboratory for Mathematical Neuroscience,
73: RIKEN Brain Science Institute, Saitama 351-0198, Japan, \\
74: "Intelligent Cooperation and Control", PRESTO, JST,
75: c/o RIKEN BSI, Saitama 351-0198, Japan, \\
76: ERATO Kawato Dynamic Brain Project,
77: 2-2 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-0288, Japan
78: }
79: %%%%%%%%%%%%%%%%%%%%%%%
80: \date{\today}% It is always \today, today,
81: % but any date may be explicitly specified
82: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
83: \begin{abstract}
84: \label{sec.sparse_noise.abst}
85: In the development of the brain,
86: it is known that synapses are pruned following
87: over-growth.
88: This pruning following over-growth seems to be a universal phenomenon
89: that occurs in almost all areas -- visual cortex,
90: motor area, association area, and so on.
91: It has been shown numerically
92: that the synapse efficiency is increased by systematic deletion.
93: We discuss the synapse efficiency
94: to evaluate the effect of pruning following over-growth, and
95: analytically show that
96: the synapse efficiency diverges as $O(|\log c|)$
97: at the limit where connecting rate $c$ is extremely small.
98: Under a fixed synapse number criterion,
99: the optimal connecting rate, which maximize memory performance, exists.
100: \end{abstract}
101: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
102: \pacs{87.10.+e, 89.70,+c, 05.90.+m}% PACS, the Physics and Astronomy
103: % Classification Scheme.
104: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
105: \keywords{
106: Synapse Over-Growth, Synapse Pruning, Synapse Efficiency, Memory Performance, Associative Memory, SCSNA}
107: \maketitle
108: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
109:
110:
111:
112:
113:
114:
115:
116:
117:
118:
119:
120:
121:
122:
123:
124:
125:
126:
127: % <AUTHORS>
128: % Kazushi Mimura, Masato Okada
129: % Dept. of Electrical Engineering, Kobe City College of Technology,
130: % Gakuenhigashi-machi 8-3, Nishi-ku, Kobe, 651-2194 Japan
131: % Institute of Physical and Chemical Research
132: % Hirosawa 2-1, Wakou-shi, Saitama, 351-0456 Japan
133: % E-mail: mimura@kobe-kosen.ac.jp, okada@brain.riken.go.jp
134:
135:
136:
137:
138:
139:
140:
141:
142:
143:
144:
145: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
146: %% INTRODUCTION
147: \section{Introduction}
148: \label{sec.sparse_noise.intro}
149:
150:
151: %It is known the phenomenon that synapses are pruned following over-growth
152: %in the development of the brain.
153: %脳の発達に関して顕著な
154: %シナプスの過剰な生成とそれに続く刈り込みが起こることが知られている．
155: In this paper, we analytically discuss synapse efficiency
156: to evaluate effects of pruning following over-growth
157: during brain development,
158: within the framework of auto-correlation-type associative memory.
159: %本稿では，この刈り込みについて自己相関型連想記憶モデルの枠組の中で議論する．
160:
161: Because this pruning following over-growth seems to be a universal phenomenon
162: that occurs in almost all areas
163: -- visual cortex, motor area, association area, and so on
164: \cite{
165: Huttenlocker1979,Huttenlocker1982,%Huttenlocker1987,
166: Bourgeois1993,Takacs1986,Innocenti1995,Eckehoff1991,
167: Rakic1994,Stryker1986,Sur1990,Wolff1995}
168: --
169: we discuss the meaning of its function from a universal viewpoint
170: rather than in terms of particular properties in each area.
171: %過剰生成と刈り込みは視覚野，運動野，連合野などの
172: %脳のほとんど全ての領野で見られる普遍的な現象であることから，
173: %その機能的な意味は領野個別の利点というよりは
174: %脳の神経回路モデルに普遍的な観点から議論すべきものであろう．
175: Of course, to discuss this phenomenon
176: as a universal property of a neural network model,
177: we need to choose an appropriate model.
178: %この現象を神経回路モデルの普遍的な性質から議論するためには，
179: %どのようなタイプのモデルを議論すればよいであろうか．
180:
181: Artificial neural network models are roughly classified into two types:
182: feed forward models and recurrent models.
183: %人工神経回路モデルには大きくわけて，
184: %フィードフォワード型とリカレント型の神経回路モデルに分類される．
185: Various learning rules are applied to the architectures of these models,
186: and correlation learning corresponding to the Hebb rule can be considered
187: a prototype of any other learning rules.
188: %それらのアーキテクチャの元にさまざまな学習則が適用される．
189: %Correlation learning corresponding to Hebb rule
190: %can be considered as a prototype of the other many learning rules.
191: %Hebb則に対応する相関学習は
192: %その他の多くの学習則のプロトタイプと考えられる．
193: For instance, correlation learning can be regarded
194: as a first-order approximation of the orthogonal projection matrix,
195: because the orthogonal projection matrix
196: can be expanded by correlation matrices \cite{Yanai1996}.
197: %例えば直交射影行列は相関行列で展開することができるので，
198: %相関学習は直交射影行列の第1近似であると考えられる\cite{Yanai1996}．
199: In this respect, we can naturally regard
200: a correlation-type associative memory model
201: as one prototype of the neural network models of the brain.
202: %その点から考えると相関型連想記憶モデルは
203: %脳の神経回路モデルのプロトタイプのひとつであると考えることは
204: %それほど不自然ではない．
205: For example, Amit et al. discussed
206: the function of the column of anterior ventral temporal cortex
207: by means of a model based on correlation-type associative memory model
208: \cite{Amit1994,Griniasty1993}.
209: %例えばAmitは相関型連想記憶モデルに基づくモデルで
210: %Anterior Ventral Temporal Cortexのコラムの機能を議論している．
211: Also, Sompolinsky discussed the effect of dilution.
212: He assumed that the capacity is proportional
213: to the number of remaining bonds,
214: and pointed out that
215: a synapse efficiency of diluted network,
216: which is storage capacity per a connecting rate,
217: is higher than a full connected network's one \cite{Sompolinsky1986b}.
218: Chechik et al. discussed the significance
219: of the function of the pruning following over-growth
220: on the basis of a correlation-type associative memory model
221: \cite{Chechik1998}.
222: %実際，Chechikら\cite{Chechik1998}は
223: %シナプスの過剰生成と刈り込みの機能的な意義を
224: %相関型連想記憶モデルに基づいて議論している
225: %\cite{Amit1994}\cite{Griniasty1993}．
226: They pointed out that
227: a memory performance, which is stored pattern number per synapse number,
228: is maximized by systematic deletion
229: that cuts synapses that are lightly weighted.
230: %彼らは相関学習した結合荷重の小さいシナプスを切断するという
231: %系統的な刈り込みにより，
232: %シナプス1本あたりの記憶容量(シナプス効率)が
233: %大きくなることを指摘している．
234: However, while it is qualitatively obvious
235: that synapse efficiency and memory performance
236: are increased by a systematic deletion,
237: we also need to consider the increase of synapse efficiency quantitatively.
238: %しかしながら，このような系統的な刈り込みが
239: %シナプス効率を上昇させることは定性的には明らかであるので，
240: %シナプス効率がどの程度上昇するかというより定量的な
241: %議論が必要なのはいうまでもない．
242:
243: In this paper,
244: we quantitatively compare
245: the effectiveness of systematic deletion to that with random deletion
246: on the basis of an auto-correlation-type associative memory model.
247: %本稿では自己相関型連想記憶モデルに関して，
248: %系統的な切断がランダムな切断に対して
249: %どの程度の効率的かを定量的に議論する．
250: In this model,
251: one neuron is connected to other neurons
252: with a proportion of $c$,
253: where $c$ is called the connecting rate.
254: %ひとつのニューロンがモデル全体の素子に対して
255: %$c$の割合で結合している場合を議論する．
256: %The parameter $c$ is called connecting rate.
257: %$c$を結合率と呼ぶ．
258: Systematic deletion is considered as a kind of nonlinear correlation learning
259: \cite{Okada1998}.
260: %系統的な切断は非線形な相関学習と考えられる．
261: At the limit where the number of neurons $N$ is extremely large,
262: it is known that
263: random deletion and nonlinear correlation learning
264: can be transformed into correlation learning with synaptic noise
265: \cite{Sompolinsky1986b,Okada1998}.
266: %回路の素子数$N$が十分大きな極限では，
267: %ランダムな切断や非線形な相関学習は
268: %シナプスノイズが存在するもとでの
269: %相関学習に変換することが知られている．
270: These two types of deletion, systematic and random, are strongly related
271: to multiplicative synaptic noise.
272: %これらの2種類の切断は
273: %乗法的シナプスノイズと深い関係を持つ．
274: First, we investigated
275: the dependence of storage capacity on multiplicative synaptic noise.
276: %そこでまず，記憶容量の乗法的ノイズ依存性を調べる．
277: At the limit where multiplicative synaptic noise is extremely large,
278: we show that storage capacity is inversely proportional
279: to the variance of the multiplicative synaptic noise.
280: %その結果，乗法的ノイズが十分大きな極限では
281: %記憶容量は乗法的ノイズの分散に反比例することを示す．
282: From this result,
283: we analytically derive that
284: the synapse efficiency in the case of systematic deletion
285: diverges as $O(|\log c|)$
286: at the limit where the connecting rate $c$ is extremely small.
287: We also show that the synapse efficiency
288: in the case of systematic deletion becomes $2 |\log c|$ times
289: as large as that of random deletion.
290: %この知見から
291: %結合率$c$が十分小さい極限で
292: %系統的切断とランダム切断のシナプス効率比が
293: %$2 | \log_e c |$であることを解析的に求める．
294:
295: In addition to such the fixed neuron number criterion
296: as the synapse efficiency,
297: a fixed synapse number criterion could be discussed.
298: %このような素子数一定とした議論の他に，
299: %シナプス数一定とする議論も考えられる．
300: At the synaptic growth stage,
301: it is natural to assume
302: that metabolic energy resources are restricted.
303: %シナプスの形成段階において，
304: %代謝資源は無尽蔵ではなく制限があると考えられる．
305: When metabolic energy resources are limited,
306: it is also important
307: that the effect of synaptic pruning is discussed
308: under limited synapse number.
309: Under this criterion,
310: the optimal connecting rate, which maximize memory performance, exists.
311: These optimal connecting rates are in agreement
312: with the computer simulation results
313: given by Chechik et al \cite{Chechik1998}.
314:
315:
316:
317:
318:
319:
320:
321:
322: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
323: %%
324: %% モデル
325: %%
326:
327:
328: \section{Model}
329: \label{sec.sparse_noise.model}
330:
331:
332: Sompolinsky discussed
333: the effects of synaptic noise and nonlinear synapse
334: by means of the replica method \cite{Sompolinsky1986b}.
335: %Sompolinsky\cite{Sompolinsky1986b}はレプリカ法によって
336: %シナプスノイズ及び非線形シナプスの影響について議論した．
337: However,
338: symmetry of the synaptic connections $J_{ij}=J_{ji}$
339: is required in the replica method
340: since the existence of the Ljapunov function is necessary.
341: %しかし，レプリカ法では，
342: %系のダイナミクスにLjapunov関数の存在が必要であるため，
343: %結合荷重$J_{ij}$の対称性$J_{ij}=J_{ji}$が要求される．
344: Therefore, there was a problem that
345: the symmetry regarding synaptic noise
346: had to be assumed in the Sompolinsky theory.
347: %そのため，Sompolinsky理論では
348: %シナプスノイズについても対称性を仮定しなければならないという問題があった．
349: To avoid this problem, Okada et al. discussed
350: additive synaptic noise, multiplicative synaptic noise,
351: random synaptic deletion, and nonlinear synapse
352: by means of the self-consistent signal-to-noise analysis (SCSNA)
353: \cite{Okada1998}.
354: %Okadaら\cite{Okada1998}は，
355: %その問題を避けるためSCSNAを用いて，
356: %加法的シナプスノイズ，乗法的シナプスノイズ，
357: %無作為シナプス切断及び非線形シナプス
358: %について議論した．
359: They showed that
360: additive synaptic noise,
361: random synaptic deletion, nonlinear synapse
362: can be transformed into multiplicative synaptic noise.
363: %Okadaは，
364: %これらの加法的シナプスノイズ，無作為シナプス切断，非線形シナプスは
365: %はすべて乗法的シナプスノイズに変換できることを示した．
366:
367: Here,
368: we discuss the synchronous dynamics as,
369: %本報告では，次の非同期ダイナミクスを取り扱う．
370: \begin{equation}
371: x_i = F(\sum_{j \neq i}^N J_{ij} x_j + h),
372: \label{5eq:sparse.equilibrium}
373: \end{equation}
374: where $F$ is the response function,
375: $x_i$ is the output activity of neuron $i$,
376: and $-h$ is the threshold of each neuron.
377: %ここで，$h$は素子のしきい値を表す．
378: Every component $\xi_i^\mu$ in a memorized pattern
379: $\mbox{\boldmath $\xi$}^\mu$ is an independent random variable,
380: %記憶パターンは，
381: \begin{equation}
382: {\rm Prob}[\xi_i^\mu=\pm 1]=\frac{1 \pm a}2,
383: \label{5eq:sparse.pattern}
384: \end{equation}
385: and the generated patterns are called sparse pattern with bias $a \; (-1<a<1)$.
386: %によって生成される，偏り$a (-1<a<1)$の
387: %スパースパターンとする．
388: We have determined that the firing rate of states in the retrieval phase
389: is the same for each memorized pattern \cite{Amit1987,Amari1989}.
390: %ここでは，想起中のネットワークの状態の発火率を
391: %記憶パターンの発火率と同じ状態に保つように決定する
392: %\cite{Amit1987}\cite{Amari1989}．
393: In this case, threshold $-h$ can be determined as,
394: %これは時刻によって次の方程式，
395: \begin{equation}
396: a = \frac{1}{N} \sum_{i=1}^N \mbox{sgn}(\sum_{j \neq i } J_{ij} x_i + h).
397: \label{5eq:sparse.constraint.1}
398: \end{equation}
399: %を満たすようにしきい値$h$は決めるとよい．
400: The firing rate becomes $f=(1+a)/2$ at the bias $a$.
401: %また，偏り$a$によりパターンの発火率$f$は
402: %$f=(1+a)/2$となる．
403:
404: Additive synaptic noise, multiplicative synaptic noise,
405: random synaptic deletion, and nonlinear synapse can be introduced
406: by synaptic connections in the following manner.
407: %これらの，
408: %加法的シナプスノイズ，乗法的シナプスノイズ，
409: %無作為シナプス切断及び非線形シナプスは，
410: %次のように結合荷重を構成することによりモデルに導入することができる．
411:
412: In the case of additive synaptic noise,
413: synaptic connections are constituted as,
414: %加法的ノイズを考慮したモデルは，
415: \begin{equation}
416: J_{ij} = \frac{J}{N(1-a^2)}
417: \sum_{\mu = 1}^{\alpha N} (\xi_i^\mu-a) (\xi_j^\mu-a)
418: + \delta_{ij},
419: \label{5eq:add.noise.J}
420: \end{equation}
421: where $\delta_{ij}$ is the additive synaptic noise.
422: %と構成する．
423: %ここで，$\delta_{ij}$は加法的ノイズを表す．
424: The symmetric additive synaptic noise $\delta_{ij}$ and $\delta_{ji}$
425: are generated according to the probability,
426: %加法的ノイズの分散を$\Delta_A^2$として，
427: %非対称加法的ノイズは，$\delta_{ij}$と$\delta_{ji}$を，
428: %次の確率で独立に生成する．
429: \begin{equation}
430: \delta_{ij} \sim N (0,\frac{\delta_A^2}{N} ), \quad
431: \delta_{ij}=\delta_{ji},
432: \end{equation}
433: where $\delta_A^2$ is the absolute strength
434: of the additive synaptic noise.
435: The parameter $\delta_A$ is assumed to be $O(1)$.
436: This means that the synaptic connection $J_{ij}$ is $O(1/\sqrt{N})$.
437: It is useful to define the parameter $\Delta_A$ as
438: \begin{equation}
439: \Delta_A \equiv \frac{\delta_A}{J/(1-a^2)},
440: \end{equation}
441: which measures the relative strength of the noise
442: and we call the parameter $\Delta_A^2$
443: the variance of the additive synaptic noise.
444: Therefore, we define the probability
445: to generate the additive synaptic noise $\delta_{ij}$ as
446: \begin{equation}
447: \delta_{ij} \sim N (0,\frac{J^2}{N(1-a^2)^2} \Delta_A^2 ), \quad
448: \delta_{ij}=\delta_{ji}.
449: \label{5eq:add.noise.pdf}
450: \end{equation}
451:
452: In the case of multiplicative synaptic noise,
453: synaptic connections are constituted as,
454: %乗法的ノイズを考慮したモデルは，
455: \begin{equation}
456: J_{ij}
457: = \frac{1+\varepsilon_{ij}}{N(1-a^2)}
458: \sum_{\mu = 1}^{\alpha N} (\xi_i^\mu-a) (\xi_j^\mu-a),
459: \label{5eq:mul.noise.J}
460: \end{equation}
461: where $\varepsilon_{ij}$ is multiplicative synaptic noise.
462: %と構成する．
463: %この$\varepsilon_{ij}$により乗法的ノイズを表す．
464: The symmetric multiplicative noise
465: $\varepsilon_{ij}$ and $\varepsilon_{ji}$
466: are generated according to the probability,
467: %乗法的ノイズの分散$\Delta_M^2$として，
468: %非対称乗法的ノイズは，$\varepsilon_{ij}$と$\varepsilon_{ji}$を
469: %次の確率で独立に生成する．
470: \begin{equation}
471: \varepsilon_{ij} \sim N(0,\Delta_M^2), \quad
472: \varepsilon_{ij}=\varepsilon_{ji},
473: \label{5eq:mul.noise.pdf}
474: \end{equation}
475: where $\Delta_M^2$ is the variance of the multiplicative synaptic noise.
476:
477: In the model of random synaptic deletion,
478: synaptic connections are constituted as,
479: %無作為シナプス切断を考慮したモデルは，
480: \begin{equation}
481: J_{ij} = \frac{c_{ij}}{Nc(1-a^2)}
482: \sum_{\mu = 1}^{\alpha N} (\xi_i^\mu-a) (\xi_j^\mu-a),
483: \label{5eq:random.cut.J}
484: \end{equation}
485: where $c_{ij}$ is a cut coefficient.
486: %と構成する．
487: The synapse that is cut is represented
488: by the cut coefficient $c_{ij}=0$.
489: %ここで，切断されるシナプスは$c_{ij}=0$とすることにより，
490: %切断が係数$c_{ij}$によって導入される．
491: In the case of symmetric random deletion,
492: the cut coefficients $c_{ij}$ and $c_{ji}$
493: are generated according to the probability,
494: %非対称切断は，$c_{ij}$と$c_{ji}$を下記の確率で独立に生成する．
495: \begin{equation}
496: \mbox{Prob}[c_{ij}=1] = 1-\mbox{Prob}[c_{ij}=0]=c, \quad
497: c_{ij}=c_{ji},
498: \label{5eq:cut.pdf}
499: \end{equation}
500: where $c$ is the connecting rate.
501: %この$c$は結合率を表す．
502:
503: In the model of nonlinear synapse,
504: synaptic connections are constituted as,
505: %非線形シナプスを考慮したモデルは，
506: \begin{eqnarray}
507: J_{ij} &=& \frac{\sqrt{p}}{N} f(T_{ij}),
508: \label{5eq:nonlinear.learning} \\
509: T_{ij} &=& \frac{1}{\sqrt{p}(1-a^2)} \sum_{\mu = 1}^{p}
510: (\xi_{i}^{\mu} - a) (\xi_{j}^{\mu} - a) \nonumber \\
511: & \sim & N(0,1) ,
512: \label{5eq:nonlinear.T}
513: \end{eqnarray}
514: where $p=\alpha N$.
515: %と構成する．
516: %ここで，$p=\alpha N$とする．
517: The nonlinear synapse is introduced
518: by applying the nonlinear function $f(x)$
519: to the conventional Hebbian connection matrix $T_{ij}$.
520: %各素子が受け取る入力$T_{ij}$に，
521: %非線形関数$f(x)$を作用させることにより非線形シナプスを導入する．
522:
523:
524:
525:
526:
527:
528:
529:
530: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
531: %%
532: %% 系統的切断
533: %%
534:
535:
536: \subsection{Systematic deletion by nonlinear synapse}
537: %非線形シナプスによる系統的切断
538: \label{sec.sparse_noise.systematic.cut}
539:
540: %Chechikら\cite{Chechik1998}は
541: %シナプスの過剰生成と刈り込みの機能的な意義を
542: %相関型連想記憶モデルに基づいて議論し
543: %\cite{Amit1994}\cite{Griniasty1993}，
544: %相関学習した結合荷重の小さいシナプスを切断するという
545: %系統的な刈り込みにより，
546: %シナプス1本あたりの記憶容量(シナプス効率)が
547: %大きくなることを指摘している．
548: Chechik et al. pointed out
549: that a memory performance,
550: which is a storage pattern number per a synapse number, is maximized
551: by systematic deletion that cuts synapses that are lightly weighted
552: \cite{Chechik1998,Amit1994,Griniasty1993}.
553: Such a systematic deletion can be represented
554: by the nonlinear function $f(x)$ for a nonlinear synapse.
555: %このような系統的切断は，
556: %前節の非線形シナプスの非線形関数$f(x)$により表現することが可能である．
557: In accordance with Chechik et al.,
558: we discuss three types of nonlinear functions
559: (Figs.\ref{5fig:nonlinear.clip},
560: \ref{5fig:nonlinear.minimal}, and \ref{5fig:nonlinear.compress}).
561: %ここではChechikらの論文に従い，
562: %シナプスの系統的切断について
563: %図\ref{5fig:nonlinear.clip}〜
564: %図\ref{5fig:nonlinear.compress}に示す3種類の非線形関数について議論する．
565: Fig.\ref{5fig:nonlinear.clip} shows clipped modification that is
566: discussed generally as
567: %図\ref{5fig:nonlinear.clip}に示す，
568: \begin{equation}
569: f_1(z,t) = \left\{ \begin{array}{ll}
570: \sgn(z), & |z|>t \\
571: 0, & \mbox{otherwise} .
572: \end{array}
573: \right.
574: \label{5eq:nonlinear.clip}
575: \end{equation}
576: %は，刈込変換であり広く議論されている．
577: %-------------レフリーの指摘により削除（ここから）
578: %In general, there are many cases
579: %where clipped synapse is the same modification as clipped modification.
580: %%一般にクリップシナプスというとこの刈込変換を表す場合が多い．
581: %-------------レフリーの指摘により削除（ここまで）
582:
583: Chechik et al. also obtained
584: the nonlinear functions shown
585: in Figs.\ref{5fig:nonlinear.minimal} and \ref{5fig:nonlinear.compress}
586: by applying the following optimization principles
587: \cite{Chechik1998,Meilijson1996}.
588: %Chechikらは，
589: %図\ref{5fig:nonlinear.minimal}及び
590: %図\ref{5fig:nonlinear.compress}の非線形関数を，
591: %最適化原理から求めている\cite{Meilijson1996}．
592: In order to evaluate the effect of synaptic pruning on the network's retrieval performance,
593: Chechik et al. study its effect on the signal-to-noise ratio (S/N) of
594: the internal field $h_i \equiv \sum_{j \ne i} J_{ij} x_j + h$ \cite{Chechik1998}.
595: The S/N is calculated by analyzing the moments of the internal field and was given as
596: \begin{eqnarray}
597: S/N
598: &=& \frac{E[h_i |\xi_i^\mu=1]-E[h_i |\xi_i^\mu=-1]} {\sqrt{V[h_i|\xi_i^\mu]}} \nonumber \\
599: &\propto& \frac{E[zf(z)]}{\sqrt{E[f(z)^2]}}
600: = \frac{\displaystyle{\int_{-\infty}^\infty Dz \; zf(z)}}
601: {\displaystyle{\int_{-\infty}^\infty Dz \; f(z)^2}} \nonumber \\
602: &\equiv& \rho(f(z),z),
603: \end{eqnarray}
604: where $z$ has standard normal distribution, i.e., $E[z^2]=V[z]=1$ and $Dz$ is Gaussian measure defined as
605: \begin{equation}
606: Dz = \frac {dz}{\sqrt {2 \pi}} \exp (-\frac{z^2}2),
607: \end{equation}
608: and $E,V$ denote the operators to calculate the expectation and the variance
609: for the random variables $\xi_i^\mu \; (i=1,\cdots ,N, \mu=1, \cdots ,\alpha N)$,
610: respectively.
611: The function $\rho(f(z),z)$ denotes the correlation coefficient.
612: Chechik et al. considered the piecewise linear function like following nonlinear functions.
613: %まず，非線形関数として区分線形関数に限定して考える．
614: %%この関数$g(x)$は$f(z)=0,z \notin A$であり，
615: %%集合$A$は刈り込まれない結合荷重の範囲を示す集合を表している．
616: In order to find the best nonlinear function $f(z)$,
617: we should maximize $\rho(f(z),z)$, which is invariant to scaling.
618: %%結合荷重の構成において，
619: %%学習時の記憶の強度に依存しない議論にするために，
620: %\begin{equation}
621: %\rho(f(z),z)=\frac{E[zf(z)]}{\sqrt{E[f(z)^2]}},
622: %\end{equation}
623: %in order to not depend on the strength of memory.
624: %を最大にする非線形関数$f(x)$を求める．
625: Namely, the best nonlinear function $f(x)$ is obtained
626: by maximizing $E[zf(z)]$ under the condition that $E[f(z)^2]$ is constant.
627: %つまり，$E[f(z)^2]$を一定にするという条件のもとで，
628: %$E[zf(z)]$を最大にする．
629: Let $\gamma$ be the Lagrange multiplier, it is sufficient to solve
630: %すなわち，Lagrange未定定数を$\gamma$として，
631: \begin{equation}
632: \int_{-\infty}^\infty Dz \; zf(z)
633: - \gamma \left( \int_{-\infty}^\infty Dz \; f(z)^2 -c_0 \right)
634: \to \mbox{max},
635: \end{equation}
636: for some constant $c_0$.
637: %とする．
638: Since the synaptic connection before acting the nonlinear function $T_{ij}$ obeys
639: a Gaussian distribution $N(0,1)$, Eq.(\ref{5eq:nonlinear.T}) is averaged over all of the synaptic connections.
640: %非線形関数を作用させる前の結合荷重$T_{ij}$は，
641: %式(\ref{5eq:nonlinear.T})のようにGauss分布$N(0,1)$に従うので，
642: %結合荷重の分布に関する平均をとっている．
643: %ここで，Gauss分布$N(0,1)$の分布関数を
644: %$\phi(z)=\exp(-z^2/2)/\sqrt{2\pi}$とおいた．
645: %また，$c$は定数である．
646:
647:
648:
649:
650:
651: %!--注釈---%
652: %これは，Sompolinsky理論\cite{Sompolinsky1986b}において，
653: %式(\ref{2eq:delta})の${{\tilde J}^2}/{J^2}$を最小にすることに対応している．
654:
655:
656:
657:
658: Thus, the nonlinear function shown in Fig.\ref{5fig:nonlinear.minimal},
659: %このようにして，
660: %図\ref{5fig:nonlinear.minimal}に示す，
661: \begin{equation}
662: f_2(z,t) = \left\{ \begin{array}{ll}
663: z, & |z|>t \\
664: 0, & \mbox{otherwise} ,
665: \end{array}
666: \right.
667: \label{5eq:nonlinear.minimal}
668: \end{equation}
669: is also obtained.
670: %を得る．
671: The deletion by this nonlinear function is called minimal value deletion.
672: %この関数による切断を最小値切断と呼ぶ．
673: Similarly, by adding the condition
674: that the total strength of synaptic connection
675: $\int Dz \; |f(z)|$ is constant,
676: the nonlinear function
677: %また，先と同様にして，
678: %全結合荷重の強度$\int |f(z)|\phi(z)dz$を一定とする条件を加えることにより
679: %図\ref{5fig:nonlinear.compress}に示す，
680: \begin{eqnarray}
681: f_3(z,t) &=& \left\{ \begin{array}{ll}
682: z-\sgn(z)t, & |z|>t \\
683: 0, & \mbox{otherwise}
684: \end{array}
685: \right. \nonumber \\
686: &=& f_2(z,t)-tf_1(z,t),
687: \label{5eq:nonlinear.compress}
688: \end{eqnarray}
689: is obtained.
690: %が得られる．
691: The deletion by this nonlinear function is called compressed deletion.
692: %この関数による切断を圧縮切断と呼ぶ．
693: We discuss systematic deletion
694: by using these three types of nonlinear functions
695: $f_1(z,t),f_2(z,t)$ and $f_3(z,t)$ given by Chechik et al \cite{Chechik1998}.
696: %これら$f_1(x),f_2(x),f_3(x)$の3種類の非線形関数をもちいて
697: %系統的切断を議論する．
698:
699:
700:
701:
702:
703: \begin{figure}
704: \epsfig{fig1.eps}{6cm}
705: \caption[Clipped synapse]{Clipped synapse.}
706: \label{5fig:nonlinear.clip}
707: \end{figure}
708:
709:
710: \begin{figure}
711: \epsfig{fig2.eps}{6cm}
712: \caption[Minimal value deletion]{Minimal value deletion.}
713: \label{5fig:nonlinear.minimal}
714: \end{figure}
715:
716:
717:
718: \begin{figure}
719: \epsfig{fig3.eps}{6cm}
720: \caption[Compress deletion]{Compressed deletion.}
721: \label{5fig:nonlinear.compress}
722: \end{figure}
723:
724:
725:
726:
727:
728:
729:
730:
731:
732:
733:
734:
735:
736:
737: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
738: %%
739: %% 結果
740: %%
741:
742:
743: \section{Results}
744: %結果
745: \label{sec.sparse_noise.result}
746:
747:
748: In this section,
749: the results concerning the multiplicative synaptic noise,
750: the random deletion, and the nonlinear synapse are shown, at the limit
751: where the effect of these deletions is extremely large.
752: %本節では，
753: %乗法的ノイズ，無作為切断，非線形シナプスが
754: %ノイズとして大きく影響する極限についての結果を示す．
755:
756: The SCSNA starts from the fixed-point equations
757: for the dynamics of an $N$-neuron network
758: as Eq.(\ref{5eq:sparse.equilibrium}).
759: The results of the SCSNA for the symmetric additive synaptic noise
760: are summarized by the following order-parameter equations
761: (see Appendix \ref{appendix.scsna}) :
762: \begin{eqnarray}
763: m &=& \frac{1}{1-a^2} \int Dz < (\xi -a) Y(z;\xi ) >_\xi ,
764: \label{eq:order_parameter_equation_m} \\
765: q &=& \int Dz < Y(z;\xi )^2 >_\xi ,
766: \label{eq:order_parameter_equation_q} \\
767: U &=& \frac{1}{\sigma} \int Dz \; z < Y(z;\xi ) >_\xi ,
768: \label{eq:order_parameter_equation_U} \\
769: \sigma^2 &=& \frac{\alpha J^2 q}{(1-JU)^2}
770: + \frac{J^2}{(1-a^2)^2} \Delta_A^2 q.
771: \label{eq:order_parameter_equation_sigma} \\
772: Y(z;\xi )
773: &=& F \biggl( J (\xi-a) m + \sigma z + h \nonumber \\
774: & & + \biggl[ \frac{\alpha J^2 U}{1-JU}+\frac {J^2}{(1-a^2)^2} \Delta_A^2 \biggr]
775: Y(z;\xi ) \biggr) , \label{eq:effevtive_response_function}
776: \end{eqnarray}
777: where $< \cdots >_\xi$ implies averaging over the target pattern,
778: $m$ is the overlap between the 1st memory pattern $\vec{\xi}^1$
779: and the equilibrium state $\vec{x}$ is defined as
780: \begin{equation}
781: m = \frac{1}{N(1-a^2)} \sum_{i=1}^N (\xi^1_i-a) x_i,
782: \label{eq:definition_overlap}
783: \end{equation}
784: note that generality is kept
785: even if the overlap was defined by only the 1st memory pattern,
786: $q$ is Edwards-Anderson order-parameter,
787: $U$ is a kind of the susceptibility, which measures
788: sensitivity of neuron output with respect to the external input,
789: $Y(z;\xi )$ is effective response function,
790: $\sigma^2$ is the variance of the noise.
791: %ここで各パラメータはそれぞれ，
792: %$Y(z)$は式(\ref{4eq:Y})を解いて得られる有効的な出力関数，
793: %$m$は式(\ref{4eq:overlap})で定義される
794: %記憶パターン$\bxi^1$と出力とのオーバラップ，
795: %$q$はEdwards-Anderson秩序変数，
796: %$U$はリカレント結合によって生じる相関，
797: %$r$はクロストークノイズの分散，
798: %$\Gamma Y(z)$は，有効的な自己結合による影響を表す項，
799: %$\sigma^2$はGauss分布に従うノイズの分散を
800: %表している．
801: We set the output function $F(x)={\rm sgn}(x)-a$ in the following sections,
802: where domain of the $x$ variable is $F(x)=1-a$ when $x \le 0$, otherwise $F(x)=-1-a$.
803:
804:
805:
806:
807: According to Okada et al. \cite{Okada1998},
808: the symmetric additive synaptic noise, the symmetric random deletion,
809: and the nonlinear synapse can be transformed
810: into the symmetric multiplicative synaptic noise as follows
811: (see Appendix \ref{appendix.equivalent_noise}):
812: %Okadaら\cite{Okada1998}は，
813: %前述のように平衡状態においては，
814: %加法的ノイズ，乗法的ノイズ，無作為切断及び
815: %非線形シナプスは次のように乗法的ノイズに変換できることを示した．
816: the additive synaptic noise is
817: \begin{equation}
818: \Delta_M^2 = \frac{\Delta_A^2}{\alpha(1-a^2)^2},
819: \label{5eq:add.to.mul}
820: \end{equation}
821: the random deletion is
822: \begin{equation}
823: \Delta_M^2 = \frac {1-c}{c},
824: \label{5eq:cut.to.mul}
825: \end{equation}
826: and the nonlinear synapse is
827: \begin{equation}
828: \Delta_M^2 = \frac{{\tilde{J}}^2}{J^2} -1,
829: \label{5eq:nonlinear.to.mul}
830: \end{equation}
831: where $J, \tilde{J^2}$ are
832: %ここで，$J, \tilde{J^2}$は，
833: \begin{eqnarray}
834: J &=& \int Dz \; z f(z),
835: \label{5eq:nonlinear.J} \\
836: {\tilde J}^2 &=& \int Dz \; f(z)^2.
837: \label{5eq:nonlinear.tildeJ}
838: \end{eqnarray}
839: %である．
840: In the following sections, symmetries of the additive synaptic noise,
841: the multiplicative synaptic noise and random deletion are assumed.
842: Storage capacity can be obtained
843: by solving the order-parameter equations.
844: %記憶容量はSCSNAのオーダパラメータ方程式を解いて求めることができる．
845:
846: Figure \ref{5fig:computer_simulation} shows
847: $m(\alpha )$ curves in the random deletion network
848: with the number of neurons $N=3000$ and the firing rate $f=0.1$
849: for the connecting rate $c=0.1$,$c=0.3$, and $c=1.0$.
850: It can be confirmed that
851: the theoretical results of the SCSNA are in good agreement
852: with the computer simulation results from Fig.\ref{5fig:computer_simulation}.
853: Since it is known that
854: theoretical results obtained by means of the SCSNA
855: are generally in good agreement
856: with the results obtained through the computer simulations
857: using various models that include synaptic noise,
858: we treat the results by means of the SCSNA only
859: \cite{Shiino1992,Shiino1993,Okada1995,Okada1996,Okada1998,Mimura1998-1,Mimura1998-2,Kimoto2001}.
860: %また，本稿で議論した結合荷重にノイズが含まれるような系に関しては，
861: %SCSNAは計算機シミュレーションの結果をよく再現することが
862: %知られているのでSCSNAによる結果のみを示す．
863:
864: Through the relationships
865: of Eqs.(\ref{5eq:add.to.mul})-(\ref{5eq:nonlinear.tildeJ}),
866: the symmetric additive synaptic noise, the symmetric random deletion, and the nonlinear synapse
867: can be discussed in terms of the symmetric multiplicative synaptic noise.
868: Therefore, first of all, we deal with the multiplicative synaptic noise.
869: %これらの関係によって，
870: %全ての乗法的ノイズによって議論することが可能となるので，
871: %まず乗法的ノイズについて議論する．
872:
873:
874:
875:
876: \begin{figure}
877: \epsfig{fig4.eps}{8cm}
878: \caption{
879: Overlaps in the random deletion network.
880: The curves represent the theoritical results.
881: The dots represent simulation results with
882: $N=3000$ and $f=0.1$
883: for the connecting rate $c=0.1$,$c=0.3$, and $c=1.0$.
884: }
885: \label{5fig:computer_simulation}
886: \end{figure}
887:
888:
889:
890:
891:
892:
893:
894:
895: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
896: %%
897: %% 乗法的ノイズ
898: %%
899:
900:
901: \subsection{Multiplicative synaptic noise}
902: %乗法的ノイズ
903: \label{sec.sparse_noise.result.mul}
904:
905: Figure \ref{5fig:mul.ac-delta.f=0.5} shows
906: the dependence of storage capacity on the multiplicative synaptic noise.
907: %記憶容量の乗法的ノイズ依存性を図\ref{5fig:mul.ac-delta.f=0.5}に示す．
908: As it is clear from Fig.\ref{5fig:mul.ac-delta.f=0.5},
909: storage capacity $\alpha_c$ is inversely proportional to
910: the variance of the multiplicative synaptic noise $\Delta_M^2$,
911: when the multiplicative synaptic noise is extremely large.
912: %図\ref{5fig:mul.ac-delta.f=0.5}からもわかるように
913: %乗法的ノイズの大きい極限では，
914: %記憶容量$\alpha_c$は乗法的ノイズの分散$\Delta_M^2$に反比例する．
915: %The reason for this is as follows.
916: %%この理由を議論する．
917: %The critical value of the additive synaptic noise $\Delta_{Ac}^2$
918: %at which storage capacity vanishes becomes $\Delta_{Ac}^2=2/\pi$
919: %(see Appendix \ref{appendix.critical_noise}) \cite{Sompolinsky1986c}.
920: %記憶容量が0になる加法的ノイズの分散の臨界値$\Delta_{Ac}^2$は，
921: %$\Delta_{Ac}^2=2/\pi$となる．
922: %From Eq.(\ref{5eq:add.to.mul}),
923: %the variance of the multiplicative synaptic noise $\Delta_M^2$
924: %is related to the variance of the additive synaptic noise $\Delta_A^2$ as
925: %%偏り$a=0$において，
926: %%式(\ref{5eq:add.to.mul})のように
927: %%乗法的ノイズの分散$\Delta_M^2$は加法的ノイズ$\Delta_A^2$の分散と，
928: %\begin{equation}
929: %\Delta_A^2 = \alpha \Delta_M^2 , \label{eq:a2m_at_a=0}
930: %\end{equation}
931: %at the limit where the multiplicative synaptic noise is extremely large when bias $a=0$.
932: %のように漸近形が求められる．
933: Storage capacity $\alpha_c$ asymptotically approaches
934: %%Therefore, storage capacity $\alpha_c$ asymptotically approaches
935: %の関係を満たすので，
936: %乗法的ノイズの分散$\Delta_M^2$が十分に大きい極限では
937: %記憶容量$\alpha_c$は
938: \begin{equation}
939: \alpha_c
940: = \frac 2{\pi \Delta_M^2},
941: \label{5eq:mul.asymptote}
942: \end{equation}
943: (see Appendix \ref{appendix.critical_noise}).
944: In the sparse limit where the firing rate is extremely small,
945: it is known that storage capacity
946: becomes $\alpha_c \simeq 1/(f|\log f|)$
947: \cite{Tsodyks1988,Amari1989,Buhmann1989,Perez-Vincente1989,Okada1996}.
948: %また，スパースパターンを記憶する場合の記憶容量は，
949: %発火率$f$が小さい極限で，
950: %$\alpha_c \simeq 1/(f|\log f|)$となることが知られている．
951: %%式(\ref{5eq:mul.asymptote})の漸近形に
952: %%係数$(\frac 12 |\log \frac 12|)/(f |\log f|)$を乗じることで得られる．
953:
954: Figure \ref{5fig:mul.ac-delta.f=0.5} shows
955: the results from the SCSNA and the asymptote at the firing rate $f=0.5$.
956: %図\ref{5fig:mul.ac-delta.f=0.5}は，
957: %式(\ref{5eq:mul.asymptote})の発火率$f=0.5$での
958: %漸近形及びSCSNAによる結果を示す．
959: Figure \ref{5fig:mul.ac-delta.all-f} shows
960: the results from the SCSNA at various firing rates.
961: %また，図\ref{5fig:mul.ac-delta.all-f}に
962: %いろいろな発火率でのSCSNAの結果を示す．
963: It can be confirmed
964: that the order of the asymptote $O(\frac 1{\Delta_M^2})$
965: does not depend on the firing rate
966: from Fig.\ref{5fig:mul.ac-delta.all-f}.
967: %図\ref{5fig:mul.ac-delta.all-f}から
968: %漸近形のオーダは発火率に依存しないことがわかる．
969:
970:
971:
972:
973:
974:
975: \begin{figure}
976: \epsfig{fig5.eps}{8cm}
977: \caption[Dependence of storage capacity
978: on multiplicative synaptic noise $\Delta_M^2$]
979: %乗法的ノイズでの記憶容量の結合率依存性
980: {
981: Dependence of storage capacity $\alpha_c$
982: on the multiplicative synaptic noise $\Delta_M^2$
983: at the firing rate $f=0.5$.
984: %乗法的ノイズでの記憶容量$\alpha_c$の結合率$c$依存性(発火率$f=0.5$)．
985: Comparison of asymptote and the results from the SCSNA.
986: %SCSNAと漸近形の比較．
987: }
988: \label{5fig:mul.ac-delta.f=0.5}
989: \end{figure}
990:
991:
992:
993:
994: \begin{figure}
995: \epsfig{fig6.eps}{8cm}
996: \caption[Dependence of storage capacity $\alpha_c$
997: on multiplicative synaptic noise $\Delta_M^2$]
998: %乗法的ノイズでの記憶容量の結合率依存性
999: {
1000: Dependence of storage capacity $\alpha_c$
1001: on the multiplicative synaptic noise $\Delta_M^2$.
1002: It can be confirmed that the order of the asymptote
1003: does not depend on the firing rate.
1004: %乗法的ノイズでの記憶容量$\alpha_c$の結合率$c$依存性．
1005: %It can be confirmed that
1006: %the order of asymptote does not depend on the firing rate.
1007: %漸近形のオーダは発火率に依存しないことが確認できる．
1008: }
1009: \label{5fig:mul.ac-delta.all-f}
1010: \end{figure}
1011:
1012:
1013:
1014:
1015:
1016:
1017:
1018:
1019:
1020: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
1021: %%
1022: %% 無作為切断
1023: %%
1024:
1025:
1026: \subsection{Random deletion}
1027: %無作為切断
1028: \label{sec.sparse_noise.result.random}
1029:
1030: Next, we discuss the asymptote of the random deletion.
1031: %無作為切断の漸近特性について議論する．
1032: The random deletion with the connecting rate $c$ can be transformed
1033: into the multiplicative synaptic noise by Eq.(\ref{5eq:cut.to.mul}).
1034: %結合率$c$の無作為切断は式(\ref{5eq:cut.to.mul})により，
1035: %乗法的ノイズに変換できる．
1036: Hence,
1037: at the limit where the connecting rate $c$ is extremely small,
1038: storage capacity becomes
1039: %これより，結合率$c$が小さい極限において
1040: %式(\ref{5eq:mul.asymptote})の乗法的ノイズの漸近形より，
1041: \begin{equation}
1042: \alpha_c
1043: = \frac{2}{\pi \Delta_M^2}
1044: = \frac {2c}{\pi (1-c)}
1045: \to \frac 2{\pi} c,
1046: \label{5eq:cut.asymptote}
1047: \end{equation}
1048: according to the asymptote of the multiplicative synaptic noise
1049: in Eq.(\ref{5eq:mul.asymptote}).
1050: %となる．
1051: In the random deletion,
1052: the synapse efficiency $S_{eff}$,
1053: which is storage capacity per the connecting rate
1054: \cite{Okada1998,Sompolinsky1986b},
1055: i.e., storage capacity per the input of one neuron,
1056: and defined as
1057: %式(\ref{5eq:cut.asymptote})より
1058: %無作為切断では，
1059: \begin{equation}
1060: S_{eff} \equiv \frac{\alpha_c}{c},
1061: \label{5eq:seff.definition}
1062: \end{equation}
1063: approaches a constant value as
1064: %で定義されるシナプス1本当たりの記憶容量を示す
1065: %シナプス効率$S_{eff}$が，結合率$c$の小さい極限で，
1066: \begin{equation}
1067: S_{eff}=\frac{\alpha_c}{c}=\frac 2{\pi},
1068: \end{equation}
1069: according to Eq.(\ref{5eq:cut.asymptote})
1070: at the limit where the connecting rate $c$ is extremely small.
1071: %と一定になることがわかる．
1072:
1073: Figure \ref{5fig:cut.ac-delta.f=0.5} shows
1074: the result from the SCSNA and the asymptote at the firing rate $f=0.5$.
1075: %図\ref{5fig:cut.ac-delta.f=0.5}は，
1076: %式(\ref{5eq:cut.asymptote})の発火率$f=0.5$での
1077: %漸近形及びSCSNAによる結果を示す．
1078: Figure \ref{5fig:cut.ac-delta.all-f} shows
1079: the results from the SCSNA at various firing rates.
1080: It can be confirmed that the order
1081: of the asymptote, $O(1)$ with respect to $c$,
1082: does not depend on the firing rate
1083: from Fig.\ref{5fig:cut.ac-delta.all-f}.
1084: %また，図\ref{5fig:cut.ac-delta.all-f}に
1085: %いろいろな発火率でのSCSNAの結果を示す．
1086: %It can be confirmed
1087: %that the order of asymptote does not depend on firing rate
1088: %as shown in Fig.\ref{5fig:cut.ac-delta.all-f}.
1089: %図\ref{5fig:cut.ac-delta.all-f}から
1090: %漸近形のオーダは発火率に依存しないことがわかる．
1091:
1092:
1093:
1094: \begin{figure}
1095: \epsfig{fig7.eps}{8cm}
1096: \caption[Dependence of the synapse efficiency $S_{eff}$
1097: on random deletion with the connecting rate $c$]
1098: %無作為切断でのシナプス効率の結合率依存性
1099: {
1100: Dependence of the synapse efficiency $S_{eff}$
1101: on the random deletion with the connecting rate $c$
1102: at the firing rate $f=0.5$.
1103: %無作為切断でのシナプス効率$S_{eff}$の結合率$c$依存性(発火率$f=0.5$)．
1104: Comparison of asymptote and the results from the SCSNA.
1105: %SCSNAと漸近形の比較．
1106: }
1107: \label{5fig:cut.ac-delta.f=0.5}
1108: \end{figure}
1109:
1110:
1111: \begin{figure}
1112: \epsfig{fig8.eps}{8cm}
1113: \caption[Dependence of the synapse efficiency $S_{eff}$
1114: on random deletion with the connecting rate $c$ at various firing rates. ]
1115: %無作為切断でのシナプス効率の結合率依存性(発火率による変化)
1116: {
1117: Dependence of the synapse efficiency $S_{eff}$
1118: on the random deletion with the connecting rate $c$ at various firing rates.
1119: It can be confirmed that the order of the asymptote
1120: does not depend on the firing rate.
1121: %無作為切断でのシナプス効率$S_{eff}$の結合率$c$依存性．
1122: %It can be confirmed that
1123: %the order of asymptote does not depend on firing rate.
1124: %漸近形のオーダは発火率に依存しないことが確認できる．
1125: }
1126: \label{5fig:cut.ac-delta.all-f}
1127: \end{figure}
1128:
1129:
1130:
1131:
1132:
1133:
1134:
1135: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
1136: %%
1137: %% 系統的切断
1138: %%
1139:
1140:
1141: \subsection{Systematic Deletion}
1142: %系統的切断
1143: \label{sec.sparse_noise.result.systematic}
1144:
1145:
1146:
1147:
1148:
1149: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
1150: %%
1151: %% 刈込変換
1152: %%
1153:
1154:
1155: \subsubsection{Clipped synapse}
1156: %刈込変換
1157: \label{sec.sparse_noise.result.systematic.clip}
1158:
1159: Synapses within the range $-t<z<t$ are pruned
1160: by the nonlinear function of Eq.(\ref{5eq:nonlinear.clip}).
1161: %式(\ref{5eq:nonlinear.clip})の非線形関数$f_1(z,t)$によって，
1162: %$-t<z<t$の範囲のシナプスが刈り込まれる．
1163:
1164: The connection rate $c$
1165: of the synaptic deletion in Eq.(\ref{5eq:nonlinear.T}) is given by,
1166: %式(\ref{5eq:nonlinear.T})の非線形関数を作用させる前の結合荷重$T_{ij}$が
1167: %Gauss分布$N(0,1)$に従うことから，
1168: %非線形関数$f_1(z,t)$による系統的切断は等価的に，
1169: \begin{eqnarray}
1170: c&=&\int_{ \{ z|f_1(z,t) \ne 0 \} } Dz
1171: =1-\erf (\frac{t}{\sqrt 2}) \nonumber \\
1172: & \to &
1173: \sqrt{\frac {2}{\pi}} t^{-1} \exp (-\frac {t^2}2), \; \; t \to \infty,
1174: \label{5eq:clip.connect}
1175: \end{eqnarray}
1176: since the synaptic connection $T_{ij}$ before acting
1177: the nonlinear function of Eq.(\ref{5eq:nonlinear.T})
1178: obeys the Gaussian distribution $N(0,1)$.
1179: %の結合率$c$をもつことになる．
1180: %ここで，$Dz=(1/\sqrt{2\pi})\exp(-z^2/2)dz$を表す．
1181: Next, $J,\tilde{J}$
1182: of Eqs.(\ref{5eq:nonlinear.J}) and (\ref{5eq:nonlinear.tildeJ}) become
1183: %また，式(\ref{5eq:nonlinear.J})，式(\ref{5eq:nonlinear.tildeJ})の
1184: %$J,\tilde{J}$は次のようになる．
1185: \begin{eqnarray}
1186: J &=&
1187: 2 \int_t^\infty Dz \; z \sgn(z)
1188: = \sqrt{\frac 2{\pi}} \exp(-\frac {t^2}2) \nonumber \\
1189: & \to & t c, \; t \to \infty ,
1190: \label{5eq:clip.J} \\
1191: {\tilde J}^2 &=&
1192: 2 \int_t^\infty Dz
1193: =1-\erf (\frac{t}{\sqrt 2})
1194: =c.
1195: \label{5eq:clip.tildeJ}
1196: \end{eqnarray}
1197: Hence,
1198: the equivalent multiplicative synaptic noise $\Delta_M^2$ is obtained as,
1199: %これより，
1200: %等価な乗法的ノイズ$\Delta_M^2$は，
1201: \begin{equation}
1202: \Delta_M^2
1203: = \frac{{\tilde J}^2}{J^2}-1
1204: \to \frac 1{t^2c}, \; \; t \to \infty .
1205: \label{5eq:clip.deltaM}
1206: \end{equation}
1207: %となる．
1208: The relationship of the pruning range $t$ and the connecting rate $c$
1209: %ここで，非線形関数において刈り込む範囲を表す$t$と
1210: %結合率$c$の関係は式(\ref{5eq:clip.connect})の両辺の対数をとると
1211: %$t \to \infty$の極限では，
1212: \begin{equation}
1213: t^2 = -2 \log c,
1214: \label{5eq:t.to.c}
1215: \end{equation}
1216: is obtained by taking the logarithm of Eq.(\ref{5eq:clip.connect})
1217: at $t \to \infty$ limit.
1218: %となる．
1219: Therefore, at the limit where the equivalent connecting rate $c$ is extremely small,
1220: storage capacity $\alpha_c$ can be obtained
1221: %これより，刈込変換による結合率$c$が小さい極限での記憶容量$\alpha_c$は
1222: %式(\ref{5eq:mul.asymptote})，
1223: %式(\ref{5eq:clip.deltaM})及び
1224: %式(\ref{5eq:t.to.c})より，
1225: \begin{equation}
1226: \alpha_c = - \frac 4{\pi} c \log c,
1227: \label{5eq:clip.asymptote}
1228: \end{equation}
1229: through Eqs.(\ref{5eq:mul.asymptote}), (\ref{5eq:clip.deltaM}),
1230: and (\ref{5eq:t.to.c}).
1231: %と求められる．
1232: The synapse efficiency becomes
1233: %シナプス効率は，
1234: \begin{equation}
1235: S_{eff}=\frac{\alpha_c}{c} = - \frac 4{\pi} \log c.
1236: \end{equation}
1237: %となる．
1238: Figure \ref{5fig:clip.ac-delta.f=0.5} shows
1239: the results from the SCSNA and the asymptote at the firing rate $f=0.5$.
1240: %図\ref{5fig:clip.ac-delta.f=0.5}は，
1241: %式(\ref{5eq:clip.asymptote})の発火率$f=0.5$での
1242: %漸近形及びSCSNAによる結果を示す．
1243: Figure \ref{5fig:clip.ac-delta.all-f} shows
1244: the results from the SCSNA at various firing rates.
1245: It can be confirmed that the order of the asymptote $O(\log c)$
1246: does not depend on the firing rate
1247: from Fig.\ref{5fig:clip.ac-delta.all-f}.
1248: %また，図\ref{5fig:clip.ac-delta.all-f}に
1249: %いろいろな発火率でのSCSNAの結果を示す．
1250: %It can be confirmed that the order of asymptote
1251: %does not depend on firing rate
1252: %as shown in the Fig.\ref{5fig:clip.ac-delta.all-f}.
1253: %図\ref{5fig:clip.ac-delta.all-f}から
1254: %漸近形のオーダは発火率に依存しないことがわかる．
1255:
1256:
1257:
1258: \begin{figure}
1259: \epsfig{fig9.eps}{8cm}
1260: \caption[]
1261: %刈込変換でのシナプス効率の結合率依存性
1262: {
1263: Dependence of the synapse efficiency $S_{eff}$ with the clipped synapse
1264: on the connecting rate $c$ at $f=0.5$.
1265: %刈込変換でのシナプス効率$S_{eff}$の結合率$c$依存性(発火率$f=0.5$)．
1266: Comparison of the results from the SCSNA and asymptote.
1267: %SCSNAと漸近形の比較．
1268: }
1269: \label{5fig:clip.ac-delta.f=0.5}
1270: \end{figure}
1271:
1272:
1273: \begin{figure}
1274: \epsfig{fig10.eps}{8cm}
1275: \caption[]
1276: %刈込変換でのシナプス効率の結合率依存性(発火率による変化)
1277: {
1278: Dependence of the synapse efficiency $S_{eff}$
1279: with the clipped synapse on the firing rate $f$.
1280: It can be confirmed that the order of the asymptote
1281: does not depend on the firing rate.
1282: %刈込変換でのシナプス効率$S_{eff}$の結合率$c$依存性．
1283: %It can be confirmed that
1284: %the order of asymptote does not depend on firing rate.
1285: %漸近形のオーダは発火率に依存しないことが確認できる．
1286: }
1287: \label{5fig:clip.ac-delta.all-f}
1288: \end{figure}
1289:
1290:
1291:
1292:
1293:
1294:
1295:
1296:
1297: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
1298: %%
1299: %% 最小値切断
1300: %%
1301:
1302:
1303: \subsubsection{Minimal value deletion}
1304: %最小値切断
1305: \label{sec.sparse_noise.result.systematic.minimal}
1306:
1307: In a similar way,
1308: the equivalent multiplicative synaptic noise $\Delta_M^2$
1309: of the systematic deletion of Eq.(\ref{5eq:nonlinear.minimal})
1310: is obtained as follows,
1311: %これより，
1312: %等価な乗法的ノイズ$\Delta_M^2$は，
1313: \begin{equation}
1314: \Delta_M^2
1315: = \frac{{\tilde J}^2}{J^2}-1
1316: \to \frac 1{t^2c}, \; \; t \to \infty .
1317: \label{5eq:minimal.deltaM}
1318: \end{equation}
1319: %となる．
1320: where
1321: the connecting rate $c$
1322: and $J,\tilde{J}$
1323: of Eqs.(\ref{5eq:nonlinear.J}) and (\ref{5eq:nonlinear.tildeJ}) are
1324: %$J,\tilde{J}$は次のようになる．
1325: %式（\ref｛5eq：nonlinear.clip）の非線形関数$f_2(z,t)$によって等価的に，
1326: \begin{eqnarray}
1327: c
1328: &=&
1329: \int_{ \{ z|f_2(z,t) \ne 0 \} } Dz
1330: =
1331: 1-\erf (\frac{t}{\sqrt 2}) \nonumber \\
1332: & \to &
1333: \sqrt{\frac {2}{\pi}} t^{-1} \exp (-\frac {t^2}2), \; \; t \to \infty,
1334: \label{5eq:minimal.connect} \\
1335: J &=&
1336: \sqrt{\frac 2{\pi}} t \exp(-\frac {t^2}2)
1337: +1-\erf (\frac{t}{\sqrt 2}) \nonumber \\
1338: & \to & \sqrt{\frac 2{\pi}} t \exp(-\frac {t^2}2), \; \; t \to \infty,
1339: \label{5eq:minimal.J} \\
1340: {\tilde J}^2 &=& J,
1341: \label{5eq:minimal.tildeJ}
1342: \end{eqnarray}
1343: respectively.
1344: Hence,
1345: at the limit where
1346: the equivalent connecting rate $c$ is extremely small,
1347: storage capacity $\alpha_c$
1348: and the synapse efficiency $S_{eff}$ can be obtained
1349: through Eqs.(\ref{5eq:mul.asymptote})，
1350: (\ref{5eq:minimal.deltaM}), and (\ref{5eq:t.to.c})
1351: as follows,
1352: %これより，最小値切断による結合率$c$が小さい極限での記憶容量$\alpha_c$は
1353: %式(\ref{5eq:mul.asymptote})，
1354: %式(\ref{5eq:minimal.deltaM})及び
1355: %式(\ref{5eq:t.to.c})より，
1356: \begin{eqnarray}
1357: \alpha_c &=& -\frac 4{\pi} c \log c, \label{5eq:minimal.asymptote} \\
1358: S_{eff} &=& - \frac 4 {\pi} \log c,
1359: \end{eqnarray}
1360: respectively.
1361: %となる．
1362: Figure \ref{5fig:minimal.ac-delta.f=0.5}
1363: shows the results from the SCSNA
1364: and the asymptote at the firing rate $f=0.5$.
1365: %図\ref{5fig:minimal.ac-delta.f=0.5}は，
1366: %式(\ref{5eq:minimal.asymptote})の発火率$f=0.5$での
1367: %漸近形及びSCSNAによる結果を示す．
1368: Figure \ref{5fig:minimal.ac-delta.all-f} shows
1369: the results from the SCSNA at various firing rates.
1370: It can be confirmed that the order of the asymptote
1371: does not depend on the firing rate
1372: from Fig.\ref{5fig:minimal.ac-delta.all-f}.
1373: %また，図\ref{5fig:minimal.ac-delta.all-f}に
1374: %いろいろな発火率でのSCSNAの結果を示す．
1375: %It can be confirmed that the order of asymptote
1376: %does not depend on firing rate
1377: %as shown in Fig.\ref{5fig:minimal.ac-delta.all-f}.
1378: %図\ref{5fig:minimal.ac-delta.all-f}から
1379: %漸近形のオーダは発火率に依存しないことがわかる．
1380:
1381:
1382:
1383: \begin{figure}
1384: \epsfig{fig11.eps}{8cm}
1385: \caption[]
1386: %最小値切断でのシナプス効率の結合率依存性
1387: {
1388: Dependence of the synapse efficiency $S_{eff}$
1389: with the minimal value deletion on the connecting rate $c$ at time $f=0.5$.
1390: %最小値切断でのシナプス効率$S_{eff}$の結合率$c$依存性(発火率$f=0.5$)．
1391: Comparison of the results from the SCSNA and the asymptote.
1392: %SCSNAと漸近形の比較．
1393: }
1394: \label{5fig:minimal.ac-delta.f=0.5}
1395: \end{figure}
1396:
1397:
1398: \begin{figure}
1399: \epsfig{fig12.eps}{8cm}
1400: \caption[]
1401: %最小値切断でのシナプス効率の結合率依存性(発火率による変化)
1402: {
1403: Dependence of the synapse efficiency $S_{eff}$
1404: with the minimal value deletion on the firing rate $f$.
1405: It can be confirming that the order of the asymptote
1406: does not depend on the firing rate.
1407: %最小値切断でのシナプス効率$S_{eff}$の結合率$c$依存性．
1408: %漸近形のオーダは発火率に依存しないことが確認できる．
1409: }
1410: \label{5fig:minimal.ac-delta.all-f}
1411: \end{figure}
1412:
1413:
1414:
1415:
1416:
1417:
1418:
1419: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
1420: %%
1421: %% 圧縮切断
1422: %%
1423:
1424:
1425: \subsubsection{Compressed deletion}
1426: %圧縮切断
1427: \label{sec.sparse_noise.result.systematic.compress}
1428:
1429:
1430:
1431:
1432:
1433:
1434:
1435:
1436: Again, in the similar way,
1437:
1438: The equivalent multiplicative synaptic noise $\Delta_M^2$
1439: of the systematic deletion of Eq.(\ref{5eq:nonlinear.compress})
1440: is given by
1441: \begin{equation}
1442: \Delta_M^2
1443: = \frac{{\tilde J}^2}{J^2}-1
1444: \to \frac 2{t^2c}, \; \; t \to \infty,
1445: \label{5eq:compress.deltaM}
1446: \end{equation}
1447: %圧縮切断においても先と同様に，
1448: %式(\ref{5eq:nonlinear.clip})の非線形関数$f_3(z,t)$によって等価的に，
1449: %の結合率$c$をもつ．
1450: where the connecting rate $c$ and
1451: $J,\tilde{J}$ of Eqs.(\ref{5eq:nonlinear.J})
1452: and (\ref{5eq:nonlinear.tildeJ}) are
1453: %$J,\tilde{J}$は次のようになる．
1454: \begin{eqnarray}
1455: c&=&\int_{ \{ z|f_3(z,t) \ne 0 \} } Dz
1456: =1-\erf (\frac{t}{\sqrt 2}), \nonumber \\
1457: & \to &
1458: \sqrt{\frac {2}{\pi}} t^{-1} \exp (-\frac {t^2}2), \; \; t \to \infty,
1459: \label{5eq:compress.connect} \\
1460: J
1461: &=&
1462: c \label{5eq:compress.J} \\
1463: {\tilde J}^2
1464: &=&
1465: \int_{-\infty}^\infty Dz \; f_2(z,t)^2
1466: +t^2 \int_{-\infty}^\infty Dz \; f_1(z,t)^2 \nonumber \\
1467: & &
1468: \qquad -2t \int_{-\infty}^\infty Dz \; f_1(z,t)f_2(z,t) \nonumber \\
1469: &=&
1470: \biggl[ \sqrt{\frac 2\pi}t\exp (-\frac {t^2}2)
1471: + 1-\erf (\frac t{\sqrt{2}}) \biggr] \nonumber \\
1472: & &
1473: +t^2 \biggl[ 1-\erf (\frac t{\sqrt{2}}) \biggr]
1474: -2 t \biggl[ \sqrt{\frac 2\pi} \exp (-\frac {t^2}2) \biggr] \nonumber \\
1475: &\to&
1476: \frac{2c}{t^2}, \; \; t \to \infty, \label{5eq:compress.tildeJ}
1477: \end{eqnarray}
1478: respectively.
1479: Here, we use a asymptotic expansion equation of the error function
1480: %ここで，誤差積分関数$\erf(x)$は部分積分により，
1481: \begin{eqnarray}
1482: \erf (x) = 1 - \frac{e^{-x^2}}{\sqrt{\pi}}
1483: \left(\frac 1x- \frac 1{2x^3} + \frac 3{4x^5} - O(x^{-7}) \right),
1484: \end{eqnarray}
1485: for $x \gg 1$.
1486: %となる．
1487: %式(\ref{5eq:compress.tildeJ.exact})において，
1488: %$J$と${\tilde J}^2$の近似のオーダを合わせるため，
1489: %第1項は係数が1なので$1/t^3$まで，
1490: %第2項は係数が$t^2$なので$1/t^5$まで誤差積分関数を近似している．
1491: In order for the first term and the second term
1492: of Eq.(\ref{5eq:compress.tildeJ}) to be same order,
1493: the asymptotic expansion equation has taken
1494: the approximation of $O(t^{-3})$ and $O(t^{-5})$ respectively.
1495: The equivalent multiplicative synaptic noise
1496: in the case of systematic deletion becomes double
1497: that of the clipped synapse of Eq.(\ref{5eq:clip.deltaM})
1498: and the minimal value deletion of Eq.(\ref{5eq:compress.deltaM}).
1499: %圧縮切断では，
1500: %式(\ref{5eq:clip.deltaM})，式(\ref{5eq:compress.deltaM})の
1501: %刈込変換及び最小値切断に較べて2倍の乗法的ノイズになることがわかる．
1502: Therefore,
1503: at the limit where the equivalent connecting rate $c$ is extremely small,
1504: storage capacity $\alpha_c$ and the synapse efficiency $S_{eff}$
1505: can be obtained
1506: through Eqs.(\ref{5eq:mul.asymptote})，
1507: (\ref{5eq:compress.deltaM}), and (\ref{5eq:t.to.c})
1508: as follows,
1509: %これより，圧縮切断による結合率$c$が小さい極限での記憶容量$\alpha_c$は
1510: %式(\ref{5eq:mul.asymptote})，
1511: %式(\ref{5eq:compress.deltaM})及び
1512: %式(\ref{5eq:t.to.c})より，
1513: \begin{eqnarray}
1514: \alpha_c &=& - \frac 2{\pi} c \log c,
1515: \label{5eq:compress.asymptote} \\
1516: S_{eff} &=& - \frac 2{\pi} \log c,
1517: \end{eqnarray}
1518: respectively.
1519: %となる．
1520: Figure \ref{5fig:compress.ac-delta.f=0.5} shows
1521: the results from the SCSNA and the asymptote at the firing rate $f=0.5$.
1522: %図\ref{5fig:compress.ac-delta.f=0.5}は，
1523: %式(\ref{5eq:compress.asymptote})の発火率$f=0.5$での
1524: %漸近形及びSCSNAによる結果を示す．
1525: Figure \ref{5fig:compress.ac-delta.all-f} shows
1526: the results by the SCSNA at various firing rates.
1527: It can be confirmed that the order of the asymptote $O(\log c)$
1528: does not depend on the firing rate
1529: from Fig.\ref{5fig:compress.ac-delta.all-f}.
1530: %また，図\ref{5fig:compress.ac-delta.all-f}に
1531: %いろいろな発火率でのSCSNAの結果を示す．
1532: %図\ref{5fig:compress.ac-delta.all-f}から
1533: %It can be confirmed that
1534: %the order of asymptote does not depend
1535: %on firing rate as shown in the Fig.\ref{5fig:compress.ac-delta.all-f}.
1536: %漸近形のオーダは発火率に依存しないことがわかる．
1537:
1538:
1539:
1540: \begin{figure}
1541: \epsfig{fig13.eps}{8cm}
1542: \caption[]
1543: %圧縮切断でのシナプス効率の結合率依存性
1544: {
1545: Dependence of the synapse efficiency $S_{eff}$ with the compressed deletion
1546: on the connecting rate $c$ at $f=0.5$.
1547: %圧縮切断でのシナプス効率$S_{eff}$の結合率$c$依存性(発火率$f=0.5$)．
1548: Comparison of the results from the SCSNA and the asymptote.
1549: %SCSNAと漸近形の比較．
1550: }
1551: \label{5fig:compress.ac-delta.f=0.5}
1552: \end{figure}
1553:
1554:
1555:
1556: \begin{figure}
1557: \epsfig{fig14.eps}{8cm}
1558: \caption[]
1559: %圧縮切断のシナプス効率の結合率依存性
1560: {
1561: Dependence of the synapse efficiency $S_{eff}$ with the compressed deletion
1562: on the firing rate $f$.
1563: It can be confirming that the order of the asymptote
1564: does not depend on the firing rate.
1565: %圧縮切断でのシナプス効率$S_{eff}$の結合率$c$依存性．
1566: %It can be confirmed that
1567: %the order of asymptote does not depend on firing rate.
1568: %漸近形のオーダは発火率に依存しないことが確認できる．
1569: }
1570: \label{5fig:compress.ac-delta.all-f}
1571: \end{figure}
1572:
1573:
1574:
1575:
1576:
1577:
1578: Figure \ref{5fig:seff.conclusion} shows
1579: the dependence of the synapse efficiency $S_{eff}$
1580: on the connecting rate $c$ obtained by means of the SCSNA.
1581: %図\ref{5fig:seff.conclusion}に
1582: %SCSNAから得られる無作為切断と系統的切断について，
1583: %シナプス効率$S_{eff}$の結合率$c$依存性を示す．
1584: Table \ref{5table:asymtotics} shows
1585: the asymptote of storage capacity
1586: with the random deletion and the systematic deletion.
1587: %また，無作為切断と系統的切断について
1588: %記憶容量の漸近形を表\ref{5table:asymtotics}に示す．
1589: %また表\ref{5table:asymtotics.seff}にシナプス効率の漸近形を示す．
1590: Hence,
1591: when using minimal value deletion as the simplest from of systematic deletion
1592: we found that the synapse efficiency
1593: in the case of systematic deletion becomes
1594: %これより，
1595: %切断しないシナプスについて変換を行わないという点において
1596: %最も単純な最小値切断を系統的切断とした場合，
1597: %無作為切断に対する系統的切断のシナプス効率比は，
1598: \begin{equation}
1599: \frac{S_{eff}({\rm systematic \; deletion})}
1600: {S_{eff}({\rm random \; deletion})}
1601: =\frac{\frac 4{\pi}| \log c|}{\frac 2{\pi}}=2|\log c| ,
1602: \end{equation}
1603: thus
1604: we have shown analytically that
1605: the synapse efficiency in the case of systematic deletion
1606: diverges as $O(|\log c|)$
1607: at the limit where the connecting rate $c$ is extremely small,
1608: and have shown that the synapse efficiency
1609: in the case of the systematic deletion becomes $2 |\log c|$ times
1610: as large as that of the random deletion.
1611: %と$2|\log c|$となることが解析的に求められる．
1612:
1613:
1614:
1615: \begin{table}
1616: \caption{
1617: Asymptote of storage capacity
1618: by the random deletion and by the systematic deletion
1619: at the firing rate $f=0.5$.
1620: %無作為切断と系統的切断による記憶容量の漸近形($f=0.5$)．
1621: }
1622: \label{5table:asymtotics}
1623: \begin{center}
1624: \begin{tabular}{ll|c}
1625: \hline
1626: Types of deletion & & Storage capacity \\
1627: & & (Asymptote) \\
1628: \hline
1629: random deletion & & $(2/\pi) c$ \\
1630: systematic deletion & clipped synapse & $(4/\pi) c| \log c| $\\
1631: & minimal value deletion & $(4/\pi) c| \log c|$ \\
1632: & compressed deletion & $(2/\pi) c| \log c|$ \\
1633: \hline
1634: \end{tabular}
1635: \end{center}
1636: \end{table}
1637:
1638:
1639:
1640: \begin{table}
1641: \caption{
1642: Asymptote of the synapse efficiency
1643: by the random deletion and the systematic deletion
1644: at the firing rate $f=0.5$.
1645: %無作為切断と系統的切断によるシナプス効率の漸近形($f=0.5$)．
1646: }
1647: \label{5table:asymtotics.seff}
1648: \begin{center}
1649: \begin{tabular}{ll|c}
1650: \hline
1651: Types of deletion & & Synapse efficiency \\
1652: & & (Asymptote) \\
1653: \hline
1654: random deletion & & $(2/\pi)$ \\
1655: systematic deletion & clipped synapse & $(4/\pi) | \log c|$ \\
1656: & minimal value deletion & $(4/\pi) | \log c|$ \\
1657: & compressed deletion & $(2/\pi) | \log c|$ \\
1658: \hline
1659: \end{tabular}
1660: \end{center}
1661: \end{table}
1662:
1663:
1664:
1665:
1666: \begin{figure}
1667: \epsfig{fig15.eps}{8cm}
1668: \caption[]
1669: %無作為切断と系統的切断によるシナプス効率の比較
1670: {
1671: Comparison of the synapse efficiency with the random deletion
1672: and that with the systematic deletion at the firing rate $f=0.5$.
1673: %無作為切断と系統的切断によるシナプス効率の比較(SCSNAによる解析,$f=0.5$)．
1674: }
1675: \label{5fig:seff.conclusion}
1676: \end{figure}
1677:
1678:
1679:
1680:
1681:
1682:
1683:
1684:
1685:
1686:
1687:
1688: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
1689: %%
1690: %% メタボリ
1691: %%
1692:
1693: \section{The memory performance under limited metabolic energy resources}
1694:
1695: Until the previous section,
1696: we have discussed the effect of synaptic pruning
1697: by evaluating the synapse efficiency
1698: which is the memory capacity normalized by connecting rate $c$.
1699: %前章までの議論では，
1700: %記憶容量を結合率で規格化したシナプス効率$S_{eff}$によって
1701: %切断の影響を議論した．
1702: When the connecting rate is $c$,
1703: the synapse number per one neuron decreases to $cN$.
1704: %結合率が$c$のとき，
1705: %1素子当たりのシナプス数は$cN$に減少する．
1706: Therefore,
1707: the synapse efficiency means
1708: the capacity per the input of one neuron.
1709: %つまり，シナプス効率は，
1710: %1素子の入力で規格化した記憶容量となっている．
1711: In the discussion by the synapse efficiency,
1712: the synapse number decreases when the connecting rate is small.
1713: %シナプス効率による議論では，
1714: %結合率が小さい場合は素子数が一定なのでシナプス数も少なくなる．
1715: \par
1716: In addition to such the fixed neuron number criterion,
1717: a fixed synapse number criterion could be discussed.
1718: %このような素子数一定とした議論の他に，
1719: %シナプス数一定とする議論も考えられる．
1720: At the synaptic growth stage,
1721: it is natural to assume
1722: that metabolic energy resources are restricted.
1723: %シナプスの形成段階において，
1724: %代謝資源は無尽蔵ではなく制限があると考えられる．
1725: When metabolic energy resources are limited,
1726: it is also important
1727: that the effect of synaptic pruning is discussed
1728: under limited synapse number.
1729: %代謝資源が一定であるという制限を考慮すれば，
1730: %形成されるシナプス数を一定とした基準で
1731: %シナプス切断の影響を議論することが重要である．
1732: Chechik et al. discussed
1733: the memorized pattern number per one synapse
1734: under a fixed synapse number criterion \cite{Chechik1998}．
1735: %Chechikらは，シナプス数一定の条件の下で，
1736: %１シナプス当たりの記憶パターン数を議論した．
1737: They pointed out
1738: the existence of an optimal connecting rate
1739: under the fixed synapse number criterion
1740: and suggested an explanation of synaptic pruning as follows:
1741: synaptic pruning following over-growth can improve
1742: performance of a network with limited synaptic resources.
1743: %Chechikらはシナプス数を一定とした場合に，
1744: %結合率には最適値が存在することを指摘し，
1745: %シナプス過剰生成後の刈込みは
1746: %シナプス資源が限られている場合に
1747: %ネットワークの性能を改善する方法であるという
1748: %新しい解釈をしめした
1749: %\cite{Chechik1998}．
1750: \par
1751: The synapse number is $N^2$ in the full connected case $c=1$.
1752: %切断がないときのシナプス数は$N^2$である．
1753: We consider the larger network with $M(>N)$ neurons.
1754: %ここで，素子数$M(>N)$の
1755: %より規模の大きいネットワークを考える．
1756: The synapse number in the lager networks
1757: with the connecting rate $c$ becomes $cM^2$.
1758: %素子数$M$のネットワークで，結合率を$c$とすると
1759: %シナプス数は$cM^2$となる．
1760: We can introduce the fixed synapse number criterion
1761: by considering a larger network
1762: which has $M$ neurons, i.e.,
1763: %結合率を$c$とするときシナプス数が，
1764: \begin{equation}
1765: N^2=cM^2,
1766: \end{equation}
1767: synapses at the connecting rate $c$.
1768: %となるような素子数$M$のネットワークを考えることにより，
1769: %シナプス数一定の場合を議論することができる．
1770: %いま，記憶パターンの臨界値$p_c$は，
1771: %\begin{equation}
1772: %p_c=\alpha_c N.
1773: %\end{equation}
1774: %である．
1775: The memorized pattern number per one synapse becomes
1776: %シナプス数を一定としたとき，
1777: %結合率$c$，素子数$M$のネットワークについて
1778: %1シナプス当たりの記憶パターン数を求めると，
1779: \begin{equation}
1780: \frac{p_c}{cM^2}
1781: = \frac {p_c}{N^2}
1782: = \frac{\alpha_c}N
1783: = \frac{\alpha_c}{\sqrt{c}M},
1784: \end{equation}
1785: %となる．
1786: where
1787: the critical memorized pattern number $p_c$ is
1788: \begin{equation}
1789: p_c=\alpha_c N.
1790: \end{equation}
1791: We define the coefficient $\alpha_c/\sqrt{c}$ as memory performance.
1792: %係数$\alpha_c/\sqrt{c}$を記憶性能という．
1793: We discuss the effect of synaptic pruning by the memory performance.
1794: %この記憶性能により切断の影響を議論する．
1795: Under limited metabolic energy resources,
1796: the optimal strategy is the maximization of the memory performance.
1797: %代謝資源一定のときに最善の選択はこの記憶性能を最大にすることである．
1798: Chechik et al. showed
1799: the existence of the optimal connecting rate
1800: which maximize memory performance \cite{Chechik1998}.
1801: %Chechikらは記憶性能を最大にする最適な切断率が存在することを示した．
1802: The memory performance can be calculated by
1803: normalising
1804: the capacity, which is given by solving the order-parameter equations,
1805: with $\sqrt{c}$.
1806: %鞍点方程式を解いて得られる記憶容量$\alpha_c$を
1807: %$\sqrt{c}$で規格化することにより記憶性能が求められる．
1808: Figure \ref{fig:metaboric} shows
1809: the dependence of the memory performance on the connecting rate
1810: in three types of pruning.
1811: %図\ref{fig:metaboric}に3種類の切断による記憶性能の結合率依存性を示す．
1812: It is confirmed
1813: that there are the optimal values
1814: which maximized the memory performance by each deletions.
1815: %図\ref{fig:metaboric}より結合率が小さい領域で，
1816: %この係数が最大となるような最適な結合率が存在することがわかる．
1817: The optimal connecting rates
1818: of clipped synapse, minimal value deletion, and compressed deletion
1819: are $c=0.036,0.038,0.084$, respectively.
1820: %クリップシナプス，最小値切断，圧縮切断の切断において，
1821: %それぞれ，結合率が，$c=0.036,0.038,0.084$ のとき
1822: %ネットワークの記憶性能は最大になる．
1823: This interesting fact may imply that
1824: the memory performance is improved without heavy pruning.
1825: %これは，極端な刈込みをすることなく，
1826: %記憶性能が改善されることを表しており興味深い．
1827: These optimal values agree with the computer simulation results
1828: given by Chechik et al \cite{Chechik1998}.
1829: %この最適値は，
1830: %Chechikらが示した最適な結合率とほぼ一致する\cite{Chechik1998}．
1831:
1832:
1833:
1834: \begin{figure}[h]
1835: \begin{center}
1836: \includegraphics[width=0.8 \linewidth,keepaspectratio]{fig16.eps}
1837: \caption{Memory performance $\alpha_c/\sqrt{c}$
1838: of networks with different number of neurons
1839: but the same total number of synapses
1840: as a function of the connecting rate $c$
1841: in the case of clipped synapse (the solid line),
1842: minimal value deletion (the dotted line),
1843: and compressed deletion (the dashed line).}
1844: \label{fig:metaboric}
1845: \end{center}
1846: \end{figure}
1847:
1848:
1849:
1850:
1851:
1852:
1853: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
1854: %%
1855: %% まとめ
1856: %%
1857:
1858:
1859: \section{Conclusion}
1860: \label{sec.sparse_noise.conclusion}
1861:
1862:
1863:
1864: We have analytically discussed the synapse efficiency,
1865: which we regarded as the auto-correlation-type associative memory,
1866: to evaluate the effect of the pruning following over-growth.
1867: %シナプスの過剰な生成とそれに続く刈り込みを
1868: %自己相関型連想記憶モデルの枠組の中で議論した．
1869: Although Chechik et al. pointed out
1870: that the synapse efficiency is increased by the systematic deletion,
1871: this is qualitatively obvious
1872: and the increase in the synapse efficiency
1873: should also be discussed quantitatively.
1874: %Chechikらは，
1875: %系統的な刈り込みによりシナプス効率が大きくなることを指摘した．
1876: %However, since it is qualitatively obvious
1877: %that synapse efficiency increases by such a deletion,
1878: %increase of synapse efficiency should be discussed quantitatively.
1879: %しかし，このような系統的な刈り込みは
1880: %シナプス効率を上昇させることは定性的には明らかであるため，
1881: %シナプス効率がどの程度上昇するかという定量的な議論が必要であった．
1882: At the limit where the multiplicative synaptic noise is extremely large,
1883: storage capacity $\alpha_c$ is inversely
1884: as the variance of the multiplicative synaptic noise $\Delta_M^2$.
1885: %乗法的ノイズが十分大きな極限では$\alpha_c \propto 1 / \Delta_M^2$と，
1886: %記憶容量は乗法的ノイズの分散に反比例することがわかった．
1887: From this result,
1888: we analytically obtained that
1889: the synapse efficiency in the case of the systematic deletion
1890: diverges as $O(|\log c|)$
1891: at the limit where the connecting rate $c$ is extremely small.
1892: %さらにこの関係から，
1893: %結合率$c$が十分小さい極限で系統的切断とランダム切断のシナプス効率比は
1894: %定量的に$2 | \log_e c|$であることが解析的に求められた．
1895:
1896: On the other hand, it is natural to assume
1897: that metabolic energy resources are restricted
1898: at the synaptic growth stage.
1899: %シナプスの形成段階において，
1900: %代謝資源は無尽蔵ではなく制限があると考えられる．
1901: When metabolic energy resources are limited,
1902: i.e., synapse number is limited,
1903: the optimal connecting rate, which maximize memory performance, exists.
1904: These optimal values are in agreement
1905: with the results given by Chechik et al \cite{Chechik1998}.
1906:
1907: In the correlation learning, which can be considered a prototype
1908: of any other learning rules,
1909: various properties can be analyzed quantitatively.
1910: %全ての学習則のプロトタイプと考えられる相関学習においては，
1911: %複雑に学習則を変更しても定量的な議論ができることがわかった．
1912: The asymptote of synapse efficiency in the model with another learning rule
1913: can be discussed in a similar way.
1914: %他の学習則についても同様な方法により，
1915: %シナプス効率の漸近特性等を議論できると考えられる．
1916: As our future work, we plan to further discuss
1917: these properties while taking into account various considerations
1918: regarding related physiological knowledge.
1919: \\
1920:
1921: %生理学的実験から得られた知見を取り込んで
1922: %議論を進めていくことが今後の課題である．
1923: %\\
1924: %\\
1925: %\\
1926: \begin{center}
1927: {\small \bf ACKNOWLEDGEMENTS}
1928: \end{center}
1929: %\\
1930:
1931: We thank the referee for the helpful comments,
1932: especially for the comment concerning the derivation of Appendix
1933: \ref{appendix.critical_noise}.
1934: This work was partially supported by a Grant-in-Aid
1935: for Scientific Research on Priority Areas No. 14084212,
1936: for Scientific Research (C) No. 14580438,
1937: for Encouragement of Young Scientists (B) No. 14780309 and
1938: for Encouragement of Young Scientists (B) No. 15700141
1939: from the Ministry of Education, Culture, Sports, Science
1940: and Technology of Japan.
1941:
1942:
1943:
1944:
1945:
1946:
1947:
1948:
1949:
1950:
1951:
1952:
1953:
1954:
1955:
1956:
1957:
1958:
1959:
1960:
1961:
1962:
1963:
1964:
1965: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
1966: %% APPENDIX
1967: \appendix
1968:
1969:
1970: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
1971: %% APPENDIX A
1972: \section{SCSNA for additive synaptic noise}
1973: \label{appendix.scsna}
1974:
1975: Derivations of the order-parameter
1976: Eqs.(\ref{eq:order_parameter_equation_m})
1977: -(\ref{eq:effevtive_response_function}) are given here.
1978: The SCSNA starts from the fixed-point equations
1979: for the dynamics of the $N$-neuron network
1980: shown as Eq.(\ref{5eq:sparse.equilibrium}).
1981: The random memory patterns are generated
1982: according to the probability distribution
1983: of Eq.(\ref{5eq:sparse.pattern}).
1984: The syanaptic connections are given by Eq.(\ref{5eq:add.noise.J}).
1985: The asymmetric additive synaptic noise, $\delta_{ij}$ and $\delta_{ji}$
1986: are independently generated according to the probability distribution
1987: of Eq.(\ref{5eq:add.noise.pdf}).
1988: Moreover, we can analyze a more general case,
1989: where $\delta_{ij}$ and $\delta_{ji}$ have an arbitary correlation such that
1990: \begin{equation}
1991: {\rm Cov}[\delta_{ij}, \delta_{ji}]
1992: = k_\delta \frac{J^2}{N(1-a^2)^2} \Delta_A^2, \quad
1993: -1 \leq k_\delta \leq 1.
1994: \end{equation}
1995: In this general case,
1996: the symmetric and the asymmetric additive synaptic noise
1997: correspond to $k_\delta =1$ and $k_\delta =0$, respectively.
1998: Here, we assume the probability distribution of the additive syanptic noise
1999: is normal distribution
2000: $\delta_{ij} \sim N(0, \frac{J^2}{N(1-a^2)^2} \Delta_A^2)$.
2001: However, any probability distributions, which have same average and variance,
2002: can be discussed by the central limit theorem
2003: in the similar way of the following disscussion.
2004: Defining the loading rate as $\alpha =p/N$,
2005: we can write the local field $h_i$ for neuron $i$ as
2006: \begin{eqnarray}
2007: h_i
2008: &\equiv& \sum_{j\ne i}^N J_{ij} x_j \nonumber \\
2009: &=& J \sum_{\mu=1}^{\alpha N} (\xi_i^\mu-a) m_\mu +
2010: \sum_{j \neq i}^N \delta_{ij} x_j - J \alpha x_i,
2011: \label{ap:add.h_i}
2012: \end{eqnarray}
2013: where $m_\mu$ is the overlap between the stored pattern $\vec{\xi}^\mu$
2014: and the equilibrium state $\vec{x}$ defined by
2015: \begin{equation}
2016: m_\mu = \frac{1}{N(1-a^2)} \sum_{i=1}^N (\xi^\mu _i-a) x_i.
2017: \label{eq:appA_definition_overlap}
2018: \end{equation}
2019: The second term including $x_j=F(h_j+h)$ in Eq.(\ref{ap:add.h_i})
2020: depends on $\delta_{ji}$.
2021: The $\delta_{ij}$ dependences of $x_j$ are extracted from $x_j$,
2022: \begin{equation}
2023: x_j = x^{(\delta_{ji})}_j + \delta_{ji} x_i x'{}^{(\delta_{ji})}_j,
2024: \label{ap:add.eq.x_j}
2025: \end{equation}
2026: where
2027: \begin{eqnarray}
2028: x ^{(\delta_{ji})}_j &=& F (h_j - \delta_{ji} x_i), \\
2029: x'{}^{(\delta_{ji})}_j &=& F'(h_j - \delta_{ji} x_i).
2030: \end{eqnarray}
2031: Substituting Eq.(\ref{ap:add.eq.x_j}) into Eq.(\ref{ap:add.h_i}),
2032: the local field $h_i$ can be expressed as
2033: \begin{eqnarray}
2034: h_i
2035: &=& J \sum_{\mu=1}^{\alpha N} (\xi_i^\mu-a) m_\mu +
2036: \sum_{j \neq i}^N \delta_{ij} x^{(\delta_{ji})}_j -
2037: J \alpha x_i \nonumber \\
2038: & & + x_i \sum_{j \neq i}^N \delta_{ij}
2039: \delta_{ji} x'{}^{(\delta_{ji})}_j.
2040: \label{ap:add.eq.h_i}
2041: \end{eqnarray}
2042: We assume that
2043: Eq.(\ref{ap:add.eq.h_i}) and $x_i=F(h_i+h)$ can be solved
2044: by using the effective response function $\tilde{F}(u)$ as,
2045: \begin{equation}
2046: x_i = \tilde{F} \biggl( J \sum_{\mu=1}^p (\xi_i^\mu-a) m_\mu
2047: + \sum_{j \neq i}^N \delta_{ij} x^{(\delta_{ji})}_j \biggr) .
2048: \label{ap:add.overlap.xi}
2049: \end{equation}
2050: Let $\vec{\xi}^1$ be the target pattern to be retrieved.
2051: Therefore,
2052: we can assume that $m_1=O(1)$ and $m_\mu =O(1/\sqrt{N}),\mu >1$.
2053: Then we can use the Taylor expansion to obtain
2054: \begin{eqnarray}
2055: m_\mu
2056: &=& \frac{1}{N(1-a^2)} \sum_{i=1}^N (\xi_i^\mu-a) x_i^{(\mu)}
2057: \nonumber \\
2058: & & +
2059: \frac{J}{N(1-a^2)} \sum_{i=1}^N (\xi^\mu_i - a)^2 m_\mu x'{}^{(\mu)}_i
2060: \nonumber \\
2061: &=& \frac{1}{N(1-a^2)} \sum_{i=1}^N
2062: (\xi_i^\mu-a) x_i^{(\mu)} + J U m_\mu
2063: \nonumber \\
2064: &=& \frac{1}{N(1-a^2)(1-J U)}
2065: \sum_{i=1}^N (\xi_i^\mu-a) x_i^{(\mu)},
2066: \label{ap:add.mu}
2067: \end{eqnarray}
2068: by substituting Eq.(\ref{ap:add.overlap.xi})
2069: into the overlap defined by Eq.(\ref{eq:appA_definition_overlap}),
2070: where
2071: \begin{eqnarray}
2072: x^{(\mu)}_i &=&
2073: \tilde{F} \biggl( J \sum_{\nu \neq \mu}^{\alpha N} (\xi_i^\nu-a) m_\nu +
2074: \sum_{j \neq i}^N \delta_{ij} x^{(\delta_{ji})}_j \biggr), \\
2075: x'{}^{(\mu)}_i &=&
2076: \tilde{F}' \biggl( J \sum_{\nu \neq \mu}^{\alpha N} (\xi_i^\nu-a) m_\nu +
2077: \sum_{j \neq i}^N \delta_{ij} x^{(\delta_{ji})}_j \biggr), \\
2078: U &=& \frac{1}{N} \sum_{i=1}^N x'{}^{(\mu)}_i.
2079: \label{eq:appA_additive_U}
2080: \end{eqnarray}
2081: Equations (\ref{ap:add.eq.h_i}) and (\ref{ap:add.mu}) give
2082: the following expression for the local field:
2083: \begin{eqnarray}
2084: h_i
2085: &=& J (\xi^1_i-a) m_1 \nonumber \\
2086: & & +
2087: \alpha \left[\frac{J^2}{1-JU}
2088: +k_\delta \frac{J^2}{(1-a^2)^2} \Delta_A^2 \right] U x_i \nonumber \\
2089: & & + \frac{J}{N(1-a^2)(1-JU)} \sum_{j \neq i}^N \sum_{\mu = 2}^{\alpha N}
2090: (\xi^\mu_i-a) (\xi^\mu_j-a) x_j^{(\mu)} \nonumber \\
2091: & & +
2092: \sum_{j \neq i}^N \delta_{ij} x^{(\delta_{ji})}_j.
2093: \label{ap:add.eq.h_i_final}
2094: \end{eqnarray}
2095: Note that the second term in Eq.(\ref{ap:add.eq.h_i_final})
2096: denotes the effective self-coupling term.
2097: The third and the last terms are
2098: summations of uncorrelated random variables, with mean 0 and variance,
2099: \begin{eqnarray}
2100: & &\frac{J^2}{N^2(1-a^2)^2(1-JU)^2} \nonumber \\
2101: & &\quad \times \sum_{j \neq i}^N \sum_{\mu = 2}^{\alpha N}
2102: (\xi_i^\mu-a)^2 (\xi_j^\mu-a)^2 (x_j^{(\mu)})^2 \nonumber \\
2103: & & \qquad \qquad = \frac{\alpha J^2}{(1-JU)^2} q,
2104: \label{ap:add.eq.sigma1}
2105: \end{eqnarray}
2106: \begin{equation}
2107: \sum_{j \neq i}^N \delta_{ij}^2 (x_j^{(\delta_{ji})})^2
2108: = \frac {J^2}{(1-a^2)^2}\Delta_A^2 q,
2109: \label{ap:add.eq.sigma2}
2110: \end{equation}
2111: respectively.
2112: The cross term of these terms have vanished.
2113: Thus, we finally obtain
2114: \begin{eqnarray}
2115: h_i &=& J (\xi^1_i-a) m_1 + \sigma z_i \nonumber \\
2116: & &+ \biggl[ \frac{\alpha J^2}{1-JU} +
2117: k_\delta \frac {J^2}{(1-a^2)^2} \Delta_A^2 \biggr] U x_i \\
2118: \sigma^2 &=& \frac{\alpha J^2 q}{(1-JU)^2} +
2119: \frac {J^2}{(1-a^2)^2}\Delta_A^2 q,
2120: \label{ap:sigma.additive_noise}
2121: \end{eqnarray}
2122: from Eqs.(\ref{ap:add.eq.sigma1}) and (\ref{ap:add.eq.sigma2}),
2123: where $z_i \sim N(0,1)$.
2124: Equation (\ref{eq:order_parameter_equation_sigma}) is
2125: given by Eq.(\ref{ap:sigma.additive_noise}).
2126: Finally,
2127: after rewriting $\xi^1_i \to \xi$, $m_1 \to m$, $z_i \to z$,
2128: and $x_i \to Y(z;\xi )$,
2129: the results of the SCSNA for the additive synaptic noise
2130: are summarized by the order-parameter equations
2131: of Eqs.(\ref{eq:order_parameter_equation_m})
2132: -(\ref{eq:order_parameter_equation_U}) as,
2133: \begin{eqnarray}
2134: m &=& \frac{1}{1-a^2} \int Dz < (\xi -a) Y(z;\xi ) >_\xi ,\nonumber \\
2135: q &=& \int Dz < Y(z;\xi )^2 >_\xi ,\nonumber \\
2136: U &=& \frac{1}{\sigma} \int Dz \; z < Y(z;\xi ) >_\xi , \nonumber
2137: \end{eqnarray}
2138: where the effective response function $Y(z;\xi )$ becomes
2139: \begin{eqnarray}
2140: Y(z;\xi )
2141: &=& F \biggl( J (\xi-a) m + \sigma z + h \nonumber \\
2142: & & + \biggl[ \frac{\alpha J^2}{1-JU} +
2143: k_\delta \frac{J^2}{(1-a^2)^2} \Delta_A^2 \biggr] U Y(z;\xi ) \biggr).
2144: \nonumber \\
2145: & &
2146: \label{ap:effective_response_function}
2147: \end{eqnarray}
2148: The effective response function of Eq.(\ref{eq:effevtive_response_function})
2149: can be obtained by substituting $k_\delta =1$
2150: into Eq.(\ref{ap:effective_response_function}).
2151:
2152:
2153:
2154:
2155:
2156:
2157:
2158:
2159:
2160:
2161:
2162:
2163:
2164:
2165:
2166: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
2167: %% APPENDIX B
2168: \section{Equivalence among three types of noise}
2169: \label{appendix.equivalent_noise}
2170:
2171:
2172:
2173:
2174: The multiplicative synaptic noise,
2175: the random synaptic deletion, and the nonlinear synapse
2176: can be discussed in the similar manner to Appendix \ref{appendix.scsna}.
2177:
2178:
2179:
2180:
2181: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
2182: %% B.1
2183: \subsection{Multiplicative synaptic noise}
2184: \label{appendix.multiplicative_noise}
2185:
2186: Derivations of the equivalent noise
2187: Eq.(\ref{5eq:add.to.mul}) is given here.
2188: We can also analyze
2189: by a similar manner to the analysis of the additive synaptic noise.
2190: The syanaptic connections are given by Eq.(\ref{5eq:mul.noise.J}).
2191: The asymmetric multiplicative synaptic noise,
2192: $\varepsilon_{ij}$ and $\varepsilon_{ji}$ are independently generated
2193: according to the probability distribution of Eq.(\ref{5eq:mul.noise.pdf}).
2194: We analyze a more general case,
2195: where $\delta_{ij}$ and $\delta_{ji}$ have an arbitary correlation such that
2196: \begin{equation}
2197: \mbox{Cov}[\varepsilon_{ij}, \varepsilon_{ji}]
2198: = k_\varepsilon \Delta_M^2, \quad
2199: -1 \leq k_\varepsilon \leq 1.
2200: \end{equation}
2201: In this general case, the symmetric and the asymmetric multiplicative synaptic noise
2202: correspond to $k_\varepsilon =1$ and $k_\varepsilon =0$, respectively.
2203: Here, we assume the probability distribution of the multiplicative synaptic noise
2204: is normal distribution
2205: $\varepsilon_{ij} \sim N(0, \Delta_M^2)$.
2206: The local field $h_i$ for neuron $i$ becomes
2207: \begin{eqnarray}
2208: h_i &=& \sum_{\mu=1}^{\alpha N} (\xi_i^\mu-a) m_\mu \nonumber \\
2209: & & + \frac{1}{N(1-a^2)} \sum_{\mu = 1}^{\alpha N} \sum_{j \neq i}^N
2210: \varepsilon_{ij} (\xi_i^\mu-a) (\xi_j^\mu-a) x_j \nonumber \\
2211: & & - \alpha x_i,
2212: \label{ap:mul.h_i}
2213: \end{eqnarray}
2214: where $m_\mu$ is the overlap
2215: defined by Eq.(\ref{eq:appA_definition_overlap}).
2216: The second term including $x_j=F(h_j+h)$ in Eq.(\ref{ap:mul.h_i})
2217: depends on $\varepsilon_{ji}$.
2218: The $\varepsilon_{ij}$ dependences and $\xi_j^\mu$ dependences of $x_j$
2219: are extracted from $x_j$,
2220: \begin{eqnarray}
2221: x_j &=& x^{(\mu)(\varepsilon_{ji})}_j
2222: + h_j^{\{ \mu,\varepsilon_{ji} \}} x'{}^{(\mu)(\varepsilon_{ji})}_j,
2223: \label{ap:mul.eq.x_j} \\
2224: h_j^{\{ \mu,\varepsilon_{ji} \}}
2225: &=& (\xi^\mu_j-a) m_\mu \nonumber \\
2226: & & + \frac{1}{N(1-a^2)}
2227: \sum_{k \neq j}^N \varepsilon_{jk} (\xi_j^\mu-a) (\xi_k^\mu-a) x_k
2228: \nonumber \\
2229: & &+\frac{\varepsilon_{ji}}{N(1-a^2)} \sum_{\nu \neq \mu}^{\alpha N}
2230: (\xi_j^\nu-a) (\xi_i^\nu-a) x_i \nonumber \\
2231: & &+\frac{\varepsilon_{ji}}{N(1-a^2)} (\xi_j^\mu-a) (\xi_i^\mu-a) x_i,
2232: \label{ap:mul.eq.h_j}
2233: \end{eqnarray}
2234: where
2235: \begin{eqnarray}
2236: x ^{(\mu)(\varepsilon_{ji})}_j&=&F (h_j - h_j^{\{ \mu,\varepsilon_{ji} \}}), \\
2237: x'{}^{(\mu)(\varepsilon_{ji})}_j&=&F'(h_j - h_j^{\{ \mu,\varepsilon_{ji} \}}).
2238: \end{eqnarray}
2239: We assume that
2240: Eq.(\ref{ap:mul.h_i}) and $x_i=F(h_i+h)$ can be solved
2241: by using the effective response function $\tilde{F}(u)$ as,
2242: \begin{eqnarray}
2243: & &x_i = \tilde{F}(\sum_{\mu=1}^p (\xi_i^\mu-a) m_\mu \nonumber \\
2244: & &+ \frac{1}{N(1-a^2)} \sum_{\mu = 1}^{\alpha N} \sum_{j \neq i}^N
2245: \varepsilon_{ij} (\xi_i^\mu-a) (\xi_j^\mu-a) x^{(\mu)(\varepsilon_{ji})}_j).
2246: \nonumber \\
2247: & &
2248: \label{ap:mul.overlap.xi}
2249: \end{eqnarray}
2250: Let $\vec{\xi}^1$ be the target pattern.
2251: We substitute Eq.(\ref{ap:mul.overlap.xi})
2252: into the overlap defined by Eq.(\ref{eq:appA_definition_overlap})
2253: and expand the resultant expression by $(\xi_i^\mu -a)m_\mu$ $(\mu >1)$,
2254: which has the order of $O(1/\sqrt{N})$.
2255: This leads to
2256: \begin{equation}
2257: m_\mu
2258: = \frac{1}{N(1-a^2)(1-U)} \sum_{i=1}^N (\xi_i^\mu-a) x_i^{(\mu)},
2259: \label{ap:mul.mu}
2260: \end{equation}
2261: where
2262: \begin{eqnarray}
2263: & &x^{(\mu)}_i =
2264: \tilde{F} \biggl( \sum_{\nu \neq \mu}^{\alpha N} (\xi_i^\nu-a) m_\nu
2265: \nonumber \\
2266: & & + \frac{1}{N(1-a^2)} \sum_{\nu \neq \mu}^{\alpha N} \sum_{j \neq i}^N
2267: \varepsilon_{ij} (\xi_i^\nu-a) (\xi_j^\nu-a) x^{(\nu)(\varepsilon_{ji})}_j
2268: \biggr), \nonumber \\
2269: & &
2270: \end{eqnarray}
2271: and $U$ is defined by the similar way of Eq.(\ref{eq:appA_additive_U})
2272: in the case of the additive synaptic noise.
2273: Equations (\ref{ap:mul.h_i}),(\ref{ap:mul.eq.x_j}) and (\ref{ap:mul.mu}) give
2274: \begin{eqnarray}
2275: h_i
2276: &=& (\xi^1_i-a) m_1 +
2277: \alpha \left[\frac{1}{1-U} +
2278: k_\varepsilon \Delta_M^2 \right] U x_i
2279: \nonumber \\
2280: & & + \frac{1}{N(1-a^2)(1-U)} \sum_{j \neq i}^N \sum_{\mu = 2}^{\alpha N}
2281: (\xi^\mu_i-a) (\xi^\mu_j-a) x_j^{(\mu)} \nonumber \\
2282: & & + \frac{1}{N(1-a^2)} \sum_{\mu = 1}^{\alpha N} \sum_{j \neq i}^N
2283: \varepsilon_{ij} (\xi_i^\mu-a) (\xi_j^\mu-a) x^{(\mu)(\varepsilon_{ji})}_j.
2284: \nonumber \\
2285: \label{ap:mul.eq.h_i_final}
2286: \end{eqnarray}
2287: The third and last terms can be regarded as the noise terms.
2288: The variance of the noise terms becomes
2289: \begin{equation}
2290: \sigma^2=
2291: \frac{\alpha q}{(1-U)^2} + \alpha \Delta_M^2 q.
2292: \label{ap:sigma.multiplicative_noise}
2293: \end{equation}
2294: Thus, after rewriting $\xi_i^1 \to \xi$ and $m_1 \to m$,
2295: we obtain the effective response function:
2296: \begin{eqnarray}
2297: Y(z;\xi ) &=&
2298: F \biggl( (\xi -a) m + \sigma z +h \nonumber \\
2299: & & + \alpha \left[\frac{1}{1-U} + k_\varepsilon \Delta_M^2 \right] U Y(z;\xi )
2300: \biggr). \nonumber \\
2301: \label{ap:mul.Y.final}
2302: & &
2303: \end{eqnarray}
2304: Finally,
2305: the equivalence between the multiplicative synaptic noise
2306: and the additive synaptic noise is obtained as follows,
2307: \begin{eqnarray}
2308: J &=& 1, \\
2309: \Delta_A^2 &=& \alpha (1-a^2)^2 \Delta_M^2, \label{ap:mul2add} \\
2310: k_\delta &=& k_\varepsilon,
2311: \end{eqnarray}
2312: by comparing
2313: Eqs.(\ref{ap:sigma.multiplicative_noise}) and (\ref{ap:mul.Y.final})
2314: to Eqs.(\ref{ap:sigma.additive_noise})
2315: and (\ref{ap:effective_response_function}).
2316:
2317:
2318:
2319:
2320:
2321:
2322:
2323:
2324:
2325:
2326:
2327:
2328: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
2329: %% B.2
2330: \subsection{Random deletion}
2331: \label{appendix.random_deletion}
2332:
2333:
2334: Derivations of the equivalent noise
2335: Eq.(\ref{5eq:cut.to.mul}) is given here.
2336: The random deletion has similar effects
2337: to the multiplicative synaptic noise.
2338: Therefore, we analyse by a similar way to
2339: the analysis of the multiplicative synaptic noise.
2340: The syanaptic connections are given by Eq.(\ref{5eq:random.cut.J}).
2341: The asymmetric cut coefficients are independently generated
2342: according to the probability distribution of Eq.(\ref{5eq:cut.pdf}).
2343: We analyze a more general case,
2344: where $c_{ij}$ and $c_{ji}$ have an arbitary correlation such that
2345: \begin{eqnarray}
2346: \mbox{Cov}[c_{ij},c_{ji}] &=& k_c \mbox{Var}[c_{ij}], \quad
2347: -1 \leq k_c \leq 1 \\
2348: \mbox{Var}[c_{ij}] &=& \mbox{E}[(c_{ij})^2] - (\mbox{E}[c_{ij}])^2
2349: \nonumber \\
2350: &=& c(1-c).
2351: \end{eqnarray}
2352: In this general case, the symmetric and asymmetric random deletion
2353: correspond to $k_c =1$ and $k_c =0$, respectively.
2354: According to a similar analysis of the multiplicative synaptic noise,
2355: the local field becomes
2356: \begin{eqnarray}
2357: h_i
2358: &=& (\xi^1_i-a) m_1 +
2359: \alpha \left[\frac{1}{1-U} + \frac{k_c(1-c)}{c} \right] U x_i
2360: \nonumber \\
2361: & & + \frac{1}{N(1-a^2)(1-U)} \sum_{j \neq i}^N \sum_{\mu = 2}^{\alpha N}
2362: (\xi^\mu_i-a) (\xi^\mu_j-a) x_j^{(\mu)} \nonumber \\
2363: & & + \frac{1}{Nc(1-a^2)} \sum_{\mu = 1}^{\alpha N} \sum_{j \neq i}^N
2364: (c_{ij}-c) \nonumber \\
2365: & & \qquad \qquad \times (\xi_i^\mu-a) (\xi_j^\mu-a) x^{(\mu)(c_{ji})}_j,
2366: \label{ap:cut.eq.h_i_final}
2367: \end{eqnarray}
2368: where
2369: \begin{eqnarray}
2370: & &x ^{(\mu)(c_{ji})}_j = F (h_j - h_j^{\{ \mu,c_{ji} \}}), \\
2371: & & x^{(\mu)}_i =
2372: \tilde{F} \biggl( \sum_{\nu \neq \mu}^{\alpha N} (\xi_i^\nu-a) m_\nu +
2373: \frac{1}{Nc(1-a^2)} \nonumber \\
2374: & & \quad \times \sum_{\nu \neq \mu}^{\alpha N} \sum_{j \neq i}^N
2375: (c_{ij}-c) (\xi_i^\nu-a) (\xi_j^\nu-a) x^{(\nu)(c_{ji})}_j \biggr) ,
2376: \nonumber \\
2377: & & \\
2378: & &h_j^{\{ \mu,c_{ji} \}} = (\xi^\mu_j-a) m_\mu
2379: + \frac{1}{Nc(1-a^2)} \nonumber \\
2380: & &\quad \times
2381: \sum_{k \neq j}^N (c_{jk}-c) (\xi_j^\mu-a) (\xi_k^\mu-a) x_k
2382: \nonumber \\
2383: & &\quad + \frac{c_{ji}-c}{Nc(1-a^2)} \sum_{\nu \neq \mu}^{\alpha N}
2384: (\xi_j^\nu-a) (\xi_i^\nu-a) x_i \nonumber \\
2385: & &\quad + \frac{c_{ji}-c}{Nc(1-a^2)} (\xi_j^\mu-a) (\xi_i^\mu-a) x_i,
2386: \label{ap:cut.eq.h_j}
2387: \end{eqnarray}
2388: and $U$ is defined by Eq.(\ref{eq:appA_additive_U}) similarly.
2389: The variance of the noise term is given by
2390: \begin{equation}
2391: \sigma^2=
2392: \frac{\alpha q}{(1-U)^2} + \alpha \frac{1-c}{c} q.
2393: \label{ap:sigma.random_deletion}
2394: \end{equation}
2395: Thus, after rewriting $\xi_i^1 \to \xi$ and $m_1 \to m$,
2396: the effective response function becomes
2397: \begin{eqnarray}
2398: Y(z;\xi ) &=&
2399: F \biggl( (\xi -a) m + \sigma z +h \nonumber \\
2400: & & + \alpha \left[\frac{1}{1-U} + \frac{k_c(1-c)}{c} \right] U Y(z;\xi )
2401: \biggr). \nonumber \\
2402: \label{ap:random_deletion.Y}
2403: & &
2404: \end{eqnarray}
2405: Finally,
2406: the equivalence between random deletion
2407: and the additive synaptic noise is obtained as follows,
2408: \begin{eqnarray}
2409: J &=& 1, \\
2410: \Delta_A^2 &=& \alpha (1-a^2)^2 \frac{1-c}{c},
2411: \label{ap:cut2add} \\
2412: k_\delta &=& k_c,
2413: \end{eqnarray}
2414: by comparing
2415: Eqs.(\ref{ap:sigma.random_deletion}) and (\ref{ap:random_deletion.Y})
2416: to Eqs.(\ref{ap:sigma.additive_noise})
2417: and (\ref{ap:effective_response_function}).
2418: Substituting Eq.(\ref{ap:cut2add}) into Eq.(\ref{ap:mul2add}),
2419: we obtain the equivalence of Eq.(\ref{5eq:cut.to.mul}).
2420:
2421:
2422:
2423:
2424:
2425:
2426:
2427:
2428:
2429:
2430: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
2431: %% B.3
2432: \subsection{Nonlinear synapse}
2433: \label{appendix.nonlinear_synapse}
2434:
2435:
2436: Derivations of the equivalent noise
2437: Eq.(\ref{5eq:nonlinear.to.mul}) is given here.
2438: The effect of the nonlinear synapse can be separated into
2439: a signal part and a noise part.
2440: The noise part can be regarded as the additive synaptic noise.
2441:
2442: The systematic deletion of synaptic connections can be achieved
2443: by introducing synaptic noise with an appropriate nonlinear function $f(x)$
2444: \cite{Sompolinsky1986b}.
2445: Note that $T_{ij}$ obeys the normal distribution $N(0,1)$
2446: for $p=\alpha N \to \infty$.
2447: According to this naive S/N analysis \cite{Okada1998},
2448: we can write the connections as
2449: \begin{eqnarray}
2450: J_{ij} &=& \frac{\sqrt{p}}{N} f(T_{ij}) \nonumber \\
2451: &=& \frac{J}{N(1-a^2)} \sum_{\mu = 1}^{p}
2452: (\xi_{i}^{\mu}-a) (\xi_{j}^{\mu}-a) \nonumber \\
2453: & & - \biggl[ \frac{\sqrt{p}}{N} f(T_{ij})
2454: -\frac{J}{N(1-a^2)}
2455: \sum_{\mu = 1}^{p} (\xi_{i}^{\mu}-a) (\xi_{j}^{\mu}-a) \biggr]
2456: \nonumber \\
2457: &=& \frac{\sqrt{p}}{N} \{J T_{ij} -[f(T_{ij})-J T_{ij}]\}.
2458: \label{ap:nonlinear.naive_sn.J}
2459: \end{eqnarray}
2460: The following derivation suggests that
2461: the residual overlap $m_\mu$ for the first term
2462: in Eq.(\ref{ap:nonlinear.naive_sn.J}) is enhanced
2463: by a factor of $1/(1-JU)$, while any enhancement to the last part is canceled
2464: because of the subtraction.
2465: It also implies that the last part corresponds to the synaptic noise.
2466: For the SCSNA of the nonlinear synapse,
2467: we can analyze by a similar manner of the analysis of the additive synaptic noise.
2468: We obtain the local field:
2469: \begin{eqnarray}
2470: h_i
2471: &=& J (\xi_i^1-a) m_1 + \alpha \left[ \frac{J^2}{1-JU}
2472: + (\tilde{J^2}-J^2)\right] U x_i \nonumber \\
2473: & & + \frac{\sqrt{p}}{N} \sum_{j \neq i}
2474: \left[f(T_{ij}) - J T_{ij} \right] x^{(T_{ji})}_j \nonumber \\
2475: & & + \frac{J}{N(1-a^2)(1-JU)}
2476: \sum_{\mu=2}^p \sum_{j \neq i}^N (\xi_i^\mu-a) (\xi_j^\mu-a) x_j^{(\mu)},
2477: \nonumber \\
2478: & &
2479: \label{ap:nl.eq.h_i_final}
2480: \end{eqnarray}
2481: where
2482: \begin{eqnarray}
2483: x^{(\mu)}_i
2484: &=&
2485: \tilde{F} \biggl( J \sum_{\nu \neq \mu}^p (\xi_i^\nu-a) m_\nu
2486: \nonumber \\
2487: & & + \frac{\sqrt{p}}{N} \sum_{j \neq i}^N
2488: [f(T^{(\mu)}_{ij}) - J T^{(\mu)}_{ij}] x^{(T_{ji})}_j \biggr),
2489: \label{ap:nonlinear.x_mu} \\
2490: x^{(T_{ji})}_j
2491: &=& F \biggl( h_j - \frac{\sqrt{p}}{N} [f(T_{ji})-J T_{ji}] x_i) \biggr) ,\\
2492: T^{(\mu )}_{ij} &=& \frac{1}{\sqrt{p}(1-a^2)}
2493: \sum_{\nu \neq \mu}^p (\xi_i^\nu-a) (\xi_j^\nu-a),
2494: \end{eqnarray}
2495: and $U$ is defined by Eq.(\ref{eq:appA_additive_U}) similarly.
2496: The variance of the noise term is given by
2497: \begin{equation}
2498: \sigma^2=
2499: \frac{\alpha J^2 q}{(1-JU)^2}q
2500: + \alpha (\tilde{J^2}-J^2).
2501: \label{ap:nonlinear.sigma}
2502: \end{equation}
2503: Thus, after rewriting $\xi_i^1 \to \xi$ and $m_1 \to m$,
2504: The effective response function becomes
2505: \begin{eqnarray}
2506: Y(z;\xi ) &=&
2507: F \biggl(
2508: J (\xi -a) m + \sigma z +h \nonumber \\
2509: & & + \alpha \left[ \frac{J^2}{1-JU}
2510: + (\tilde{J^2}-J^2) \right] U Y(z;\xi ) \biggr). \nonumber \\
2511: \label{ap:nonlinear.Y}
2512: & &
2513: \end{eqnarray}
2514: Finally,
2515: the equivalence between the nonlinear synapse
2516: and the additive synaptic noise is obtained as follows,
2517: \begin{eqnarray}
2518: \Delta_A^2 &=& \alpha (1-a^2)^2 \biggl( \frac{\tilde{J^2}}{J^2}-1 \biggr),
2519: \label{ap:nonlinear2add} \\
2520: J &=& \int Dx \; x f(x) \\
2521: \tilde{J^2} &=& \int Dx f(x)^2
2522: \end{eqnarray}
2523: by comparing
2524: Eqs.(\ref{ap:nonlinear.sigma}) and (\ref{ap:nonlinear.Y})
2525: to Eqs.(\ref{ap:sigma.additive_noise})
2526: and (\ref{ap:effective_response_function}).
2527: Substituting Eq.(\ref{ap:nonlinear2add}) into Eq.(\ref{ap:mul2add}),
2528: we obtain the equivalence of Eq.(\ref{5eq:nonlinear.to.mul}).
2529:
2530:
2531:
2532:
2533:
2534:
2535: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
2536: %% APPENDIX C
2537: \section{Asymptote for large multiplicative synaptic noise}
2538: %Critical value of additive synaptic noise
2539: \label{appendix.critical_noise}
2540:
2541:
2542: Derivations of the asymptote of storage capacity
2543: in a large multiplicative synaptic noise $\Delta_M$ is given here.
2544: %In the case of the additive synaptic noise,
2545: %the critical value $\Delta_{Ac}$,
2546: %where storage capacity vanishes, exists.
2547: %%ノイズの臨界値 $\delta_c$ を求める．
2548:
2549: In Eqs.(\ref{eq:order_parameter_equation_m})
2550: -(\ref{eq:order_parameter_equation_U}),
2551: let $a=0$, $J=1$, and $F(x)=\sgn (x)$,
2552: the order-parameter equations become
2553: \begin{eqnarray}
2554: m &=& \erf \biggl( \frac{m}{\sqrt{2} \sigma} \biggr) , \label{ap:m} \\
2555: q &=& 1, \\
2556: U &=& \frac 1\sigma \sqrt{\frac{2}{\pi}}
2557: \exp \biggl( -\frac{m^2}{2\sigma^2} \biggr) , \label{ap:U}
2558: \end{eqnarray}
2559: the threshold becomes $h=0$,
2560: the effective response function of
2561: Eq.(\ref{eq:effevtive_response_function})
2562: and the variance of the noise become
2563: \begin{eqnarray}
2564: Y(z;\xi ) &=& {\rm sgn}(\xi m + \sigma z), \\
2565: \sigma^2 &=& \frac {\alpha}{(1-U)^2} + \Delta_A^2, \label{ap:sigma2}
2566: \end{eqnarray}
2567: respectively,
2568: where the error function ${\rm erf}(x)$ is defined as
2569: \begin{equation}
2570: {\rm erf}(x) = \frac2{\sqrt{\pi}} \int_0^x e^{-u^2} du.
2571: \end{equation}
2572: The slope of the r.h.s. of Eq.(\ref{ap:m}) is given by
2573: %(\ref{ap:m})式の右辺の傾きは，
2574: \begin{equation}
2575: \frac {d}{dm} \erf \biggl( \frac{m}{\sqrt{2} \sigma} \biggr)
2576: = \frac 1\sigma \sqrt{\frac 2 \pi}
2577: \exp \biggl( -\frac {m^2}{2\sigma^2} \biggr) . \label{ap:erf_slope}
2578: \end{equation}
2579: Equation (\ref{ap:m}) has nontrivial solutions $m\ne 0$
2580: within the range where the slope of the r.h.s. of Eq.(\ref{ap:erf_slope}) at $m=0$ is greater than 1.
2581: %となるので，
2582: %$m=0$ で，この傾きが $1$ よりも大きいときは，
2583: %(\ref{ap:m})式が非自明な解 $m\ne 0$ を持つ．
2584: Therefore, the critical value of the noise $\sigma_c^2$ is given by
2585: %よって，
2586: \begin{equation}
2587: \sigma_c^2 = 2/\pi.
2588: \end{equation}
2589: This shows that a retrieval phase exists only for $\sigma<\sigma_c$.
2590: %と，ノイズが $\sigma<\sigma_c$ のときに想起相が存在する．
2591: We define the parameter $\tau (<1)$ defined as
2592: %ノイズ $\sigma$ が臨界値 $\sigma_c$ に
2593: %近い場合を評価する．
2594: %そこで，
2595: \begin{equation}
2596: \tau=\frac \sigma{\sigma_c},
2597: \end{equation}
2598: to solve for $m$ as a function of $\sigma$
2599: in the vicinity of this critical value $\sigma_c$.
2600: The critical value of the additive synaptic noise
2601: is discussed in the case of $\tau \simeq 1$.
2602: %とおき， $\tau \simeq 1$ として評価する．
2603: The overlap $m$ shows the first order phase transition
2604: when $\Delta_A$ is small,
2605: but it is regarded as the second order phase transition at large $\Delta_A$ region.
2606: The overlap becomes $m \ll 1$ when $\tau \simeq 1$
2607: and $\Delta_A$ is sufficiently large,
2608: therefore the nontrivial solution of $m$ is given as
2609: \begin{equation}
2610: m
2611: \simeq
2612: \frac m\tau - \frac {m^3}{6\sigma^2\tau} + O(m^4)
2613: =
2614: \sigma_c \tau \sqrt{6(1-\tau )}, \label{ap:nontrivial_m}
2615: \end{equation}
2616: by Taylor expansion including terms up to the third order.
2617: %このとき，$m \ll 1$ なので誤差関数を3次まで展開することにより，
2618: %$m$ の非自明な解を求めると，
2619: %となる．
2620: Substituting Eq.(\ref{ap:nontrivial_m}) into Eq.(\ref{ap:U}), $U$ becomes
2621: %また，$U$ は，
2622: \begin{equation}
2623: U
2624: \simeq
2625: \frac 1\tau \biggl( 1-\frac {m^2}{2\sigma^2} \biggr) +O(m^4)
2626: =
2627: 3 -2\tau^{-1} . \label{ap:U_at_small_m}
2628: \end{equation}
2629: %となる．
2630: From Eq.(\ref{5eq:add.to.mul}),
2631: the variance of the multiplicative synaptic noise $\Delta_M^2$
2632: is related to the variance of the additive synaptic noise $\Delta_A^2$ as
2633: %偏り$a=0$において，
2634: %式(\ref{5eq:add.to.mul})のように
2635: %乗法的ノイズの分散$\Delta_M^2$は加法的ノイズ$\Delta_A^2$の分散と，
2636: \begin{equation}
2637: \Delta_A^2 = \alpha \Delta_M^2 , \label{ap:a2m_at_a=0}
2638: \end{equation}
2639: when bias $a=0$.
2640: Therefore, substituting Eqs.(\ref{ap:U_at_small_m}) and (\ref{ap:a2m_at_a=0}) into Eq.(\ref{ap:sigma2}),
2641: the variance of the noise $\sigma$ is given as
2642: %よって，ノイズ $\sigma$ は，
2643: \begin{equation}
2644: \sigma^2=\frac{\alpha \tau^2}{4(1-\tau )^2}+\alpha \Delta_M^2.
2645: \label{ap:sigma2_aprx}
2646: \end{equation}
2647: %となる．
2648: The loading rate $\alpha$ becomes
2649: \begin{equation}
2650: \alpha = \frac 8\pi \cdot \frac{\tau^2(1 - \tau)^2}{\tau^2 + 4 \Delta_M^2 (1 - \tau^2)}.
2651: \label{ap:small_m_op_eq}
2652: \end{equation}
2653: %--------------------------------------------------修正ここから
2654: When the overlap is small enough, i.e., $m \ll 1$,
2655: the order-parameter equations of Eqs. (\ref{ap:m})-(\ref{ap:sigma2}) reduce to Eq. (\ref{ap:small_m_op_eq}).
2656: Solving Eq.(\ref{ap:small_m_op_eq}) for the fixed value of $\alpha$ and $\Delta_M$, we obtain the parameter $\tau$.
2657: Substituting $\tau$ into Eq. (\ref{ap:nontrivial_m}), we can obtain the overlap $m$ for given $\alpha$ and $\Delta_M$.
2658: It is easily confirmed that the $\tau$ increases with $\alpha$ for the fixed value of $\Delta_M$.
2659: This means that the maximal value of $\tau$ which holds Eq. (\ref{ap:small_m_op_eq})
2660: corresponds to the maximum value of $\alpha$, that is storage capacity $\alpha_c$.
2661: The critical value $\tau_c$ is equal to the value which maximizes the loading rate of Eq.(\ref{ap:small_m_op_eq})
2662: and becomes
2663: %When the overlap is small, i.e., $m \ll 1$,
2664: %the order parameter equations of Eqs. (\ref{ap:m})-(\ref{ap:sigma2})
2665: %are equivalent to Eq. (\ref{ap:small_m_op_eq}).
2666: %For the given multiplicative synaptic noise $\Delta_M$,
2667: %the loading rate $\alpha$ regards as the function of $\tau$.
2668: %Storage capacity is defined as $\alpha_c =\alpha (\tau_c)$
2669: %with the maximum value $\tau =\tau_c$ which satisfies $\alpha = \alpha (\tau ) >0$.
2670: %The critical value $\tau_c$ is equal to the value which maximizes the loading rate of Eq.(\ref{ap:small_m_op_eq})
2671: %and becomes
2672: %%When $\tau > \tau_c$, storage capacity $\alpha=0$.
2673: %%The first term of Eq. (\ref{ap:sigma2_aprx}) is
2674: %%the variance of the cross-talk noise derived from memorized patterns.
2675: %%In the case that the variance of the additive synaptic noise $\Delta_A$ is fixed,
2676: %%the number of memorized patterns becomes large when the parameter $\tau$ becomes large.
2677: %%Therefore, the maximum value of the loading rate equals to storage capacity $\alpha_c$.
2678: \begin{equation}
2679: \tau_c = \frac{(2\Delta_M)^{2/3}}{1+(2\Delta_M)^{2/3}} \simeq 1-(2\Delta_M)^{-2/3}, \label{ap:tau_c}
2680: \end{equation}
2681: in a large $\Delta_M$ limit.
2682: %Let the variance of the noise $\sigma=\sigma_c$ when $\alpha = \alpha_c$, we obtain
2683: %%記憶率 $\alpha$ が記憶容量 $\alpha_c$ に一致するとき，
2684: %%$\sigma=\sigma_c$ となることから (\ref{ap:sigma2_aprx})式より，
2685: %\begin{eqnarray}
2686: %\sigma_c^2
2687: %&=& \frac{\alpha_c \tau_c^2}{4(1-\tau_c )^2}+ \alpha_c \Delta_M^2 \nonumber \\
2688: %&=& \alpha_c \biggl( \frac{(1-(2\Delta_M)^{-2/3})^2}{4(2\Delta_M)^{-4/3}} + \Delta_M^2 \biggr) \nonumber \\
2689: %&\simeq& \alpha_c \biggl( \frac 14 (2\Delta_M)^{4/3} + \Delta_M^2 \biggr) \nonumber \\
2690: %&\simeq& \alpha_c \Delta_M^2,
2691: %\end{eqnarray}
2692: %and
2693: %\begin{equation}
2694: %\sigma_c
2695: %=\tau \sigma_*
2696: %=(1-(2\Delta_M)^{-2/3}) \sigma_*
2697: %\simeq \sigma_* ,
2698: %\end{equation}
2699: %in a large $\Delta_M$ limit.
2700: Therefore, substituting Eq. (\ref{ap:tau_c}) into Eq.(\ref{ap:small_m_op_eq}),
2701: we obtain Eq.(\ref{5eq:mul.asymptote}) as follows:
2702: \begin{equation}
2703: \alpha_c =\frac 2{\pi \Delta_M^2}.
2704: \end{equation}
2705: %を得る．
2706:
2707:
2708:
2709:
2710:
2711: %▼▼▼▼▼前の説明ここから▼▼▼▼▼
2712: %Let the variance of the additive synaptic noise
2713: %be $\Delta_A=\sigma_c \kappa (0 \le \kappa \le 1)$,
2714: %storage capacity $\alpha_c$ is given as
2715: %%いま， $\kappa \simeq 1$ として
2716: %%$\Delta_A=\sigma_c \kappa$ とおくと記憶容量 $\alpha_c$ は
2717: %\begin{equation}
2718: %\alpha_c=4\sigma_c^2\tau^{-2}(1-\tau )^2(1- \kappa^2).
2719: %\end{equation}
2720: %%と評価できる．
2721: %When the variance of the additive synaptic noise $\Delta_A$ is constant,
2722: %i.e., $\kappa$ is constant, the loading rate $\alpha$ is
2723: %%加法的ノイズ $\Delta_A$ を一定（$\kappa$ 一定）として，
2724: %%$\tau \to 1$ の極限を考えると，
2725: %\begin{equation}
2726: %\lim_{\tau \to 1} \alpha = \alpha_c.
2727: %\end{equation}
2728: %%に注意すれば，
2729: %Hence, the first term of Eq.(\ref{ap:sigma2}) approaches to
2730: %%(\ref{ap:sigma2})式のノイズ $\sigma$ の第1項は
2731: %%加法的ノイズが大きい（$\kappa \to 1$）とき，
2732: %\begin{eqnarray}
2733: %\lim_{\kappa \to 1}
2734: %\lim_{\tau \to 1} \frac {\alpha}{(1-U)^2}
2735: %&=&
2736: %\lim_{\kappa \to 1} \lim_{\tau \to 1}
2737: %\frac
2738: %{4\sigma_c^2\tau^{-2}(1-\tau )^2(1- \kappa^2)}
2739: %{[1-(3 -2\tau^{-1})]^2} \nonumber \\
2740: %&=&
2741: %%%%%%%%%%% \sigma_c^2 \lim_{\kappa \to 1} (1-\kappa^2) =
2742: %0,
2743: %\end{eqnarray}
2744: %in $\kappa \to 1$ limit.
2745: %%となる．
2746: %Therefore, the critical value
2747: %of the additive synaptic noise $\Delta_{Ac}^2$ is given by
2748: %%これより，加法的ノイズの臨界値 $\Delta_{Ac}$ は，
2749: %\begin{eqnarray}
2750: %\Delta_{Ac}^2
2751: %&=&
2752: %\lim_{\kappa \to 1} \lim_{\tau \to 1} \Delta_{A}^2
2753: %\nonumber \\
2754: %&=&
2755: %\lim_{\kappa \to 1} \lim_{\tau \to 1}
2756: %\biggl(
2757: %\sigma^2 - \frac {\alpha}{(1-U)^2}
2758: %\biggr)\nonumber \\
2759: %&=& \sigma_c^2,
2760: %\end{eqnarray}
2761: %from Eq.(\ref{ap:sigma2}).
2762: %This shows that a retrieval phase exists only for $\Delta_A^2<\Delta_{Ac}^2=2/\pi$
2763: %\cite{Sompolinsky1986c}.
2764: %▲▲▲▲▲前の説明ここまで▲▲▲▲▲
2765:
2766:
2767:
2768:
2769:
2770:
2771:
2772:
2773:
2774:
2775:
2776:
2777:
2778:
2779:
2780:
2781:
2782:
2783: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
2784: %% BIBLIOGRAPHY
2785: \begin{thebibliography}{12}
2786:
2787:
2788:
2789: %%%%%%%%%%%% elimination
2790: %
2791: \bibitem{Huttenlocker1979}
2792: P. R. Huttenlocker, Brain Res. {\bf 163} 195 (1979).
2793:
2794: %
2795: \bibitem{Huttenlocker1982}
2796: P. R. Huttenlocker, C. De Courten, L. J. Garey, and H. Van der Loos,
2797: Neurosci. lett. {\bf 33}, 247 (1982).
2798:
2799: %
2800: %\bibitem{Huttenlocker1987}
2801: %P. R. Huttenlocker and C. De Courten,
2802: %J. Neurosci. (1987).
2803:
2804: %
2805: \bibitem{Bourgeois1993}
2806: J. P. Bourgeois and P. Rakic,
2807: J. Neurosci. {\bf 13}, 2801 (1993).
2808:
2809: %
2810: \bibitem{Takacs1986}
2811: J. Takacs and J. Hamori,
2812: J. of Neurosci. Research {\bf 38}, 515 (1994).
2813:
2814: %
2815: \bibitem{Innocenti1995}
2816: G. M. Innocenti,
2817: Trends Neurosci. {\bf 18}, 397 (1995).
2818:
2819: %
2820: \bibitem{Eckehoff1991}
2821: M. F. Eckenhoff and P. Rakic,
2822: Developmental Brain Research {\bf 64}, 129 (1991).
2823:
2824: %
2825: \bibitem{Rakic1994}
2826: P. Rakic, J. P. Bourgeois, and P. S. Goldman-Rakic,
2827: Progress in Brain Research {\bf 102}, 227 (1994).
2828:
2829: %
2830: \bibitem{Stryker1986}
2831: M. P. Stryker,
2832: J. of Neurosci. {\bf 6}, 2117 (1986).
2833:
2834: %
2835: \bibitem{Sur1990}
2836: A. W. Roe, S. L. Pallas, J. O. Hahm, and Sur,
2837: Science {\bf 250}, 818 (1990).
2838:
2839: %
2840: \bibitem{Wolff1995}
2841: J. R. Wolff, R. Laskawi, W. B. Spatz, and M. Missler,
2842: Behavioural Brain Research {\bf 66}, 13 (1995).
2843:
2844:
2845:
2846:
2847:
2848: %
2849: \bibitem{Yanai1996}
2850: H. Yanai and S. Amari,
2851: IEEE Trans. on Neural Netw. {\bf 7}, 803 (1996).
2852:
2853: %
2854: \bibitem{Amit1994}
2855: D. J. Amit, N. Brunel, and M. V. Tsodyks,
2856: Journal of Neuroscience {\bf 14}, 6435 (1994).
2857:
2858: %
2859: \bibitem{Griniasty1993}
2860: M. Griniasty, M. V. Tsodyks, and D. J. Amit,
2861: Neural Computation {\bf 5}, 1 (1993).
2862:
2863: %
2864: \bibitem {Sompolinsky1986b}
2865: H. Sompolinsky,
2866: Phys. Rev. A {\bf 34}, 2571 (1986).
2867: %J.L. van Hemmen and I. Morgenstern,
2868: %{\it Heidelberg Colloquium on Glassy Dynamics} (Springer-Verlag, 1986), 485.
2869:
2870: %
2871: \bibitem{Chechik1998}
2872: G. Chechik, I. Meilijson, and E. Ruppin,
2873: Neural Computation {\bf 10}, 1759 (1998).
2874:
2875: %
2876: \bibitem{Okada1998}
2877: M. Okada, T. Fukai, and M. Shiino,
2878: Phys. Rev. E {\bf 57}, 2095 (1998).
2879:
2880: %
2881: \bibitem{Amit1987}
2882: D. J. Amit, H. Gutfreund, and H. Sompolinsky,
2883: Phys. Rev. A {\bf 35}, 2293 (1987).
2884:
2885: %
2886: \bibitem{Amari1989}
2887: S. Amari,
2888: Neural Networks {\bf 2}, 451 (1989).
2889:
2890:
2891:
2892:
2893:
2894: %
2895: \bibitem{Meilijson1996}
2896: I. Meilijson and E. Ruppin,
2897: Biol. Cybern. {\bf 74}, 479 (1996).
2898:
2899: %
2900: \bibitem{Okada1996}
2901: M. Okada,
2902: Neural Netw. {\bf 9}, 1429 (1996).
2903:
2904:
2905:
2906: %
2907: \bibitem{Shiino1992}
2908: M. Shiino and T. Fukai,
2909: J. Phys. A {\bf 25}, L375 (1992).
2910:
2911: %
2912: \bibitem{Shiino1993}
2913: M. Shiino and T. Fukai,
2914: Phys. Rev. E {\bf 48}, 867 (1993).
2915:
2916: %
2917: \bibitem{Okada1995}
2918: M. Okada,
2919: Neural Netw. {\bf 8}, 833 (1995).
2920:
2921:
2922: %
2923: \bibitem{Mimura1998-1}
2924: K. Mimura, M. Okada, and K. Kurata,
2925: IEICE Trans. Information and systems E81-D {\bf 8}, 928 (1998).
2926:
2927: %
2928: \bibitem{Mimura1998-2}
2929: K. Mimura, M. Okada, and K. Kurata,
2930: IEICE Trans. Information and systems E81-D {\bf 11}, 1298 (1998).
2931:
2932: %
2933: \bibitem{Kimoto2001}
2934: T. Kimoto, M. Okada,
2935: Biol. Cybern. {\bf 85}, 319 (2001).
2936:
2937: %
2938: \bibitem{Sompolinsky1986c}
2939: H.Sompolinsky, in J.L.van Hemmen, I.Morgenstern (eds.),
2940: Heidelberg Colloquium on Glassy Dynamics, 485, Springer-Verlag (1987).
2941:
2942:
2943:
2944:
2945: %%%%%%%%%%%%%%%% 1/(flog f)
2946: %
2947: \bibitem{Tsodyks1988}
2948: M. V. Tsodyks, M. V. Feigelman,
2949: Europhys. Lett. {\bf 6}, 101 (1988).
2950:
2951: %
2952: \bibitem{Buhmann1989}
2953: J. Buhmann, R. Divko, and K. Schulten,
2954: Phys. Rev. A {\bf 39}, 2689 (1989).
2955:
2956: %
2957: \bibitem{Perez-Vincente1989}
2958: C. J. Perez-Vincente and D. J. Amit,
2959: J. Phys. A {\bf 22}, 559 (1989).
2960:
2961:
2962:
2963:
2964:
2965:
2966:
2967:
2968:
2969:
2970:
2971: \end{thebibliography}
2972:
2973:
2974:
2975:
2976:
2977:
2978:
2979:
2980:
2981:
2982: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
2983: \end{document}
2984:
2985: