1: \documentclass[pre,twocolumn,superscriptaddress,amsfonts]{revtex4}
2: %\documentclass[pre,preprint,superscriptaddress,amsfonts]{revtex4}
3:
4: % $Id: SC.tex,v 1.15 2002/04/26 09:15:23 tanakat Initial $
5:
6: \usepackage{bm,amsmath,dcolumn}
7: \usepackage[dvips]{graphicx}
8:
9: \newcommand\qq{\bm{q}}
10: \newcommand\rr{\bm{r}}
11: \newcommand\uu{\bm{u}}
12: \newcommand\vv{\bm{v}}
13: \newcommand\xx{\bm{x}}
14: \newcommand\yy{\bm{y}}
15: \newcommand\JJ{\bm{J}}
16: \renewcommand\SS{\bm{S}}
17: \newcommand\ttau{\bm{\tau}}
18: \newcommand\xxi{\bm{\xi}}
19: \newcommand\zzeta{\bm{\zeta}}
20:
21: \newcommand\cG{{\mathcal{G}}}
22: \newcommand\cH{{\mathcal{H}}}
23: \newcommand\cL{{\mathcal{L}}}
24:
25: \newcommand\BSC{{\textrm{BSC}}}
26: \newcommand\BIAWGNC{{\textrm{BIAWGNC}}}
27: \newcommand\BILC{{\textrm{BILC}}}
28:
29: \newcommand\RS{{\textrm{RS}}}
30: \newcommand\ferro{{\textrm{ferro}}}
31: \newcommand\subopt{{\textrm{sf}}}
32: \newcommand\para{{\textrm{para}}}
33: \newcommand\C{{\textsf{C}}}
34:
35: \newcommand\sign{\mathop{\mathrm{sign}}\nolimits}
36: \newcommand\diag{\mathop{\mathrm{diag}}\nolimits}
37: \newcommand\Extr{\mathop{\mathrm{Extr}}}
38:
39:
40: \begin{document}
41: \title{Typical performance of low-density parity-check codes\\
42: over general symmetric channels}
43: \author{Toshiyuki Tanaka}
44: \affiliation{Department of Electronics and Information Engineering,
45: Tokyo Metropolitan University, 1-1 Minami-Osawa, Hachioji-shi, Tokyo,
46: 192-0397 Japan}
47: \affiliation{Neural Computing Research Group, Aston University,
48: Aston Triangle, Birmingham, B4 7ET, United Kingdom}
49: \author{David Saad}
50: \affiliation{Neural Computing Research Group, Aston University,
51: Aston Triangle, Birmingham, B4 7ET, United Kingdom}
52:
53: \date{\today}
54:
55: \begin{abstract}
56: Typical performance of low-density parity-check (LDPC) codes
57: over a general binary-input output-symmetric memoryless channel
58: is investigated using methods of statistical mechanics.
59: Theoretical framework for dealing with general symmetric channels
60: is provided, based on which
61: Gallager and MacKay-Neal codes are studied as
62: examples of LDPC codes.
63: It has been shown that the basic properties of these codes
64: known for particular channels,
65: including the property to potentially saturate
66: Shannon's limit, hold for general symmetric channels.
67: The binary-input additive-white-Gaussian-noise channel
68: and the binary-input Laplace channel
69: are considered as specific channel noise models.
70: \end{abstract}
71:
72: \pacs{02.50.-r, 75.10.Hk, 89.70.+c, 89.20.Kk}
73:
74: \maketitle
75:
76: \section{Introduction}
77: We investigate the typical performance of low-density parity-check
78: (LDPC) codes over a general binary-input output-symmetric (BIOS)
79: memoryless channel. Previous statistical physics based analyses of
80: LDPC codes have discovered some interesting properties, including the
81: fact that they can, in principle, saturate the information-theoretic
82: upper bound (Shannon's bound defined by the channel coding
83: theorem~\cite{Shannon48}) with low connectivity values. Existing
84: statistical mechanical studies on the LDPC codes, however, have been
85: mostly confined to the case of binary symmetric channel (BSC), which
86: fits into the statistical-mechanical framework in a natural
87: way~\cite{KMS00,MKSV00,VSK00,VSK02}.
88: Notable exceptions are the work by Montanari~\cite{montanari} that discusses
89: the case of binary-input additive-white-Gaussian-noise channel (BIAWGNC)
90: as well as the BSC case and the study of Sourlas
91: codes~\cite{Sourlas89}, a simple LDPC code, in which non-BSC
92: channels are addressed~\cite{Rujan93,NW99,VSK99}. From the
93: statistical-mechanical point of view, LDPC codes are regarded as
94: random spin systems; it is therefore natural to expect that they will
95: exhibit some sort of universality, just as typical
96: statistical-mechanical systems do, so that general properties of LDPC
97: codes observed in the BSC case will be preserved when different
98: communication channels are considered. In this paper we investigate
99: the properties of LDPC codes in binary-input output-symmetric channels
100: and show that this is generally the case. In particular, we show that
101: the finite connectivity LDPC codes can saturate Shannon's bound for
102: general BIOS channel.
103:
104: The paper is organized as follows: In section~\ref{sec:framework} we
105: introduce the general framework, notation, codes and the channels that
106: we will focus on. In section~\ref{sec:analysis} we will briefly
107: describe the calculation for the various channels, while the results
108: obtained will be described in section~\ref{sec:res}, followed by the
109: conclusions.
110:
111: \section{The general framework}
112: \label{sec:framework}
113: \subsection{Symmetric channels}
114: \label{sec:channel}
115:
116: We consider the general class of binary-input output-symmetric (BIOS)
117: memoryless channel. The input of the channel is binary ($\pm1$), and
118: the output may take any real value. The characteristics of a channel
119: is described by the channel transition probabilities, $P(y|x=1)$ and
120: $P(y|x=-1)$. Let $p(y)\equiv P(y|x=1)$. A symmetric channel is
121: characterized as a channel whose transition probabilities satisfy
122: $P(y|x=-1)=P(-y|x=1)=p(-y)$. Various types of channel models of
123: practical interest fall into the class of BIOS channels, including the
124: binary symmetric channel (BSC)
125: \begin{equation}
126: p_{\BSC}(y)=(1-p)\delta(y-1)+p\delta(y+1),
127: \end{equation}
128: the binary-input additive-white-Gaussian-noise channel (BIAWGNC)
129: \begin{equation}
130: p_{\BIAWGNC}(y)=\frac{1}{\sqrt{2\pi\sigma^2}}
131: e^{-{(y-1)^2/2\sigma^2}},
132: \end{equation}
133: and the binary-input Laplace channel (BILC)
134: \begin{equation}
135: p_{\BILC}(y)=\frac{1}{\lambda}e^{-|y-1|/\lambda},
136: \end{equation}
137: Each of the
138: parameters $p$, $\sigma^2$, and $\lambda$ represents the degree of
139: degradation induced by the channel noise. We call each
140: of them the noise level and let $d$ denotes the generic one.
141:
142: An apparent technical difficulty in dealing with
143: a general channel of real-valued output
144: is that it is not at all obvious how to define
145: the syndrome from the received signal:
146: The modulo 2 arithmetic involved in computing
147: syndrome in the BSC case is not directly applicable
148: to the cases of real-valued received signal.
149: This difficulty is resolved by using a truncation
150: procedure~\cite{MacKay99}: We conceptually consider another {\em
151: fictitious} binary-input binary-output channel in addition to the
152: channel under study. Let $r$ be the (fictitious) output symbol of
153: this fictitious channel. We can assign to $r$ either of the values
154: $\pm1$ arbitrarily, and the binary channel noise $\zeta$ for the
155: fictitious channel is defined therefrom, via $r=x\zeta$. For the sake
156: of making the argument simple, we assign $r=1$ without loss of
157: generality. Since the prior probability of $\zeta$ ({\em before}
158: receiving $y$) should be $P(\zeta=\pm1)=1/2$, the joint distribution
159: of $y$ and $\zeta$ is given by
160: \begin{equation}
161: P(y,\zeta)=\frac{p(\zeta y)}{2}
162: \end{equation}
163: since the truncation procedure used here yields $x=\zeta$.
164:
165:
166: \subsection{Gallager code}
167: \label{sec:Gallager}
168:
169: LDPC codes have been originally introduced by Gallager in his seminal
170: work from 1963~\cite{Gallager62}. Gallager's original
171: construction~\cite{Gallager62} is one of the most extensively studied
172: LDPC codes in the information theory literature. It is defined by its
173: parity-check matrix $A=[C_1|C_2]$ of dimensionality $(M-N)\times M$,
174: which is taken to be random and very sparse. The submatrix $C_2$, of
175: dimensionality $(M-N)\times(M-N)$, is assumed invertible.
176:
177: In the encoding step, the encoder computes a codeword
178: from the information vector $\xxi\in\{0,1\}^N$ by employing a generator
179: matrix $G$
180: \begin{equation}
181: \xx=G^T\xxi\mod 2,
182: \end{equation}
183: where the generator matrix is defined by
184: \begin{equation}
185: G=[I|C_2^{-1}C_1]\mod 2.
186: \end{equation}
187: This construction ensures $AG^T=0\mod 2$.
188: The information code rate for unbiased messages is $R=N/M$.
189:
190: In regular Gallager codes, the number of non-zero elements per row of
191: $A$ is fixed to be $K$. We call it the row constraint. Average
192: number of non-zero elements per column is then $C\equiv K(M-N)/M$,
193: whereas we will consider the case in which the number of non-zero
194: elements in each column is forced to be exactly $C$, which we term the
195: {\em column constraint}. {\em Irregular} Gallager codes can be
196: defined by relaxing these constraints. It has been known that making
197: code construction irregular may improve performance
198: significantly~\cite{RSU01}, but we will not discuss irregular codes in
199: the current paper. We call the resulting regular Gallager code a
200: $(C,K)$-Gallager code.
201:
202:
203: \subsection{MN code}
204: \label{sec:MN}
205:
206: We also discuss a variant of LDPC codes, called the MN
207: code~\cite{MN95,MacKay99}. The generator matrix $G^T$ of the MN code
208: is defined by
209: \begin{equation}
210: G^T=C_n^{-1}C_s\mod 2,
211: \end{equation}
212: where $C_s$ and $C_n$ are sparse matrices of dimensionality $M\times
213: N$ and $M\times M$, respectively; $C_n$ is assumed invertible. The
214: information rate for the code is $R=N/M$ for unbiased message.
215:
216: In regular MN codes the row and column constraints are imposed on both
217: matrices $C_s$ and $C_n$. The number of non-zero elements per row
218: of $C_s$ and $C_n$ should be exactly $K$ and $L$, respectively.
219: Also here, we do not discuss irregular MN codes~\cite{KS99} in this
220: paper. The number of non-zero elements per column of $C_s$ and
221: $C_n$ are set to $C$ and $L$, respectively, where $C=KM/N$ holds. We
222: call the resulting code a $(K,C,L)$-MN code.
223:
224:
225: \section{Analysis}
226: \label{sec:analysis}
227:
228: \subsection{Gallager code}
229: \label{sec:rep-gal}
230:
231: The basic idea behind the statistical-mechanical treatment of the LDPC
232: codes is the equivalence between the decoding problem and the thermal
233: equilibrium distribution of a dilute Ising spin system. In order to
234: see this in the Gallager code case, one should first note that the
235: decoding problem is to find $\ttau$ which is best supported (i.e.,
236: most probable) by the received signal $\yy$ among the set of $\ttau$
237: satisfying the parity-check equation ($A\zzeta=A\ttau\mod 2$ if we
238: write it in the $\{0,1\}$-notation). The set is expressed as
239: \begin{equation}
240: \biggl\{\ttau\biggm|\lim_{\gamma\to\infty}
241: \exp\biggl[-\gamma\sum_{\mu=1}^{M-N}
242: \biggl(J_\mu\prod_{j\in\cL(\mu)}\tau_j-1\biggr)\biggr]=1
243: \biggr\},
244: \end{equation}
245: where
246: \begin{equation}
247: \cL(\mu)=\{j|A_{\mu j}=1\}
248: \end{equation}
249: denotes the set of indices for which
250: the parity-check matrix $A$ has 1's in $\mu$-th row, and
251: \begin{equation}
252: J_\mu=\prod_{j\in\cL(\mu)}\zeta_j
253: \end{equation}
254: is $\mu$-th check. The posterior probability of $\ttau$ conditioned
255: on the received signal $\yy$ then acquires the following
256: Gibbs-Boltzmann form:
257: \begin{equation}
258: P_\gamma(\ttau|\yy)=\frac{1}{Z}
259: \exp\bigl[-\beta\cH_\gamma(\ttau;\yy,\JJ)\bigr]
260: \end{equation}
261: in which we have to take the limit $\gamma\to\infty$
262: and consider it at $\beta=1$ (Nishimori's
263: temperature~\cite{Rujan93,Nishimori93,Iba99,nishimori_book})
264: in order to obtain the true posterior.
265: The Hamiltonian $\cH_\gamma(\ttau;\yy,\JJ)$ is
266: defined as
267: \begin{eqnarray}
268: \cH_\gamma(\ttau;\yy,\JJ)
269: &=&
270: -\gamma\sum_{\mu=1}^{M-N}
271: \biggl(J_\mu\prod_{j\in\cL(\mu)}\tau_j-1\biggr)
272: \nonumber\\
273: &&{}-\sum_{j=1}^M\log p(\tau_j y_j),
274: \end{eqnarray}
275: The channel characteristics enters into the Hamiltonian
276: as the term $\log p(\tau_jy_j)$ which,
277: by noting that $\tau_j$ takes $\pm1$,
278: can be rewritten as
279: \begin{equation}
280: \log p(\tau_jy_j)=
281: \tau_j\frac{1}{2}\log\frac{p(y_j)}{p(-y_j)}
282: +\frac{1}{2}\log p(y_j)p(-y_j).
283: \end{equation}
284: From this expression it immediately follows that it is the
285: log-likelihood ratio $h_j\equiv(1/2)\log(p(y_j)/p(-y_j))$ of the
286: channel noise $y_j$ which serves as the external field acting on site
287: $j$, and that the channel characteristics defines the field
288: distribution. Analyzing the effect of having different communication
289: channels on the code properties, therefore reduces to
290: investigating the effect of different field distributions on the physical
291: properties of the system. The field distributions $p(h)$ for various
292: channel models are as follows:
293: \begin{itemize}
294: \item BSC:
295: \begin{eqnarray}
296: p_{\BSC}(h)&=&(1-p)\delta\biggl(h-\frac{1}{2}\log\frac{1-p}{p}\biggr)
297: \nonumber\\
298: &&{}+p\delta\biggl(h+\frac{1}{2}\log\frac{1-p}{p}\biggr)
299: \end{eqnarray}
300: \item BIAWGNC:
301: \begin{equation}
302: p_{\BIAWGNC}(h)=\sqrt{\frac{\sigma^2}{2\pi}}
303: e^{-(h-\sigma^{-2})^2/2\sigma^{-2}}
304: \end{equation}
305: \item BILC:
306: \begin{eqnarray}
307: p_{\BILC}(h)&=&\frac{1}{2}\delta(h-\lambda^{-1})
308: +\frac{e^{-2\lambda^{-1}}}{2}\delta(h+\lambda^{-1})
309: \nonumber\\
310: &&{}+\chi[-\lambda^{-1}<h<\lambda^{-1}]
311: \frac{1}{2}e^{h-\lambda^{-1}},
312: \end{eqnarray}
313: where $\chi[X]$ is the indicator function,
314: taking 1 when $X$ is true and 0 otherwise.
315: \end{itemize}
316: \begin{figure}
317: \centering
318: \begin{minipage}{45mm}
319: \centering
320: \includegraphics[width=45mm]{fd-bsc.eps}\\
321: (a)\ BSC
322: \end{minipage}\ %
323: \begin{minipage}{45mm}
324: \centering
325: \includegraphics[width=45mm]{fd-biawgnc.eps}\\
326: (b)\ BIAWGNC
327: \end{minipage}\ %
328: \begin{minipage}{45mm}
329: \centering
330: \includegraphics[width=45mm]{fd-bilc.eps}\\
331: (c)\ BILC
332: \end{minipage}\ %
333: \caption{Field distributions corresponding to various BIOS channels.}
334: \label{fig:fd}
335: \end{figure}
336: Sketches of these field distributions are given in Fig.~\ref{fig:fd}.
337:
338: We assume that the free energy of the system is self-averaging, that
339: is,
340: \begin{equation}
341: f=-\frac{1}{\beta}\lim_{M\to\infty}M^{-1}
342: \langle\log Z\rangle_{A,\yy},
343: \end{equation}
344: and evaluate the average $\langle\cdot\rangle_{A,\yy}$ over the
345: received signal $\yy$ and the randomness of the parity-check matrix
346: $A$ using the replica method,
347: \begin{equation}
348: f=-\frac{1}{\beta}\lim_{M\to\infty}
349: \lim_{n\to0}M^{-1}
350: \frac{\partial}{\partial n}\log\langle Z^n\rangle_{A,\yy}.
351: \end{equation}
352: In calculating the free energy, we perform the gauge transformation
353: $\tau_j\to\zeta_j\tau_j$, $y_j\to\zeta_j y_j$. The average over $\yy$
354: can be taken with respect to $\prod_{j=1}^Mp(y_j)$ after having
355: performed the gauge transformation. We have to introduce a random
356: tensor to take average over $A$.
357:
358: Following basically the same procedure as in~\cite{MKSV00} and
359: exchanging the order of the two limits, taking the limit $M\to\infty$
360: first, one obtains
361: \begin{equation}
362: f=-\frac{1}{\beta}\lim_{n\to0}\frac{\partial}{\partial n}
363: \Extr_{\qq,\hat\qq}\biggl[
364: \frac{C}{K}\cG_1(\qq)-\cG_2(\qq,\hat\qq)+\cG_3(\hat\qq)
365: \biggr],
366: \end{equation}
367: where
368: \begin{eqnarray}
369: \cG_1(\qq)&\equiv&\log
370: \sum_{m=0}^n\sum_{\langle\alpha_1\cdots\alpha_m\rangle}
371: q_{\alpha_1\cdots\alpha_m}^K-n\log2,
372: \nonumber\\
373: \cG_2(\qq,\hat\qq)&\equiv&
374: \sum_{m=0}^n\sum_{\langle\alpha_1\cdots\alpha_m\rangle}
375: q_{\alpha_1\cdots\alpha_m}
376: \hat q_{\alpha_1\cdots\alpha_m},
377: \nonumber\\
378: \cG_3(\hat\qq)&\equiv&
379: \log\Biggl[
380: \sum_{\tau^1,\ldots,\tau^n}
381: \left\langle\prod_{\alpha=1}^np(\tau^\alpha y)\right\rangle_y
382: \nonumber\\
383: &&%\hphantom{\log}
384: \times\frac{1}{C!}
385: \biggl(
386: \sum_{m=0}^n\sum_{\langle\alpha_1\cdots\alpha_m\rangle}
387: \hat q_{\alpha_1\cdots\alpha_m}
388: \tau^{\alpha_1}\cdots\tau^{\alpha_m}
389: \biggr)^C
390: \Biggr].
391: \nonumber\\
392: \end{eqnarray}
393:
394: To proceed further we adopt the replica-symmetric (RS) ansatz and let
395: \begin{equation}
396: \label{eq:RS}
397: q_{\alpha_1\cdots\alpha_m}=q_0\int u^m\pi(u)\,du,
398: \quad
399: \hat q_{\alpha_1\cdots\alpha_m}=\hat q_0\int\hat u^m\hat\pi(\hat u)\,d\hat u.
400: \end{equation}
401: We will use the following simplifying notation.
402: \begin{equation}
403: \pi^K(\uu)\,d\uu\equiv\prod_{j=1}^K\pi(u_j)\,du_j
404: \end{equation}
405: The replica-symmetric free energy $f^\RS$ becomes
406: \begin{eqnarray}
407: f^\RS&=&\frac{1}{\beta}\Extr_{\pi,\hat\pi}
408: \Biggl\{
409: \frac{C}{K}\log 2
410: \nonumber\\
411: &+&C\iint\log(1+u\hat u)\,\pi(u)\,\hat\pi(\hat u)\,du\,d\hat u
412: \nonumber\\
413: &-&
414: \frac{C}{K}\int
415: \log\Bigl(1+\prod_{j=1}^Ku_j\Bigr)
416: \,\pi^K(\uu)\,d\uu
417: \nonumber\\
418: &-&\int
419: \biggl\langle
420: \log\biggl[p(y)\prod_{l=1}^C(1+\hat u_l)
421: +p(-y)\prod_{l=1}^C(1-\hat u_l)\biggr]
422: \biggr\rangle_y
423: \nonumber\\
424: &&\hphantom{\int}\times
425: \hat\pi^C(\hat\uu)\,d\hat\uu
426: \Biggr\},
427: \end{eqnarray}
428: in which $q_0$ and $\hat q_0$ have been eliminated
429: using the extremization condition $q_0\hat q_0=C$.
430: Heuristic construction of a sufficient condition
431: to the extremization problem
432: with respect to $\pi$ and $\hat\pi$ is possible,
433: and it gives the following saddle-point equations.
434: \begin{eqnarray}
435: \label{eq:sp-gal}
436: \pi(u)
437: &=&\int
438: \biggl\langle\delta\biggl[
439: u-\tanh\biggl(h(y)
440: +\sum_{l=1}^{C-1}\tanh^{-1}\hat u_l\biggr)
441: \biggr]\biggr\rangle_y
442: \nonumber\\
443: &&\hphantom{int}\times
444: \hat\pi^{C-1}(\hat\uu)\,d\hat\uu
445: \nonumber\\
446: \hat\pi(\hat u)
447: &=&\int
448: \delta\biggl(\hat u-\prod_{j=1}^{K-1}u_j\biggr)
449: \,\pi^{K-1}(\uu)\,d\uu
450: \end{eqnarray}
451: We have let
452: \begin{equation}
453: h(y)\equiv\frac{1}{2}\log\frac{p(y)}{p(-y)}.
454: \end{equation}
455: The performance of the code is quantified by
456: the overlap $m=M^{-1}\sum_{k=1}^M\zeta_j\langle\tau_j\rangle$,
457: which is given as
458: \begin{equation}
459: m=\int \sign(z)\,P(z)\,dz,
460: \end{equation}
461: where
462: \begin{eqnarray}
463: P(z)
464: &=&\int
465: \biggl\langle\delta\biggl[
466: z-\tanh\biggl(h(y)
467: +\sum_{l=1}^C\tanh^{-1}\hat u_l\biggr)
468: \biggr]\biggr\rangle_y
469: \nonumber\\
470: &&\hphantom{\int}\times
471: \hat\pi^C(\hat\uu)\,d\hat\uu.
472: \end{eqnarray}
473:
474:
475: \subsection{MN code}
476: \label{sec:rep-MN}
477:
478: The decoding problem for the MN code is
479: to find $\SS$ and $\ttau$ which are
480: the best suitable in view of the received signal $\yy$
481: among the sets of $\SS$ and $\ttau$
482: satisfying the parity-check equation
483: ($C_s\SS+C_n\ttau=C_s\xxi+C_n\zzeta\mod 2$
484: if written in the $\{0,1\}$-notation).
485: Defining the $\mu$th component of the check $\JJ$ as
486: \begin{equation}
487: J_\mu=\prod_{j\in\cL_s(\mu)}\xi_j
488: \prod_{l\in\cL_n(\mu)}\zeta_l,
489: \end{equation}
490: where
491: \begin{equation}
492: \cL_s(\mu)=\{j|(C_s)_{\mu j}=1\},
493: \quad
494: \cL_n(\mu)=\{l|(C_n)_{\mu l}=1\},
495: \end{equation}
496: the posterior probability of $\SS$ and $\ttau$
497: conditioned on the received signal $\yy$
498: and the check $\JJ$ is given by
499: \begin{equation}
500: P_\gamma(\SS,\ttau|\yy,\JJ)
501: =\frac{1}{Z}\exp\bigl[-\beta\cH_\gamma(\SS,\ttau;\yy,\JJ)\bigr],
502: \end{equation}
503: in the limit $\gamma\to\infty$ and at $\beta=1$,
504: where the Hamiltonian $\cH_\gamma(\SS,\ttau;\yy,\JJ)$
505: is defined as
506: \begin{eqnarray}
507: \label{eq:Hamiltonian-MN}
508: \cH_\gamma(\SS,\ttau;\yy,\JJ)
509: &=&-\gamma\sum_{\mu=1}^M\biggl(J_\mu
510: \prod_{j\in\cL_s(\mu)}S_j
511: \prod_{l\in\cL_n(\mu)}\tau_l
512: -1\biggr)
513: \nonumber\\
514: &&{}-F_s\sum_{j=1}^NS_j-\sum_{l=1}^M\log p(\tau_l y_l),
515: \end{eqnarray}
516: where $F_s$ is a parameter representing the bias of the information
517: vector $\xxi$ in such a way that $P(\xi_j=\pm1)=(1\pm\tanh F_s)/2$
518: holds. The form of Eq.~(\ref{eq:Hamiltonian-MN}) clearly shows that
519: the channel characteristics again acts as random field on
520: $\{\tau_l\}$, where the log likelihood ratio gives the actual value of
521: the field.
522:
523: The replica calculation can be done
524: along the same way as in the case of the Gallager code.
525: We have performed the gauge transformation
526: $S_j\to\xi_j S_j$, $\tau_j\to\zeta_j\tau_j$,
527: and $y_j\to\zeta_j\tau_j$.
528: The free energy $f$ becomes
529: \begin{eqnarray}
530: \label{eq:MN-free-energy}
531: f&=&-\frac{1}{\beta}\lim_{n\to0}\frac{\partial}{\partial n}
532: \Extr_{\qq,\hat\qq,\rr,\hat\rr}\biggl[
533: \frac{C}{K}\cG_1(\qq,\rr)
534: \nonumber\\
535: &&\hphantom{-\frac{1}{\beta}\lim_{n\to0}\frac{\partial}{\partial n}
536: \Extr_{\qq,\hat\qq,\rr,\hat\rr}\biggl[}
537: -\cG_2(\qq,\hat\qq,\rr,\hat\rr)+\cG_3(\hat\qq,\hat\rr)
538: \biggr],
539: \nonumber\\
540: \end{eqnarray}
541: where
542: \begin{eqnarray}
543: \cG_1(\qq,\rr)&\equiv&\log
544: \sum_{m=0}^n\sum_{\langle\alpha_1\cdots\alpha_m\rangle}
545: q_{\alpha_1\cdots\alpha_m}^K
546: r_{\alpha_1\cdots\alpha_m}^L
547: \nonumber\\
548: &&{}-n\log2,
549: \nonumber\\
550: \cG_2(\qq,\hat\qq,\rr,\hat\rr)&\equiv&
551: \sum_{m=0}^n\sum_{\langle\alpha_1\cdots\alpha_m\rangle}
552: q_{\alpha_1\cdots\alpha_m}
553: \hat q_{\alpha_1\cdots\alpha_m}
554: \nonumber\\
555: &&{}+\frac{M}{N}\sum_{m=0}^n\sum_{\langle\alpha_1\cdots\alpha_m\rangle}
556: r_{\alpha_1\cdots\alpha_m}
557: \hat r_{\alpha_1\cdots\alpha_m},
558: \nonumber\\
559: \end{eqnarray}
560: and
561: \begin{eqnarray}
562: \cG_3(\hat\qq,\hat\rr)
563: &\equiv&
564: \log\Biggl[
565: \sum_{S^1,\ldots,S^n}
566: \Bigl\langle e^{F_s\sum_{\alpha=1}^n\xi S^\alpha}
567: \Bigr\rangle_\xi
568: \nonumber\\
569: &&{}\times\frac{1}{C!}
570: \biggl(
571: \sum_{m=0}^n\sum_{\langle\alpha_1\cdots\alpha_m\rangle}
572: \!\!
573: \hat q_{\alpha_1\cdots\alpha_m}
574: S^{\alpha_1}\cdots S^{\alpha_m}
575: \biggr)^C
576: \Biggr]
577: \nonumber\\
578: &+&\frac{M}{N}
579: \log\Biggl[
580: \sum_{\tau^1,\ldots,\tau^n}
581: \biggl\langle\prod_{\alpha=1}^np(\tau^\alpha y)\biggr\rangle_y
582: \nonumber\\
583: &&{}\times\frac{1}{L!}
584: \biggl(
585: \sum_{m=0}^n\sum_{\langle\alpha_1\cdots\alpha_m\rangle}
586: \!\!
587: \hat r_{\alpha_1\cdots\alpha_m}
588: \tau^{\alpha_1}\cdots\tau^{\alpha_m}
589: \biggr)^L
590: \Biggr].
591: \nonumber\\
592: \end{eqnarray}
593:
594: We adopt the RS ansatz as before, under which we have
595: \begin{equation}
596: \label{eq:RS-MN}
597: r_{\alpha_1\cdots\alpha_m}=r_0\int v^m\rho(v)\,dv,
598: \quad
599: \hat r_{\alpha_1\cdots\alpha_m}=\hat r_0\int\hat v^m
600: \hat\rho(\hat v)\,d\hat v,
601: \end{equation}
602: in addition to Eq.~(\ref{eq:RS}).
603: The replica-symmetric free energy $f^\RS$ becomes
604: \begin{eqnarray}
605: f^\RS&=&\frac{1}{\beta}\Extr_{\pi,\hat\pi,\rho,\hat\rho}\Biggl\{
606: \frac{C}{K}\log2
607: \nonumber\\
608: &+&C\iint\log(1+u\hat u)\,\pi(u)\,\hat\pi(\hat u)\,du\,d\hat u
609: \nonumber\\
610: &+&\frac{CL}{K}\iint\log(1+v\hat v)\,\rho(v)\,\hat\rho(\hat v)\,dv\,d\hat v
611: \nonumber\\
612: &-&\frac{C}{K}\iint
613: \log\biggl(1+\prod_{k=1}^Ku_k\prod_{l=1}^Lv_l\biggr)
614: \nonumber\\
615: &&\hphantom{\frac{C}{K}\iint}
616: \times
617: \pi^K(\uu)\,d\uu
618: \,\rho^L(\vv)\,d\vv
619: \nonumber\\
620: &-&\int
621: \biggl\langle
622: \log\biggl[\sum_{S=\pm1}e^{F_s\xi S}
623: \prod_{k=1}^C(1+S\hat u_k)\biggr]
624: \biggr\rangle_\xi
625: \hat\pi^C(\hat\uu)\,d\hat\uu
626: \nonumber\\
627: &-&\frac{C}{K}\int
628: \biggl\langle
629: \log\biggl[\sum_{\tau=\pm1}p(\tau y)
630: \prod_{l=1}^L(1+\tau\hat v_l)\biggr]
631: \biggr\rangle_y
632: \nonumber\\
633: &&\hphantom{\frac{C}{K}\int}\times
634: \hat\rho^L(\hat\vv)\,d\hat\vv
635: \Biggr\},
636: \end{eqnarray}
637: in which $q_0$, $\hat q_0$, $r_0$, and $\hat r_0$ have been
638: eliminated using the extremization conditions,
639: $q_0\hat q_0=C$ and $r_0\hat r_0=L$.
640:
641: Construction of a heuristic solution
642: to the extremization problem can be done in the same manner,
643: which yields the following saddle-point equations:
644: \begin{eqnarray}
645: \pi(u)&=&\int
646: \biggl\langle\delta\biggl[
647: u-\tanh\biggl(F_s\xi +\sum_{l=1}^{C-1}\tanh^{-1}\hat u_l\biggr)
648: \biggr]\biggr\rangle_\xi
649: \nonumber\\
650: &&\hphantom{\int}\times
651: \hat\pi^{C-1}(\hat\uu)\,d\uu
652: \nonumber\\
653: \hat\pi(\hat u)&=&\iint
654: \delta\biggl(
655: \hat u-\prod_{k=1}^{K-1}u_k\prod_{l=1}^Lv_l
656: \biggr)
657: \,\pi^{K-1}(\uu)\,d\uu
658: \,\rho^L(\vv)\,d\vv
659: \nonumber\\
660: \rho(v)&=&\int
661: \biggl\langle\delta\biggl[
662: v-\tanh\biggl(h(y)
663: +\sum_{l=1}^{L-1}\tanh^{-1}\hat v_l\biggr)
664: \biggr]\biggr\rangle_y
665: \nonumber\\
666: &&\hphantom{\int}\times
667: \hat\rho^{L-1}(\hat\vv)\,d\hat\vv
668: \nonumber\\
669: \hat\rho(\hat v)&=&\iint
670: \delta\biggl(\hat v-\prod_{k=1}^Ku_k\prod_{l=1}^{L-1}v_l\biggr)
671: \,\pi^K(\uu)\,d\uu
672: \,\rho^{L-1}(\vv)\,d\vv
673: \nonumber\\
674: \end{eqnarray}
675:
676: The overlap is then evaluated by
677: \begin{equation}
678: m=\int \sign(z)\,P(z)\,dz,
679: \end{equation}
680: where
681: \begin{eqnarray}
682: P(z)&=&\int
683: \biggl\langle\delta\biggl[
684: z-\tanh\biggl(F_s\xi +\sum_{l=1}^C\tanh^{-1}\hat u_l\biggr)
685: \biggr]\biggr\rangle_\xi
686: \nonumber\\
687: &&\hphantom{\int}\times
688: \hat\pi^C(\hat\uu)\,d\hat\uu.
689: \end{eqnarray}
690:
691: It is worthwhile mentioning that, when the message is unbiased
692: ($F_s=0$) and $K$ is even, saddle-point solutions have the following
693: symmetry: For each solution $\{\pi(u),\hat\pi(\hat
694: u),\rho(v),\hat\rho(\hat v)\}$ there is another solution
695: $\{\pi(-u),\hat\pi(-\hat u),\rho(v),\hat\rho(\hat v)\}$. The latter
696: has the same overlap as that of the former with the opposite sign.
697:
698:
699: \section{Results}
700: \label{sec:res}
701:
702: \subsection{Gallager code}
703: \label{sec:res-gal}
704:
705: \subsubsection{Analytical solutions}
706:
707: Of particular interest is the ferromagnetic state,
708: which corresponds to an error-free communication.
709: One can see that the assertion
710: \begin{equation}
711: \pi(u)=\delta(u-1),\quad\hat\pi(\hat u)=\delta(\hat u-1)
712: \end{equation}
713: always satisfies the saddle-point equation~(\ref{eq:sp-gal})
714: irrespective of the values of $K$ and $C$
715: (provided that $K,C\ge2$),
716: thereby providing a solution.
717: The overlap and the free energy of the solution
718: at $\beta=1$ are $m_\ferro=1$
719: and $f_\ferro=-\langle\log p(y)\rangle_y$, respectively.
720: One can therefore identify this as the ferromagnetic solution.
721:
722: Another solution, which can be found in the limit $K\to\infty$, is the
723: sub-optimal ferromagnetic solution
724: \begin{equation}
725: \pi(u)=\left\langle\delta\bigl[
726: u-\tanh h(y)
727: \bigl]\right\rangle_y,
728: \quad\hat\pi(\hat u)=\delta(\hat u),
729: \end{equation}
730: for which
731: \begin{equation}
732: m_\subopt=\bigl\langle\sign\bigl[p(y)-p(-y)\bigr]\bigr\rangle_y
733: \end{equation}
734: and
735: \begin{equation}
736: f_\subopt=\frac{C}{K}\log2
737: -\bigl\langle\log\bigl[p(y)+p(-y)\bigr]\bigr\rangle_y.
738: \end{equation}
739: The difference of the free energy is expressed as
740: \begin{equation}
741: f_\subopt-f_\ferro=\C-R\log2,
742: \end{equation}
743: where $\C$ is the channel capacity of the BIOS channel defined as
744: \begin{equation}
745: \C=\log 2-\bigl\langle\log\bigl[p(y)+p(-y)\bigr]\bigr\rangle_y
746: +\bigl\langle\log p(y)\bigr\rangle_y.
747: \end{equation}
748: This proves that the thermodynamic transition between the
749: ferromagnetic and sub-optimal ferromagnetic solutions (no other
750: solution has been identified in this case) occurs at the theoretical
751: limit. This means that the maximum rate $R_{\rm max}$, up to which
752: error-free communication is theoretically possible, asymptotically
753: achieves the theoretical limit as $K\to\infty$. This result has been
754: known for BSC channel~\cite{VSK00,VSK02} in the physics literature and
755: is in agreement with results reported in the information theory
756: literature~\cite{MacKay99}. The current result is an extension to the
757: case of a general BIOS channel.
758:
759:
760: \subsubsection{Numerical solutions of saddle-point equations}
761:
762: In finite-$K$ cases no simple analytical solution exists
763: other than the ferromagnetic one, so one has
764: to solve the saddle-point equations numerically. We have done it for
765: BIAWGNC and BILC. The dependence of the overlap $m$ on the noise
766: level $d$ ($\sigma^2$ for BIAWGNC, and $\lambda$ for BILC) is
767: qualitatively the same as that observed in BSC: For $K\ge3$ the
768: ferromagnetic solution is locally stable over the whole range of noise
769: levels. At $d=d_s$, another solution with $m<1$ appears, which
770: defines the spinodal point. At a higher noise level $d=d_t>d_s$
771: thermodynamic transition takes place, beyond which the ferromagnetic
772: solution with $m=1$ becomes metastable (see Fig.~\ref{fig:pd-gal}).
773: Table~\ref{tab:Gal-BIAWGNC} summarizes the results for the BIAWGNC
774: case, showing the spinodal point $\sigma_s^2$ (the value of
775: the variance at which new, non ferromagnetic, solutions emerge), the
776: thermodynamic transition point $\sigma_t^2$ (at which the
777: thermodynamic transition occurs), and $\sigma_0^2$, the
778: information-theoretic upper bound of the variance allowing error-free
779: communication.
780:
781: \begin{figure}
782: \centering
783: \includegraphics[width=80mm]{ph-gal.eps}
784: \caption{Noise-overlap diagram for Gallager code. Thick solid
785: lines stand for the stable state, thin solid lines for metastable
786: state, and broken lines for unstable states. The ferromagnetic
787: solution is characterized by the $m=1$ solution, while $m<1$
788: defines the suboptimal ferromagnetic solution.} \label{fig:pd-gal}
789: \end{figure}
790:
791: Table~\ref{tab:Gal-BILC} summarizes the results for the BILC case,
792: showing the values of
793: the spinodal point $\lambda_s$, the thermodynamic transition point
794: $\lambda_t$, and the information-theoretic upper bound $\lambda_0$.
795:
796: It should be noted that
797: the results for the spinodal point
798: agree well with the results
799: obtained by the density evolution approach~\cite{RU01},
800: as expected, since the saddle-point equations
801: by the replica analysis
802: happen to coincide with the time evolution equations
803: in the density evolution.
804:
805: \begin{table}
806: \caption{The variances $\sigma_s^2$ and $\sigma_t^2$
807: at the spinodal point and thermodynamic transition, respectively,
808: for the BIAWGNC for various code parameters;
809: $\sigma_0^2$, denoting the information-theoretical upper bound
810: for error-free communication, is also shown.}
811: \label{tab:Gal-BIAWGNC}
812: \begin{ruledtabular}
813: \begin{tabular}{rrdddd}
814: $C$ & $K$ & \mbox{$R$}
815: & \mbox{$\sigma_s^2$} & \mbox{$\sigma_t^2$}
816: & \mbox{$\sigma_0^2$} \\\hline
817: 3 & 6 & 0.5 & 0.775 & 0.899 & 0.958 \\
818: 4 & 8 & 0.5 & 0.701 & 0.943 & 0.958 \\
819: 5 & 10 & 0.5 & 0.629 & 0.952 & 0.958 \\
820: 3 & 5 & 0.4 & 1.017 & 1.253 & 1.321 \\
821: 4 & 6 & 0.333 & 1.020 & 1.666 & 1.681 \\
822: 3 & 4 & 0.25 & 1.598 & 2.325 & 2.401
823: \end{tabular}
824: \end{ruledtabular}
825: \end{table}
826:
827: \begin{table}
828: \caption{The parameter values $\lambda_s$ and $\lambda_t$
829: at the spinodal point and thermodynamic transition, respectively,
830: for the BILC with various code parameters;
831: $\lambda_0$, denoting the information-theoretical upper bound
832: for error-free communication, is also shown.}
833: \label{tab:Gal-BILC}
834: \begin{ruledtabular}
835: \begin{tabular}{rrdddd}
836: $C$ & $K$ & \mbox{$R$}
837: & \mbox{$\lambda_s$} & \mbox{$\lambda_t$}
838: & \mbox{$\lambda_0$} \\\hline
839: 3 & 6 & 0.5 & 0.651 & 0.712 & 0.752 \\
840: 4 & 8 & 0.5 & 0.618 & 0.741 & 0.752 \\
841: 5 & 10 & 0.5 & 0.581 & 0.746 & 0.752 \\
842: 3 & 5 & 0.4 & 0.773 & 0.875 & 0.914 \\
843: 4 & 6 & 0.333 & 0.782 & 1.045 & 1.055 \\
844: 3 & 4 & 0.25 & 1.018 & 1.260 & 1.298
845: \end{tabular}
846: \end{ruledtabular}
847: \end{table}
848:
849: %+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
850:
851: \subsection{MN code}
852: \label{sec:res-MN}
853:
854: \subsubsection{Analytical solutions}
855:
856: In the following we restrict our discussion of the MN code to the
857: unbiased case $F_s=0$. The ferromagnetic solution, corresponding to
858: the error-free communication, can be constructed for the MN code with
859: $L\ge2$. (In fact, in the case $L=1$ the matrix $C_n$ reduces to a
860: simple permutation matrix, so that we have to estimate each element of
861: noise separately. This case is not at all interesting and therefore
862: we will not discuss it any more.) It is given by
863: \begin{eqnarray}
864: \label{eq:MN-para-sol}
865: &&\pi(u)=\delta(u-1),\quad
866: \hat\pi(\hat u)=\delta(\hat u-1),
867: \nonumber\\
868: &&\rho(v)=\delta(v-1),\quad
869: \hat\rho(\hat v)=\delta(\hat v-1),
870: \end{eqnarray}
871: for which $m_\ferro=1$ and
872: \begin{equation}
873: f_\ferro=-\frac{C}{K}\bigl\langle\log p(y)\bigr\rangle_y.
874: \end{equation}
875: The MN code has the following paramagnetic solution
876: for $K\ge2$:
877: \begin{equation}
878: \begin{array}{ll}
879: \pi(u)=\delta(u),
880: &\hat\pi(\hat u)=\delta(\hat u),
881: \\
882: \rho(v)=\left\langle\delta\bigl[
883: v-\tanh h(y)
884: \bigl]\right\rangle_y,\quad{}
885: &\hat\rho(\hat v)=\delta(\hat v),
886: \end{array}
887: \end{equation}
888: which yields $m_\para=0$ and
889: \begin{equation}
890: f_\para=\biggl(\frac{C}{K}-1\biggr)\log2
891: -\frac{C}{K}\bigl\langle\log\bigl[p(y)+p(-y)\bigr]\bigr\rangle_y.
892: \end{equation}
893: Again, since
894: \begin{equation}
895: f_\para-f_\ferro=\frac{C}{K}(\C-R\log2)
896: \end{equation}
897: holds, we conclude that for the MN code the maximum rate $R_{\rm
898: max}$, theoretically allowing error-free communication, achieves
899: the theoretical limit as long as $K\ge2$, $L\ge2$, provided that
900: there is no locally stable solution other than the ferromagnetic and
901: paramagnetic solutions. This result is an extension of the result
902: reported in~\cite{KMS00,MKSV00} to the case of a general BIOS
903: channel.
904:
905: It should be noted that
906: the paramagnetic solution~(\ref{eq:MN-para-sol}) is also valid
907: in the limit $L\to\infty$ for the case $K=1$.
908: This means that the above-mentioned result also holds
909: for the case of $K=1$ asymptotically in the limit $L\to\infty$.
910:
911:
912: \subsubsection{Numerical solutions of saddle-point equations}
913:
914: In order to explore solutions other than the ferromagnetic
915: and paramagnetic solutions,
916: we have to solve the saddle-point equations numerically.
917: We have done it for the BIAWGNC and BILC cases.
918: We observed qualitatively the same characteristics
919: as those reported in~\cite{MKSV00}.
920:
921: The obtained numerical results suggest that the qualitative physical
922: properties are categorized into three types according to the $K$
923: value: cases with $K=1$, $K=2$ and $K\ge3$, whereas it is only
924: affected quantitatively by the values of $C$ and $L$, as described in
925: the following.
926:
927: The structure of noise-overlap diagram for the MN code with $K=1$ is
928: qualitatively the same as that for Gallager code (see
929: Fig.~\ref{fig:pd-gal}): At very low noise level only the ferromagnetic
930: solution with $m=1$ exists. At a certain noise level $d=d_s$ another
931: metastable solution with $m<1$ appears, and it becomes dominant beyond
932: $d=d_t>d_s$. Since the latter solution is obtained only numerically,
933: there is no guarantee that the thermodynamical transition $d_t$ is
934: equal to the information-theoretical limit $d_0$. Numerical results
935: show that in general $d_t$ {\em is} smaller than $d_0$: However, it is
936: also observed that, for fixed $C$, increasing $L$ makes $d_s$ smaller
937: and $d_t$ larger, the latter of which approaches the
938: information-theoretical limit $d_0$ as $L\to\infty$,
939: as discussed at the end of the previous subsection. Even for finite
940: $L$ the value of $d_t$ may be numerically very close to $d_0$,
941: especially when the rate $R$ is small. These properties have already
942: been reported for the BSC case~\cite{MKSV00}, so that our finding
943: implies that they also hold for the BIAWGNC and BILC cases, revealing
944: some sort of universality.
945:
946: \begin{table}
947: \caption{The variances $\sigma_s^2$, $\sigma_t^2$, and $\sigma_b^2$
948: at the spinodal point and thermodynamic transition,
949: and at bifurcation of paramagnetic solution, respectively,
950: for $(K,C,L)$-MN codes over the BIAWGNC and various code parameters;
951: $\sigma_0^2$, denoting the information-theoretical upper bound
952: for error-free communication, is also shown.}
953: \label{tab:MN-BIAWGNC}
954: \begin{ruledtabular}
955: \begin{tabular}{rrrddddd}
956: $K$ & $C$ & $L$ & \mbox{$R$}
957: & \mbox{$\sigma_s^2$} & \mbox{$\sigma_t^2$}
958: & \mbox{$\sigma_b^2$} & \mbox{$\sigma_0^2$} \\\hline
959: 1 & 2 & 3 & 0.5 & 0.775 & 0.901 & - & 0.958 \\
960: 1 & 2 & 4 & 0.5 & 0.703 & 0.944 & - & 0.958 \\
961: 1 & 2 & 5 & 0.5 & 0.630 & 0.955 & - & 0.958 \\
962: 1 & 3 & 2 & 0.333 & 1.338 & 1.423 & - & 1.681 \\
963: 1 & 3 & 3 & 0.333 & 1.129 & 1.659 & - & 1.681 \\
964: 1 & 3 & 4 & 0.333 & 0.913 & 1.672 & - & 1.681 \\
965: \noalign{\vspace{4pt}}
966: 2 & 3 & 2 & 0.667 & 0.536 & 0.587 & 0.612 & 0.588 \\
967: 2 & 3 & 3 & 0.667 & 0.430 & 0.588 & 0.459 & 0.588 \\
968: 2 & 3 & 4 & 0.667 & 0.368 & 0.588 & 0.385 & 0.588 \\
969: 2 & 4 & 2 & 0.5 & 0.809 & 0.958 & 0.919 & 0.958 \\
970: 2 & 5 & 2 & 0.4 & 1.039 & 1.321 & 1.175 & 1.321
971: \end{tabular}
972: \end{ruledtabular}
973: \end{table}
974:
975: \begin{table}
976: \caption{The parameter values $\lambda_s$, $\lambda_t$, and $\lambda_b$
977: at the spinodal point and thermodynamic transition,
978: and at bifurcation of paramagnetic solution, respectively,
979: for $(K,C,L)$-MN codes over the BILC and various code parameters;
980: $\lambda_0$, denoting the information-theoretical upper bound
981: for error-free communication, is also shown.}
982: \label{tab:MN-BILC}
983: \begin{ruledtabular}
984: \begin{tabular}{rrrddddd}
985: $K$ & $C$ & $L$ & \mbox{$R$}
986: & \mbox{$\lambda_s$} & \mbox{$\lambda_t$}
987: & \mbox{$\lambda_b$} & \mbox{$\lambda_0$} \\\hline
988: 1 & 2 & 3 & 0.5 & 0.652 & 0.714 & - & 0.752 \\
989: 1 & 2 & 4 & 0.5 & 0.619 & 0.740 & - & 0.752 \\
990: 1 & 2 & 5 & 0.5 & 0.582 & 0.748 & - & 0.752 \\
991: 1 & 3 & 2 & 0.333 & 0.903 & 0.934 & - & 1.055 \\
992: 1 & 3 & 3 & 0.333 & 0.831 & 1.040 & - & 1.055 \\
993: 1 & 3 & 4 & 0.333 & 0.735 & 1.051 & - & 1.055 \\
994: \noalign{\vspace{4pt}}
995: 2 & 3 & 2 & 0.667 & 0.525 & 0.551 & 0.597 & 0.553 \\
996: 2 & 3 & 3 & 0.667 & 0.464 & 0.553 & 0.493 & 0.553 \\
997: 2 & 3 & 4 & 0.667 & 0.419 & 0.553 & 0.437 & 0.553 \\
998: 2 & 4 & 2 & 0.5 & 0.689 & 0.751 & 0.771 & 0.752 \\
999: 2 & 5 & 2 & 0.4 & 0.807 & 0.914 & 0.894 & 0.914
1000: \end{tabular}
1001: \end{ruledtabular}
1002: \end{table}
1003:
1004: \begin{figure}
1005: \centering
1006: \begin{minipage}{75mm}
1007: \centering
1008: \includegraphics[width=75mm]{ph2a.eps}\\
1009: (a)
1010: \end{minipage}
1011: \quad
1012: \begin{minipage}{75mm}
1013: \centering
1014: \includegraphics[width=75mm]{ph2b.eps}\\
1015: (b)
1016: \end{minipage}
1017: \caption{Noise-overlap diagram for the cases with $K=2$. }
1018: \label{fig:bif-diag-K2}
1019: \end{figure}
1020:
1021: The noise-overlap diagram for the cases with $K=2$ has the general
1022: structure shown in Fig.~\ref{fig:bif-diag-K2}. The diagram is
1023: characterized by three transition points: the spinodal point
1024: $d_s$, the thermodynamic transition $d_t$, and the bifurcation point
1025: $d_b$. The order of the thermodynamic transition $d_t$ and the
1026: bifurcation point $d_b$ varies with the values of $C$ and $L$, so that
1027: the bifurcation pattern for the cases with $K=2$ is further divided
1028: into two sub-categories depending on the order of the two transitions:
1029: $d_s<d_b<d_t$ for the first group, and $d_s<d_t<d_b$ for the second
1030: group. The noise-overlap diagrams for these groups are illustrated in
1031: Fig.~\ref{fig:bif-diag-K2} (a) and (b), respectively. By the local
1032: stability analysis the bifurcation point $d_b$ is determined by
1033: \begin{equation}
1034: \label{eq:para-stability}
1035: \int v\rho(v)\,dv=(C-1)^{-1/L},
1036: \end{equation}
1037: which allows us to decide the type of bifurcation of a particular
1038: case. See the appendix for derivation of
1039: Eq.~(\ref{eq:para-stability}). As a result, we found that only a few
1040: cases with small values of $C$ and $L$ fall into the second category.
1041: The values of $C$ and $L$ for which the $(2,C,L)$-MN code fall into
1042: the second category depend on the channel characteristics; as far as
1043: we have observed, only the cases with $L=2$ fall into the second
1044: group. For the BIAWGNC case, the $(2,3,2)$-MN code is the only one
1045: instance, whereas for the BILC case, both $(2,3,2)$- and $(2,4,2)$-MN
1046: codes belong to this group. (For the BSC case, $(2,3,2)$-,
1047: $(2,4,2)$-, and $(2,5,2)$-MN codes belong to this group.) All the
1048: $(2,C,L)$-MN codes but those mentioned above are in the first group.
1049: For the cases in the second group,
1050: the thermodynamic transition $d_t$ must be less than
1051: the information-theoretic limit $d_0$: However, it turns out
1052: numerically that $d_t$ is very close to $d_0$.
1053:
1054: We observed that the noise-overlap diagram for the cases with $K\ge 3$
1055: is relatively simple for the BIAWGNC and BILC cases, just as in the
1056: BSC case (Fig.~\ref{fig:bif-diag-K3}): The ferromagnetic solution with
1057: $m=1$ (and its mirror image with $m=-1$ when $K$ is even) and the
1058: paramagnetic solution are the only stable solutions found, both of
1059: which are locally stable over the whole range of the noise level. The
1060: system exhibits a first-order transition at the information-theoretic
1061: limit $d_t$. We did not find any solutions other than the
1062: ferromagnetic and paramagnetic solutions.
1063:
1064: \begin{figure}
1065: \centering
1066: \includegraphics[width=75mm]{ph3.eps}
1067: \caption{Noise-overlap diagram for the cases with $K\ge3$.}
1068: \label{fig:bif-diag-K3}
1069: \end{figure}
1070:
1071:
1072: \section{Conclusions}
1073: \label{sec:conclusion}
1074:
1075: We have analyzed typical performance of LDPC codes
1076: over BIOS channel using statistical mechanics.
1077: We have shown for the case of LDPC codes
1078: that the log-likelihood ratio of the received signal
1079: serves as an external random field acting on each site,
1080: and that channel characteristics
1081: define the distribution of the random field.
1082: The Gallager and MN codes are analyzed,
1083: to find that the basic properties of these codes remain unchanged
1084: regardless of channel characteristics.
1085: In particular, it has been shown that
1086: these codes potentially saturate Shannon's limit
1087: asymptotically, as $K\to\infty$, for the Gallager code;
1088: and when $K,L\ge2$\ ---\ with a few exceptions with small $C$ and $L$
1089: values --- and asymptotically as $L\to\infty$ for $K=1$,
1090: for the MN code.
1091: Saddle-point solutions have also been numerically evaluated
1092: extensively for the cases of BIAWGNC and BILC channels, from which
1093: noise-overlap diagrams, as well as the transition and bifurcation
1094: points, have been characterized.
1095:
1096:
1097: \begin{acknowledgments}
1098: We would like to thank Yoshiyuki Kabashima for his helpful
1099: suggestions, Jort van Mourik for providing computer programs, and
1100: Nikos Skantzos for helpful discussions. Support from EPSRC research
1101: grant GR/N00562 is acknowledged.
1102: \end{acknowledgments}
1103:
1104:
1105: \appendix* % Only one appendix
1106:
1107: \section{Stability of paramagnetic solution for $K\ge2$}
1108:
1109: To probe the stability of paramagnetic solution, which exists for
1110: $K\ge2$, we analyze the stability with respect to $\qq$ and $\rr$
1111: only, and do not consider stability with respect to $\hat\qq$ and
1112: $\hat\rr$; these conjugate variables are subsidiary to their
1113: counterparts, $\qq$ and $\rr$, respectively, so that the former should
1114: not be considered as independent variables.
1115:
1116: Let $A$, $B$, $\ldots$ denote sets of replica indices such as
1117: $\langle\alpha_1\cdots\alpha_m\rangle$, $m\ge1$. We first evaluate
1118: the Hessian of the free energy (\ref{eq:MN-free-energy}) with respect
1119: to $4\times(2^n-1)$ variables $\{q_A,\hat q_A,r_A,\hat r_A\}$:
1120: \begin{equation}
1121: H=\left(
1122: \begin{array}{cc}
1123: \begin{array}{cc}
1124: H_{\qq \qq} & H_{\qq \hat\qq} \\
1125: H_{\qq \hat\qq} & H_{\hat\qq \hat\qq}
1126: \end{array}
1127: & \hbox{\Large$O$} \\
1128: \hbox{\Large$O$} &
1129: \begin{array}{cc}
1130: O & H_{\rr \hat\rr} \\
1131: H_{\rr \hat\rr} & H_{\hat\rr \hat\rr}
1132: \end{array}
1133: \end{array}\right),
1134: \end{equation}
1135: where
1136: \begin{eqnarray}
1137: \bigl(H_{\qq \qq}\bigr)_{AB}
1138: &=&\left\{
1139: \begin{array}{ll}
1140: 0 & (K\ge3)\\
1141: \displaystyle
1142: -\frac{C}{q_0^2}\left(\frac{r_A}{r_0}\right)^L\delta_{AB} & (K=2)
1143: \end{array}\right.
1144: \nonumber\\
1145: \bigl(H_{\qq\hat\qq}\bigr)_{AB}
1146: &=&\delta_{AB}
1147: \nonumber\\
1148: \bigl(H_{\hat\qq\hat\qq}\bigr)_{AB}
1149: &=&-\frac{C(C-1)}{\hat q_0^2}\delta_{AB}
1150: \nonumber\\
1151: \bigl(H_{\rr\hat\rr}\bigr)_{AB}&=&\frac{M}{N}\delta_{AB}
1152: \nonumber\\
1153: \bigl(H_{\hat\rr\hat\rr}\bigr)_{AB}
1154: &=&-\frac{M}{N}\frac{L(L-1)}{\hat r_0^2}\delta_{AB}
1155: \end{eqnarray}
1156: The block-diagonal structure of the Hessian
1157: allows us to decompose the stability problem
1158: into two, one with respect to $\qq$,
1159: and another with respect to $\rr$.
1160:
1161: Following the argument in the appendix of~\cite{Tanaka-ITpre}, one can say
1162: that the system is stable with respect to $\qq$ if the matrix
1163: $H_c\equiv H_{\qq\qq}
1164: -H_{\qq\hat\qq}\bigl(H_{\hat\qq\hat\qq}\bigr)^{-1}H_{\qq\hat\qq}$ is
1165: positive definite. This condition takes into account the fact that
1166: $\hat\qq$ depends on $\qq$. A corresponding statement holds for the
1167: stability with respect to $\rr$.
1168:
1169: The stability with respect to $\rr$ is straightforward, by noting that
1170: the matrix $H_{\hat\rr\hat\rr}$ is negative definite, which means that
1171: $H_c=-(M/N)^2\bigl(H_{\hat\rr\hat\rr}\bigr)^{-1}$ is positive
1172: definite.
1173:
1174: We consider the stability with respect to $\qq$. For $K\ge3$, we have
1175: $H_c=[\hat q_0^2/C(C-1)]I$, where $I$ is the identity matrix, so that
1176: the stability immediately follows, irrespective of the noise level of
1177: the channel. For $K=2$, the matrix $H_c$ is diagonal, and its $A$-th
1178: element is
1179: \begin{equation}
1180: \bigl(H_c\bigr)_{AA}=-\frac{C}{q_0^2}
1181: \left(\frac{r_A}{r_0}\right)^L+\frac{\hat q_0^2}{C(C-1)}.
1182: \end{equation}
1183: Using the equality which holds under the RS ansatz,
1184: \begin{equation}
1185: \frac{r_A}{r_0}
1186: =\int_{-1}^1 v^m\rho(v)\,dv,
1187: \end{equation}
1188: where $A=\alpha_1\cdots\alpha_m$,
1189: we have, as the stability condition,
1190: \begin{equation}
1191: E_m\equiv\int_{-1}^1 v^m\rho(v)\,dv<(C-1)^{-1/L}.
1192: \end{equation}
1193: for $m=1,\ldots,n$.
1194: Since it can be shown that $E_{2m-1}=E_{2m}$
1195: and $E_{2m}\ge E_{2m+2}$,
1196: the critical condition determining the stability is
1197: \begin{equation}
1198: E_1<(C-1)^{-1/L}.
1199: \end{equation}
1200:
1201: \begin{thebibliography}{99}
1202: %
1203: \bibitem{Shannon48}C. E. Shannon, Bell Syst. Tech. J. {\bf 27}, 379 (1948);
1204: {\bf 27}, 623 (1948).
1205: %
1206: \bibitem{KMS00}Y. Kabashima, T. Murayama, and D. Saad, Phys. Rev.
1207: Lett. {\bf84}, 1355 (2000).
1208: %
1209: \bibitem{MKSV00}T. Murayama, Y. Kabashima, D. Saad, and R. Vicente,
1210: Phys. Rev. E {\bf 62}, 1577 (2000).
1211: %
1212: \bibitem{VSK00} R.~Vicente, D.~Saad and Y.~Kabashima,
1213: Europhys.~Lett. {\bf 51} 698 (2000).
1214: %
1215: \bibitem{VSK02} J. van Mourik, D. Saad, and Y. Kabashima
1216: (unpublished).
1217: %
1218: \bibitem{montanari} A.~Montanari, Eur.~Phys.~J.~B, {\bf 23} 121
1219: (2001).
1220: %
1221: \bibitem{Sourlas89}N. Sourlas, Nature (London) {\bf 339}, 693 (1989).
1222: %
1223: \bibitem{Rujan93}P. Ruj\'an, Phys. Rev. Lett. {\bf 70}, 2968 (1993).
1224: %
1225: \bibitem{NW99}H. Nishimori and K. Y. Michael Wong,
1226: Phys. Rev. E {\bf 60}, 132 (1999).
1227: %
1228: \bibitem{VSK99}R. Vicente, D. Saad, and Y. Kabashima,
1229: Phys. Rev. E {\bf 60}, 5352 (1999).
1230: %
1231: \bibitem{MacKay99}D. J. C. MacKay, IEEE Trans. Inform. Theory
1232: {\bf 45}, 399 (1999).
1233: %
1234: \bibitem{Gallager62}R. G. Gallager, IRE Trans. Inform. Theory
1235: {\bf IT-8}, 21 (1962).
1236: %
1237: \bibitem{RSU01}T. J. Richardson, M. A. Shokrollahi,
1238: and R. L. Urbanke, IEEE Trans. Inform. Theory {\bf 47}, 619 (2001).
1239: %
1240: \bibitem{MN95}D. J. C. MacKay and R. M. Neal,
1241: in {\em Cryptography and Coding, 5th IMA Conference},
1242: Lecture Notes in Computer Science 1025,
1243: edited by C. Boyd (Springer, 1995), p.~100.
1244: %
1245: \bibitem{KS99}I. Kanter and D. Saad, Phys. Rev. Lett. {\bf 83}, 2660 (1999).
1246: %
1247: \bibitem{Nishimori93}H. Nishimori, J. Phys. Soc. Jpn. {\bf 62}, 2973 (1993).
1248: %
1249: \bibitem{Iba99}Y. Iba, J. Phys. A {\bf 32} 3875 (1999).
1250: %
1251: \bibitem{nishimori_book} H.~Nishimori, {\em Statistical Physics of Spin
1252: Glasses and Information Processing} (Oxford University Press, Oxford,
1253: UK, 2001).
1254: %
1255: \bibitem{RU01}T. J. Richardson and R. L. Urbanke,
1256: IEEE Trans. Inform. Theory {\bf 47}, 599 (2001).
1257: %
1258: \bibitem{Tanaka-ITpre}T. Tanaka (unpublished).
1259: % submitted to IEEE Trans. Inform. Theory (unpublished).
1260: %
1261: \end{thebibliography}
1262:
1263: \end{document}
1264: