1: \documentclass[twocolumn,preprintnumbers,amsmath,amssymb]{revtex4}
2: %\documentclass[preprint,showpacs,preprintnumbers,amsmath,amssymb]{revtex4}
3:
4: \usepackage{epsfig}
5: \usepackage{dcolumn}
6: \usepackage{bm}
7: \usepackage{psfrag}
8:
9: \newlength{\mywa}
10: \setlength{\mywa}{.35\textwidth}
11: \newlength{\myw}
12: \setlength{\myw}{5.cm}
13:
14: \newcommand{\img}{\mathbf{i}}
15:
16: \begin{document}
17: \preprint{CNL-2003}
18:
19: %-----------------------------------------------------------------------------
20: \title{On the Propagation of Congestion Waves in the Internet}
21: \author{J\'ozsef St\'eger}%
22: \email{steger@complex.elte.hu}
23: \author{P\'eter Vaderna}
24: \altaffiliation[Also at ]{Traffic Analysis and Network Performance Laboratory\\
25: Ericsson Hungary Ltd.
26: }
27: \email{Peter.Vaderna@eth.ericsson.se}
28: \author{G\'abor Vattay}\altaffiliation[Also at ]{Collegium Budapest, Institute for Advanced Study}
29: \email{vattay@complex.elte.hu}
30: \affiliation{
31: Department of Physics of Complex Systems\\
32: Eotvos University, Budapest \\
33: }%
34: \date{\today}
35: %-----------------------------------------------------------------------------
36:
37: \begin{abstract}
38: Traffic modeling of communication networks such as Internet has become a very important field of research. A number of interesting phenomena are found
39: in measurements and traffic simulations. One of them is the propagation
40: of congestion waves opposite to the main packet flow direction.
41: The purpose of this paper is to model and analyze packet congestion on
42: a given route and to give a possible explanation to this phenomenon.
43: \end{abstract}
44: \maketitle
45:
46: %-----------------------------------------------------------------------------
47: % Introduction
48: %-----------------------------------------------------------------------------
49:
50: In the recent past many aspects of computer data traffic on the Internet have been investigated. Signatures of long-range correlations\cite{4}, scaling\cite{1}, chaos\cite{2} and phase transitions\cite{3} have been found. In Ref.\cite{csab} $1/f^\alpha$ noise has been observed in the time series of round trip times similar to those observed in highway traffic measurements\cite{higuchi}. Since the slowing down and acceleration of packet flow in computer networks are very similar to those of cellular automaton models of cars in highway traffic\cite{5}, it has been argued that a valid analogy exists between these subjects which later has been demonstrated quantitatively\cite{6,Huisinga}. One of the most peculiar features of car traffic and flow of granular media is the propagation of density waves\cite{congestion}. In car traffic stop-and-go type congestion waves propagate against the direction of the flow. In Ref.\cite{taka} the authors made an attempt to observe the propagation of congestion in the routers of a real computer network. They studied the spatio-temporal correlations of the level of congestion in routers and showed that congestion can propagate from a heavily loaded router to one of its empty neighboring routers. Today computer networks can be studied with the help of network simulators, which has been developed by engineers. These tools enable us to assemble any computer network configuration, to use the most commonly used packet sending mechanism TCP/IP protocol and to emulate its real behavior without building the system from hardware components. These simulators can imitate the behavior of hardware elements (computers, routers, lines etc.) accurately so that the results of the simulations are nearly identical with those obtained from measurements. One of the most popular tools is the Berkeley Network Simulator
51: (\texttt{ns-2.26}\cite{ns}) which is used in this paper.
52:
53:
54:
55:
56:
57:
58: In this letter we study the network traffic generated in a unidirectional ring of identical routers connected as shown in the left part of Fig.~\ref{topology}.
59: \begin{figure}[htb]
60: \begin{center}
61: \psfrag{0}[c][c][.45]{0}
62: \psfrag{1}[c][c][.45]{1}
63: \psfrag{i}[c][c][.45]{$i$}
64: \psfrag{i-1}[c][c][.45]{$i-1$}
65: \psfrag{N-1}[c][c][.45]{$N-1$}
66: \psfrag{tau}[c][c][.8]{$\tau$}
67: \psfrag{C}[c][c][.8]{$C$}
68: \psfrag{B}[c][c][.8]{$B$}
69: \psfrag{Xi}[c][c][.8]{$X_i$}
70: \psfrag{keplet}[c][c][.8]{$C_{i-1}=\sum\limits^{N-1}_{j=0, j\neq i}X_j $}
71: \psfrag{label}[c][c][.8]{$X_i=\frac{w_i P}{T_{RTT}}$}
72: \includegraphics[width=.24\textwidth]{figures/topology_ns.eps}
73: \includegraphics[width=.22\textwidth]{figures/topology.eps}
74: \caption{The ring structure. {\bf Left,} Network simulator: The TCP agent at site $i$ continuously transfers data trough the ring of network routers connected with constant capacity $C$ lines with time delay $\tau$. The packets not processed yet by the routers are queued in buffers of size $B$. The stream of data can traverse in clockwise direction only to arrive at the destination node at site $i-1$. In the simulations a realistic set of parameters has been set ($C=10^7~$bits/s, $\tau=.031~$s, $B=300~$packets, $P=4416~$bits, $N=10$). {\bf Right,}Continuous model: Each TCP is modeled with its sending rate $X_i$ and its traffic flow traverses the ring topology. On the link connecting sites $i-1$ and $i$ all the sending rates of traversing TCP flows are summed up to yield the total link utilization.% $C_{i-1}=\sum\limits^{N-1}_{j=0}X_j - X_i$.
75: \label{topology}}
76: \end{center}
77: \end{figure}
78: The ring geometry mimics periodic boundary conditions. This way we can study the propagation of congestion in an isolated, clean setup, which makes it possible to compare the results to those obtained in cellular automaton, car and granular traffic simulations. We show that this system drives itself into a critical congested state in a self-organized way. Both the position of the congested router and the packet sending activity at the sites propagate against the direction of the packet flow. The profile of the congestion wave can be reconstructed from the activities of the computers connected to the ring. The velocity of the congestion wave can be measured. Then a simple model of the system is introduced which is able to explain the main features of the congestion wave such as shape and speed. A formula for the velocity of the congestion wave is obtained and tested.
79:
80:
81:
82: In our model system the ring is formed by $N$ identical routers which can forward packets in clockwise direction. Routers are connected with a line of capacity $C$ (measured in data bits per second) with a constant forwarding delay $\tau$ (measured in seconds). The incoming data flow of a router can temporarily exceed the capacity of the outgoing line. To avoid data loss in this situation the router contains a buffer of size $B$ (measured in data packets) where packets can be stored. The computers are instructed to send data persistently to their anti-clockwise neighbors, so that the packets traverse the longest possible route in the ring. The traffic is "granular" as computers send in data packets of size $P$ (measured in bits). The data traffic of computers is controlled by the TCP/IP protocol\cite{jacobson}. This protocol ensures that the data packet-sending rate is decreased whenever congestion occurs and that it is increased when there is an available unused capacity in the system. The control algorithm became quite sophisticated over the years. Here we can only sketch the main properties of the most popular version used currently.
83:
84: After establishing connection between two computers over the network TCP algorithm regulates the packet-sending rate. First a single packet is sent out. Upon receiving that packet the receiver acknowledges the arrival of the packet by sending back a small size acknowledgement packet (ACK). The time elapsed between the sending out of a packet and receiving the corresponding ACK is called round trip time (RTT). The TCP maintains an internal variable, the Congestion Window ($w$), which is used to control the number of packets sent out when the ACK is received. It starts with the initial value $w=1$ and then it is increased according to $w\mapsto w+1/w$ each time an ACK is received. Two new packets are sent out if the congestion window crosses an integer value and only a single packet otherwise. This way the integer part of the window $[w]$ gives the number of sent but not yet acknowledged packets in the network. Assuming constant RTT during this process, the congestion window and the number of packets out in the network are increased linearly in time. This process lasts until a packet is lost somewhere in the network, indicating congestion. As a response the packet-sending rate should be decreased. So, the TCP reduces the value of the congestion window $w \mapsto \beta w$ $(\beta < 1)$ and does not send out any new packets in response to ACKs until the number of still unacknowledged packets decreases to the integer part of the new (reduced) value of the congestion window. After that the packet-sending algorithm returns to the original linear increase phase described above.
85:
86: In case the congestion window variable is large we can neglect its granularity and can treat it as a continuous variable. In the linear increase phase the window variable is increased by one in each RTT period and its dynamics can be well approximated with the differential equation
87: \begin{equation}
88: \frac{dw}{dt}=\frac{1}{T_{RTT}(t)},
89: \end{equation}
90: where $T_{RTT}(t)$ is the actual value of the RTT measured by the TCP. In case of packet loss the window is reduced
91: \begin{equation}
92: w(t_{+})=\beta w(t_{-}),
93: \end{equation}
94: where $t_{\pm}=t\pm0$ denotes the time right after and before the packet loss. The sending rate $X$ (bit/sec) in the continuous approximation can be estimated as the amount of data sent within an RTT, $X=Pw/T_{RTT}$. Neglecting the change of RTT on the scale of RTTs the sending rate satisfies the following pair of equations:
95: \begin{eqnarray}
96: \frac{dX}{dt}&=&\frac{P}{T_{RTT}^2(t)},\label{xx}\\
97: X(t_{+})&=&\beta X(t_{-}) \;\;\; \mbox{at packet loss},
98: \end{eqnarray}
99: where typically $\beta=1/2$.
100:
101: Next, we present the results of our simulation study carried out with the network simulator. The geometry and parameters of the setup is on the left part of Fig.~\ref{topology}, which is essentially the same as we could have observed in real hardware realization. Fig.~\ref{spatio} shows the spatiotemporal diagram of the congestion wave occurring in the network simulator. One can see that after a short initial transient (up to $500~$s) the pattern remains stable and propagates in anti-clockwise direction. In this respect it resembles the congestion propagation in car traffic. The speed of the congestion wave pattern is almost constant. Its average can be determined by measuring the average speed of the center of mass of the pattern. This should be carefully defined in the present situation as the system is spatially periodic. Taken into account the periodic boundary conditions the center of mass of the pattern can be defined by $\langle{} i \rangle(t)=\arg\left(\Sigma_{j=0}^{N-1} X_j(t) e^{\img (2\pi/N) j}\right)$. The speed of the pattern is the time derivative of this quantity.
102:
103: \begin{figure}[htb]
104: \begin{center}
105: \includegraphics[width=\mywa]{figures/ns_300_1000.eps}
106: \caption{Spatiotemporal diagram of congestion propagation. The horizontal axis is the real time (in seconds) of the system and the site index ($i$) is on the vertical axis. In this simulation the number of sites was $N=10$. Notice, that the sites $i=0$ and $i=N-1$ are neighbors in the ring topology. The sending rate $X_i$ is color-coded according to the scale indicated at the right side of the figure. Deep blue indicates very low sending rate due to high congestion. It can be seen that the most congested site propagates in anti-clockwise direction with almost constant speed, while the packet traffic itself is clockwise directed.
107: \label{spatio}}
108: \end{center}
109: \end{figure}
110:
111: Once the speed of the pattern is determined we can analyze the shape of the profile. Representing the sending rates $X_{i'+\lfloor\langle{}i \rangle\rfloor}(t)$ in co-moving coordinates $i'$ relative to the center of mass we recover the shape of the traveling wave pattern. Averaging the new series in time the profile of the front emerges as in Fig.~\ref{front}.
112: \begin{figure}[htb]
113: \begin{center}
114: \psfrag{Site}[c][c][.8]{Site}
115: \psfrag{Average bandwidth occupied [bps]}[c][c][.8]{Time averaged sending rate [bits/s] }
116: \psfrag{Data points of the network simulator}[r][r][.7]{}
117: \includegraphics[width=\myw,angle=270.]{figures/ns_profil.eps}
118: \caption{The shape of the traveling wave profile. It has been determined by averaging the time series in co-moving spatial coordinates.
119: \label{front}}
120: \end{center}
121: \end{figure}
122:
123: While the continuous equations constitute gross simplification of the original TCP dynamics, the main properties of the traveling wave can be recovered from them with some additional assumption made on the packet loss process as we show next.
124:
125: The bandwidth $C_{i-1}(t)$ utilized on the link connecting nodes $i-1$ and $i$ is the sum of sending rates of TCPs whose traffic flows through that link. In our case the flows of all TCPs traverse that link except the one starting at node $i$ and ending at node $i-1$:
126: \begin{equation}
127: C_{i-1}(t)=\sum_{j=0, j\neq i}^{N-1} X_j(t) =\sum_{j=0}^{N-1} X_j(t)-X_i(t),
128: \label{modell}
129: \end{equation}
130: where site $i=N$ is identified with site $i=0$ due to periodicity. The traffic of ACK packets emanating in $i-1$ and absorbed in $i$ is low due to the small size of ACK packets and its contribution to traffic can be neglected (see also the right part of Fig.~\ref{topology}). Congestion and packet loss occur in the system whenever one of the bandwidths $C_i(t)$ reaches the link capacity $C$. According to (\ref{modell}) the largest link utilization $C_i(t)$ is at site $i=i^*-1$ where $i^*$ is the site where the sending rate $X_{i^*}(t)$ is the lowest. We then have to investigate which TCP flow will lose packet on link $i^*-1$. In principle all the TCP flows traversing the congested link can lose packets, so only the TCP at site $i^*$ is immune. Our observation is that the TCP flow starting at the actual congested link (with sending rate $X_{i^*-1}$) experiences the packet loss almost surely. This is due to the fact that the processing of the small acknowledgement packets is faster than that of data packets and they tend to queue up and to arrive in batches. As a response, TCPs send out data packets also in batches. Then obviously the TCP that ejects this burst of data packets directly into an almost saturated buffer will lose packets in the process.
131:
132: The mechanism described above is responsible for the emergence of the congestion wave in the system. The TCP at site $i^*-1$ suffers packet losses repeatedly until its sending rate $X_{i^*-1}$ becomes smaller than $X_{i^*}$. From then on $X_{i^*-1}$ will be the lowest in the system, link utilization $C_{i^*-2}$ will be the highest and TCP at site $i^*-2$ suffers the packet losses. This way congestion propagates site by site anticlockwise in the system. After several rounds of congestion propagation the propagating front of Fig.\ref{front} emerges.
133:
134: The shape of the front is linear with a sharp drop connecting its ends. We can determine the parameters of the linear front in our model. Let the nodes forming the linear part of the front range from $0$ to $N-1$ and let the minimum sending rate first be at the $0$th node
135: \begin{equation}
136: X_i(0)=a+bi,
137: \label{linear}
138: \end{equation}
139: as it is shown in Fig.~\ref{explain} that illustrates the evolution of a system containing $N=5$ nodes.
140: \begin{figure}[htb]
141: \begin{center}
142: \psfrag{1}[c][c][.9]{0}
143: \psfrag{2}[c][c][.9]{1}
144: \psfrag{N-1}[c][c][.9]{$N-1$}
145: \psfrag{a}[r][l][.9]{$a$}
146: \psfrag{a+b}[r][l][.9]{$a+b$}
147: \psfrag{X}[c][c][.9]{$X$}
148: \psfrag{i}[c][c][.9]{$i$}
149: \includegraphics[width=\myw]{figures/cascade.eps}
150: \caption{Sending rate evolution in time. $N=5$, $l=2$, $X_4$ decreases until
151: it reaches the value of $X_0$.
152: \label{explain}}
153: \end{center}
154: \end{figure}
155: The packets start to be dropped at site $N-1$. We start our description at the first packet drop, which occurs when $C_{N-1}(0)=\sum_{i=0}^{N-1}X_i(0)-X_0(0)=C$ holds. This initial condition gives the first condition for the initial shape of the front
156: \begin{equation}
157: C=(N-1)\left(a+\frac{N}{2}b\right).
158: \label{limit}
159: \end{equation}
160: Immediately after the packet drop the sending rate at site $N-1$ decreases to $X_{N-1}(+0)=\beta X_{N-1}(0)$, while the rest of the sending rates stay unchanged. Then the sending rates increase in a uniform manner with an amount ${X'}_i=X_i+(1-\beta)X_{N-1}/(N-1)$ until the next packet loss occurs. In particular, from before the first packet loss until the second packet loss the sending rate at site $N-1$ changes to ${X'}_{N-1}=qX_{N-1}(0)$, where $q=\beta+\frac{1-\beta}{N-1}$. This process is then repeated $l$ times until the sending rate $q^l X_{N-1}$ becomes lower than the actual value of $X_0$. This way the wave moves one site to the left, while its linear shape is preserved as the sending rates of all nodes, except $N-1$, increase with $b$. From Eq.~\ref{xx} one can calculate the time needed for this $T_p=bT_{RTT}^2/P$. Accordingly the speed of
161: the congestion wave is $v:=1/T_p$ (measured in site/sec.).
162:
163: The formula derived for the speed of the congestion waves can be tested against real data produced by the network simulator. In the simulation one can measure the average round trip time observed by the TCPs, the mean slope of the linear part of the front $b$ and the speed of the front. On the main part of Fig.~\ref{tests} one can see that the measured values satisfy the relation $vbT_{RTT}^2/P= 1$ very well.
164: \begin{figure}[ht]
165: \begin{center}
166: \includegraphics[width=\myw, angle=270]{figures/tests.eps}
167: \caption{
168: Testing the relation $vbT_{RTT}^2/P= 1$ for various system sizes $N=5-20$ in the network simulator. After the end of transients the values of $b$, $T_{RTT}$ and $v$ have been obtained
169: by averaging over samples of $5000$ sec. {\bf Inset:}
170: Testing Eq.~\ref{resultslope} in the network simulator. Average values of $b$ are plotted against
171: the results of (\ref{resultslope}) with $q^l$ obtained from the simulation.
172: \label{tests}}
173: \end{center}
174: \end{figure}
175:
176: With the help of (\ref{limit}) and (\ref{linear}) the slope of the front in the model can also be directly expressed as
177: \begin{equation}
178: b=\frac{2C(1-q^l)}{(N-1)(N-2)(q^l+1)},\label{resultslope}
179: \end{equation}
180: where $l$ should be determined independently. Our mathematical model allows several positive integer values of $l$ with an upper bound due to the positiveness of $b$. We found that only the largest possible $l$ value is stable against small perturbations of the wave front. Systems started at lower $l$ values always shift towards a greater value of $l$. In the network simulator we always observed the realization of the most stable (the highest possible $l$) solutions of the model. In the inset of Fig.~\ref{tests} we compare the measured values of $b$ with (\ref{resultslope}). We again find good agreement.
181:
182: As a summary we showed that congestion waves are formed naturally in the data traffic of today's computer networks. The mechanism behind the wave formation is that packet losses occur most likely in computers nearest to the site of the actual congestion and other computers sharing the congested link increase their sending rates, moving the site of the congestion one site downstream. This basic mechanism is quite general and can create congestion moving against the direction of the data traffic in more complicated geometries. A formula for the speed of the congestion wave has been derived in a simple ring topology and network simulations have confirmed it. Such formulas can be developed for more complicated geometries, which is our next research goal.
183:
184: The authors thank the National Science Fund, Hungary (OTKA T37903) for the support.
185:
186:
187:
188:
189:
190: \begin{thebibliography}{99}
191: \bibitem{1} S. Valverde, R. V. Sole,
192: Physica {\bf A 312}, 636 - 648 (2002), J. Yuan, Y. Ren and X. Shan,
193: Phys. Rev. {\bf E 61}, 1067 (2000)
194:
195: \bibitem{2} A. Veres, M. Boda, Proceedings of
196: IEEE INFOCOM Tel Aviv, March 2000, p. 1715--1723 (2000)
197: http://www.ieee-infocom.org/2000/papers/74.pdf
198:
199: \bibitem{3} M.Takayasu, H.Takayasu, and K.Fukuda,
200: Physica {\bf A 277}, 248-260 (2000)
201:
202:
203: \bibitem{4} W. E. Leland, M. S. Taqq, W. Willinger and D. V. Wilson,
204: Proceedings of ACM SIGCOMM, San Francisco, California,
205: Ed. Deepinder P. Sidhu, p. 183--193 (1993); K. Fukuda, H. Takayasu and M. Takayasu,
206: Physica {\bf A 287}, 289-301 (2000);
207:
208:
209: \bibitem{csab} I. Csabai, J. Phys. {\bf A27}, L417 (1994);
210:
211: \bibitem{higuchi} T. Musha and H. Higuchi, Japan Appl. Phys. {\bf 15} 1271 (1976)
212:
213: \bibitem{jacobson} Jacobson, V., {\em Congestion Avoidance and Control},
214: Proceedings of SIGCOMM '88, August 1988, pp. 314-329.
215:
216: \bibitem{5} D. Chowdhury, L. Santen and A. Schadschneider,
217: Statistical physics of vehicular traffic and some related systems.
218: Physics Reports, 329(4-6):199-329 (2000);
219: B. S. Kerner and P. Konhäuser.
220: Structure and parameters of clusters in traffic flow.
221: Physical Review E, 50(1):54 (1994);
222: K. Nagel and M. Paczuski.
223: Emergent traffic jams.
224: Physical Review E, 51:2909 (1995)
225: O. Biham, A. Middleton, and D. Levine.
226: Self-organization and a dynamical transition in traffic-flow models.
227: Physical Review A, 46:R6124 (1992)
228:
229: \bibitem{6} G. Simon, I. Csabai
230: The analogies of highway and computer network traffic
231: Physica {\bf A 307} 516 - 526 (2002)
232:
233: \bibitem{Huisinga} T. Huisinga, R. Barlovic, W. Knospe, A. Schadschneider, M. Schreckenberg,
234: Physica {\bf A 294}, 249 (2001)
235:
236: \bibitem{congestion} K. Nagel and M. Schreckenberg, J. Physique {\bf I 2} 2221 (1992);
237: K. Nagel and H. J. Hemann, Physica {\bf A199} 254 (1993);
238: M. Leibig, Phys. Rev. {\bf E 49} 184 (1994)
239:
240: \bibitem{7} M. Bando, K. Hasebe, A. Nakayama, A. Shibata, and Y. Sugiyama.
241: Dynamical model of traffic congestion and numerical simulation.
242: Physical Review E, 51(2):1035 (1995);
243: A. Schadschneider.
244: Statistical physics of traffic flow.
245: Physica {\bf A 285}, 101 (2000);
246:
247: \bibitem{taka}K.Fukuda, H.Takayasu, and M.Takayasu, Fractals, {\bf 7}(1) 23-31 (1999)
248:
249: \bibitem{ns} S.~McCanne and S.~Floyd.
250: Ns (network simulator).
251: Available via http://www-nrg.ee.lbl.gov/ns. (1995)
252:
253: \end{thebibliography}
254:
255: \end{document}
256:
257: