1: \begin{abstract}

2: Joint optimization of nonlinear precoders and receive filters is studied

3: for both the uplink and downlink in a cellular system.

4: For the uplink, the base transceiver station (BTS) receiver implements

5: successive interference cancellation, and for the downlink,

6: the BTS station pre-compensates for the interference with

7: Tomlinson-Harashima precoding (THP). Convergence of alternating

8: optimization of receivers and transmitters in a single cell

9: is established when filters are updated according

10: to a minimum mean squared error (MMSE)

11: criterion, subject to appropriate power constraints.

12: {\em Adaptive} algorithms are then introduced for updating

13: the precoders and receivers in the absence of channel state

14: information, assuming time-division duplex transmissions

15: with channel reciprocity. Instead of estimating the channels,

16: the filters are directly estimated according to a least

17: squares criterion via bi-directional training:

18: Uplink pilots are used to update the feedforward and feedback filters,

19: which are then used as interference pre-compensation

20: filters for downlink training of the mobile receivers.

21: Numerical results show that nonlinear filters can provide

22: substantial gains relative to linear filters with limited forward-backward iterations.

23: \end{abstract}

24: