1: \begin{abstract}

2: We propose an \ac{AE}-based transceiver for a \ac{WDM} system impaired by hardware imperfections. We design our \ac{AE} following the architecture of conventional communication systems. This enables  to initialize the AE-based transceiver to have similar performance to its conventional counterpart prior to training and  improves the training convergence rate. We first train the \ac{AE} in a single-channel system, and show that it achieves performance improvements by putting energy outside the desired bandwidth, and therefore cannot be used for a \ac{WDM} system.

3: We then train the \ac{AE} in a \ac{WDM} setup. Simulation results show that the proposed \ac{AE} significantly outperforms the conventional approach. More specifically, it increases the spectral efficiency of the considered system by reducing the guard band by $37\%$ and $50\%$ for a root-raised-cosine filter-based matched filter with $10\%$ and $1\%$ roll-off, respectively. An ablation study indicates that the performance gain can be ascribed to the optimization of the symbol mapper, the pulse-shaping filter, and the symbol demapper. Finally, we use reinforcement learning to learn the pulse-shaping filter under the assumption that the channel model is unknown. Simulation results show that the reinforcement-learning-based algorithm achieves similar performance to the standard supervised end-to-end learning approach assuming perfect channel knowledge.

4: \end{abstract}

5: