1: \begin{abstract}

2: 

3: Deep Neural Networks (DNNs) are powerful algorithms that have been proven capable of extracting non-Gaussian information from weak lensing (WL) data sets. Understanding which features in the data determine the output of these nested, non-linear algorithms is an important but challenging task. We analyze a DNN that has been found in previous work to accurately recover cosmological parameters in simulated maps of the WL convergence ($\kappa$). We derive constraints on the cosmological parameter pair $(\Omega_m,\sigma_8)$ from a combination of three commonly used WL statistics (power spectrum, lensing peaks, and Minkowski functionals), using ray-traced %noiseless 

4: simulated $\kappa$ maps.  We show that the network can improve the inferred parameter constraints relative to this combination by $20\%$ 

5: even in the presence of realistic levels of shape noise. 

6: We apply a series of well established saliency methods to interpret the DNN and find that the most relevant pixels are those with extreme $\kappa$ values. For noiseless maps, regions with negative $\kappa$ account for $86-69\%$ of the attribution of the DNN output, defined as the square of the saliency in input space.

7: In the presence of shape nose, the attribution concentrates in high convergence regions, with $36-68\%$ of the attribution in regions with $\kappa > 3 \sigma_{\kappa}$.

8: \end{abstract}

9: