1: \begin{abstract}

2: 

3: Large Language Models (LLMs) have shown promising performance as knowledge repositories and form the backbone of modern AI systems. The fact retrieval capability of LLMs has been shown to be positively correlated with the frequency of entities in their pretraining corpus. In this study, we investigate the internal reasoning and fact retrieval mechanisms of LLMs through a comprehensive analysis centered on three critical dimensions, i.e., the influence of fact popularity, sensitivity of the models to lexical variations in query formulation, and the progression of hidden state representations across LLM layers. We observe that popular questions lead to early convergence of the internal states towards the correct answer, and the output responses tend to be highly inconsistent across different rephrased queries at a higher popular scale. The retrieved facts tend to become increasingly dissimilar across the rephrased variations of highly popular queries. Moreover, LLMs address relevant parts of highly popular queries poorly as compared to their less popular counterparts. We also discover that LLMs fail to separate facts retrieved from their parametric memory related to highly popular distinct subjects. Through a case study, we unveil these hidden strains in the internal mechanisms of LLMs when dealing with highly popular queries, a phenomenon we define as \textit{information anxiety}. These findings highlight the necessity for further research to reconsider assumptions about the straightforward handling of such frequently encountered facts.

4: \end{abstract}

5: