1: \begin{abstract}

2: We study the quantum phases and phase transitions of the Kane-Mele Hubbard 

3: (KMH) model on a zigzag ribbon of honeycomb lattice at a finite size via the 

4: weak-coupling renormalization group (RG) 

5: approach. In the non-interacting limit, the KM model is known to 

6: support topological edge states where electrons show helical property 

7: with orientations of the spin and momentum being locked. 

8: The effective inter-edge hopping terms are generated due to finite-size effect. 

9: In the presence of an on-site Coulomb repulsive interaction and the 

10: inter-edge hoppings,  

11: special focus is put on the stability of the topological edge states 

12: (TI phase) in 

13: the KMH model against (i) the charge and spin gaped (II) phase, (ii) 

14: the charge gaped but spin gapless (IC) phase and (iii) 

15: the spin gaped but charge gapless (CI) phase depending on 

16: the number (even/odd) of the zigzag ribbons, 

17: doping level (electron filling factor) 

18: and the ratio of the Coulomb interaction to the inter-edge tunneling. 

19: We discuss different 

20: phase diagrams for even and odd numbers of zigzag ribbons. 

21: We find the TI-CI, II-IC, and II-CI  

22: quantum phase transitions are of the Kosterlitz-Thouless (KT) 

23: type.  

24: By computing various correlation functions, we further analyze 

25: the nature and leading instabilities of these phases.

26: \end{abstract}

27: