A Radiative Seesaw Model in a Noninvertible Selection Rule with the Assistance of a Nonholomorphic Modular <i>A</i> 4 Symmetry

Abstract We propose a two-loop neutrino mass model where fermionic and bosonic dark matter (DM) candidates are simultaneously connected to the neutrinos. However, the fermionic DM candidate is favored compared to the bosonic one due to generating the fermionic DM mass at the one-loop level. In order to obtain our desired Lagrangian and Higgs potential, we introduce a $Z_3$ gauging Tambara–Yamagami noninvertible fusion rule with the assistance of a nonholomorphic modular $A_4$ symmetry. The fusion rule forbids the mass of the DM candidate at tree level but its mass is generated at the one-loop level where the DM mass term dynamically violates the fusion rule. After that, the neutrino mass matrix is induced at the one-loop level where a remnant $Z_2$ symmetry still remains. The symmetry assures the stability of our DM candidate. The nonholomorphic modular $A_4$ symmetry plays a role in forbidding the interactions between the SM particles and heavier fermions $X_R$ and an isospin singlet inert scalar boson $S_0$ that run in the DM mass loop, in addition to reducing our free parameters that lead to our predictions for the lepton sector. We perform a $\chi ^2$ numerical analysis for the lepton masses, mixing angles, and phases, and we show several predictions for the normal and inverted hierarchies. Then, we demonstrate our lepton-flavor violations (LFVs), muon anomalous magnetic dipole moment, and the relic density of the DM candidate fixing the best-fit point of the lepton sector.