Rosh Roy, Tanmay Karmakar, Lavkesh Lachhvani, Bhoomi Khodiyar, P. K. Chattopadhyay, Abhijit Sen
A detailed experimental study is conducted in a cylindrical magnetized plasma to investigate the characteristics of Rayleigh–Taylor and Kelvin–Helmholtz instabilities, delineate their dominant parameter regimes, and examine their nonlinear interactions. This is accomplished through modifying the density gradient scale length (Ln) and velocity shear scale length (Ls) by changing Rm that defines the ratio of the magnetic field in the main chamber to that in the source chamber of a linear magnetized plasma device. It is found that at Rm=49, where LnLs, KH instability prevails. In the intermediate regime where Ln∼Ls (Rm=35), a coupled RT–KH instability emerges, characterized by nonlinear three-wave interactions that facilitate energy transfer across modes, as qualitatively confirmed in the bicoherence spectrum of density and potential fluctuations. Furthermore, at higher pressure in the mid Rm regime reveal stronger multiple three-wave interactions, indicating enhanced mode coupling and nonlinear energy redistribution. These findings provide new insights into the dynamics of RT and KH instabilities in magnetized plasmas, leading to an understanding of the development of turbulence via energy cascades and instability-driven mixing. The results are relevant to laboratory, astrophysical and fusion plasmas, where these instabilities govern the evolution of turbulence and structure formation.