Broadband tunable microwave photonic radar for simultaneous detection of human respiration, heartbeat, and speech with deep learning-based speech recognition

Multimodal vital sign monitoring and speech detection hold significant importance in medical health, public safety, human-computer interaction, and other fields. This study proposes a broadband tunable microwave photonic radar system that can simultaneously monitor respiration, heartbeat, and speech. The system works by generating broadband radar signals to detect subtle skin displacements caused by these physiological activities. It then utilizes phase variations in radar echo signals to extract and reconstruct the corresponding physiological signals. To enhance the processing capability for speech signals, a convolutional neural network with a dual-channel feature fusion model is incorporated, enabling high-precision speech recognition. In addition, the system’s frequency-tunable characteristic allows it to flexibly switch frequency bands to adapt to different working environments, greatly improving its practicality and environmental adaptability. In the proof-of-concept experiments, speech signals are reconstructed and recognized in the Ku, K, and Ka bands, achieving recognition accuracies of 97.20%, 98.07%, and 97.43%, respectively. The capability to detect multimodal vital signs is also thoroughly validated using a respiratory and heartbeat simulator. During a 20-s monitoring period, while accurately reconstructing speech, the maximum average error counts for respiration and heartbeat monitoring are 0.39 and 0.87, respectively, proving its reliability and effectiveness in multimodal vital sign monitoring.