Feature-Engineered Trojan Malware Detection on Windows-Based IoT Gateways Using a Custom Deep Neural Network and Automated Monitoring Pipeline

The growth of Internet of Things (IoT) environments has expanded the attack surface of modern systems. Trojan attacks are a major challenge as they evade conventional detection mechanisms and operate silently within legitimate processes. This paper presents an automated Trojan detection framework for Windows-based IoT gateways. The framework combines custom dataset generation informative feature engineering and deep learning-driven analysis. A dataset of 3000 real world executable samples was created through controlled sandbox execution and forensic monitoring. The process captured behavioral static and network-level characteristics. An initial set contained 146 extracted features. A multi-stage feature selection process identified 33 informative attributes. This step allowed efficient learning and preserved discriminative power. A custom deep neural network model named TrDNN was developed using these features. The model captures complex nonlinear patterns linked to Trojan activity. The framework was evaluated against five classical machine learning models. It was also compared with five deep learning baselines. Results show that TrDNN achieves strong detection performance. The accuracy is 0.975. The precision is 0.972. The recall is 0.969. The F1 score is 0.970. The study also examines inference time and energy consumption. The model shows a balance between detection effectiveness, computational cost and energy efficiency. This makes it suitable for resource-constrained IoT gateway deployment. The detection model was integrated into an automated real-time monitoring pipeline. The system enables continuous process surveillance through Windows command line automation with minimal operational overhead. Statistical validation used paired t tests, Wilcoxon signed rank tests and McNemar chi-square test. The performance gains are statistically significant and do not indicate overfitting. The framework provides a reliable, efficient and deployable solution for Trojan detection in modern IoT systems.