A Secure Framework for Mitigating Man-in-the- Middle Attacks in 5G-IoT Using AMQTT and Smart PMD Key Exchange

Fifth Generation (5G) networks improved global connectivity with extensive Internet of Things (IoT) integration, but they also introduce security challenges, particularly in client-server communication such as password authentication and secure data exchange. This paper focuses on the security issue of Man-in-the-Middle (MitM) attacks and proposes a secure communication framework that integrates an enhanced Advanced Message Queuing Telemetry Transport (AMQTT) protocol with Software-Defined Networking (SDN) using in intelligent encryption and decryption module offering authentication of the message and security. Encryption is enabled by the Proactive MQTT Defence (PMD) algorithm, which exchanges securely Diffie–Hellman-style key agreement over prime fields asymmetric session keys without sending passwords. Following authenticating successfully, data is encrypted converting plaintext to ciphertext and is transmitted securely to the recipient using, which decrypts it to plaintext. The encryption and decryption modules employ multi-layer of protection, includes HMAC (Hash-based Message Authentication Code) for integrity verification, the Varman cipher principle, XOR mixing, S-box (substitution layer), and adaptive permutation to ensure secure data transmission. Experimental results proved that the proposed model effectively mitigates MitM attacks. Key performance metrics include a processing time of 0.1 seconds, a time series delay of 0.2 seconds, and latency of 3.9 milliseconds for a 15 MB payload. Furthermore, traffic overhead is reduced to 0.139 milliseconds for a 10 MB payload, throughput reaches 1631 Mbps for a 5 MB payload, and energy consumption is limited to 0.428 Joules. The integration of new security framework effective defense mechanism against MitM attacks and stable and secure communication for next-generation networks.