High-Frequency Patch-Based Generative Adversarial Network (HF-PGAN): A Face Inpainting Network for Large Missing Regions in Virtual Reality Scenario

Face inpainting in computer vision restores missing or damaged facial regions, important for virtual reality (VR) systems to enhance visual experiences in social and virtual interactions. Deep generative approaches, such as Generative Adversarial Networks (GANs), have advanced the face inpainting problem but face challenges in generating high-frequency features for large missing regions. Repairing missing pixels with high-frequency details becomes more difficult as the extent of missing regions increases due to VR headsets, leading to artifacts and deviations from the original image. Furthermore, instability in the discriminator’s density ratio calculation during training results in poor generator performance in high-dimensional spaces. This study introduces HF-PGAN, a High-Frequency Patch-Based Generative Adversarial Network for face inpainting to overcome these limitations. HF-PGAN utilizes a modified U-Net generator for large missing regions (50\% of the original image) and a modified patch-based CNN discriminator with additional residual blocks as skip connections to address the vanishing gradient problem, along with a zero-centered gradient penalty at the patch level for improved generalization and stability during training. HF-PGAN outperformed state-of-the-art approaches, obtaining PSNR values of 30.568 and 29.497 on the CelebA-HQ and FFHQ datasets, respectively, and successfully restores high-frequency details, such as skin, hair, and intricate facial texture. In conclusion, the results demonstrate the effectiveness of HF-PGAN, making it a promising solution for VR, photo editing, and facial image restoration, notably improving high-frequency face inpainting, particularly for VR-induced large missing regions.