Semantic Context-Aware Automated Front-End Code Generation for Mobile applications using a Vision Language Code Transformer

Automated front-end Code Generation (CG) is required due to the rising demand for fast and error-free development of mobile applications. Existing research has overlooked semantic context-aware automated front-end code generation for mobile applications, resulting in less reliable generated interfaces. Thus, this paper proposes a novel Visual Top-k Attention Bidirectional Encoder Representations from Transformers (VisualTABERT) and Code Longformer Attention Text-To-Text Transfer Transformer (CodeLAT5+) enabled semantic context-aware automated front-end CG system. First, the mobile application screenshots are collected and preprocessed. The UI components are then segmented using a Spatial Pyramid Pooling–enhanced YOLOv8 (SPP-YOLOv8) model, followed by text extraction and attribute identification for each detected element. Similar UI elements are grouped based on the extracted text attributes and segmented UI elements employing Density Survival Function Based Spatial Clustering of Applications with Noise (DSFBSCAN). Then using a VisualTABERT based module, the semantic relationships between UI elements and their associated textual content are established. Consequently, by using Affine Polynomial Kernel Transformation (APKT), the coordinate alignment is done for the UI elements. According to the grouped similar UI elements, semantic relationships, and coordinate alignment, the structured intermediate representation is performed via Schema Mapping (SM). Finally, CodeLAT5+ generates the target source code by utilizing the structured intermediate representation obtained from the preceding transformation process. As per the outcome, the proposed model achieved a higher Mean Reciprocal Rank (MRR) (0.9256) than the conventional methods.