MALWARE CLASSIFICATION VIA BYTECODE VISUALIZATION AND MULTIMODAL DEEP LEARNING

Abstract

The rapid proliferation of Android malware poses a critical threat to mobile security, driven by the open-source nature of the Android ecosystem, broad access to its Software Development Kit (SDK), and the availability of multiple app distribution channels. Traditional detection methods, including signature-based, static, and dynamic analysis, often fail against novel or obfuscated variants that use encryption, packing, and polymorphism to evade detection. This dissertation addresses these limitations through three progressive, interconnected contributions. Contribution I introduces a semantic bytecode-to-image encoding method based on Shannon entropy, computed over sliding windows of Dalvik Executable (DEX) bytecode and mapped to the red and blue channels of an RGB image. This encoding directly exposes obfuscation artifacts in encrypted, packed, and compressed code regions that blind color-mapping approaches cannot distinguish. Evaluated on 184,474 samples from AndroZoo for binary malware classification, the entropy encoder outperforms the MalNet and Classbyte baselines across five state-of-the-art CNN architectures, achieving up to 95.77% accuracy and 98.25% ROC-AUC.

Contribution II extends this encoding into a full multiclass framework and dataset, MalVis, by incorporating byte-level N-gram frequency statistics into the green channel. By combining entropy and N-gram signals, MalVis images capture both low-level randomness and high-level structural code patterns. To support the research community, we release the MalVis dataset, the largest publicly available Android malware visualization resource, comprising over 1.3 million labeled RGB images spanning nine malware families and benign samples. To validate that CNN models rely on the proposed encodings rather than spurious image artifacts, we apply Grad-CAM and Grad-CAM++ attention mapping, confirming that model attention consistently aligns with the entropy- and N-gram-encoded channels across all malware classes.

Contribution III introduces ViCoMal, a late-fusion multimodal framework that addresses the single point of failure (SPOF) inherent in unimodal approaches. ViCoMal pairs MalVis image-based malware representations with 65 DEX-based contextual features, including 19 novel engineered features consisting of normalized risk scores, binary capability indicators, and rule-based behavioral profiles extracted using static analysis. Five CNN architectures and seven classical machine learning models are trained independently per modality, and their class-probability outputs are combined through eight fusion strategies. To handle a severe 169:1 class imbalance, SMOTE oversampling and class-weighted training are applied jointly. Evaluated under five-fold stratified cross-validation on the full 1.3M ViCoMal dataset, the best ensemble, which averages the predictions of the top three models (Random Forest, ResNet50, and XGBoost), achieves 91.84% accuracy, outperforming the strongest image-only baseline by 3.43 percentage points and the strongest contextual-only baseline by 2.05 percentage points.

Together, these contributions establish a scalable, interpretable, and high-performing pipeline for Android malware detection, advancing the state of the art in both malware visualization and multimodal security analysis.