AN MINH
← Back to Projects

AI Research

VisPhb: Spiking Neural Networks & Knowledge Distillation

An analysis and implementation of combining Spiking Neural Networks (SNN) with Knowledge Distillation (KD) using PhoBERT for Vietnamese Sentiment Analysis on the UIT-VSFC dataset.

Client
Academic Research
Role
AI Researcher
Stack
Python, PyTorch, SNN (LIF), PhoBERT, Knowledge Distillation
Date
2026
Repository
VisPhb SNN and KD Training Architecture
Fig 1.0 — VisPhb SNN and KD Training Architecture

Technological & Algorithmic Highlights

SNN & KD Integration

Trained Spiking Neural Network students from fine-tuned PhoBERT teachers, replacing top-k FFN layers with Leaky Integrate-and-Fire (LIF) spiking layers.

Multi-tier Experimentation

Explored multiple KD variants including logit-only KD, feature alignment, and full two-stage KD with Wikipedia pre-alignment.

Operation Flow

  1. 01Teacher PhoBERT model is fine-tuned on the UIT-VSFC dataset.
  2. 02Student model initializes with PhoBERT backbone, replacing FFNs with LIF spiking layers.
  3. 03Knowledge Distillation transfers soft targets and hidden states from teacher to student.
  4. 04Equilibrium-state gradient approximation (SpikingBERT-style) and other variants are evaluated.
  5. 05Metrics including theoretical neuromorphic energy and actual GPU energy are logged.

System Infrastructure

Training scripts handle multiple variations of student models, managing surrogate gradients and complex distillation losses.

Infrastructure parameters

  • • Backbone: PhoBERT-base-v2
  • • Dataset: UIT-VSFC, Vietnamese Wikipedia
  • • Spiking Logic: Leaky Integrate-and-Fire (LIF)