PhD Researcher in RL @ Coventry University · Research Software Developer @ UCL · AAAI 2026 workshop paper (oral) · 2nd Runner-Up, Prime Minister's National Innovation Award · ex-AI Engineer @ CureMD
Website · Google Scholar · LinkedIn · CV · Email
DOMAINS
Click a badge above to jump to a domain, or expand any section below.
TECH STACK
Python · C++ · MATLAB · PyTorch · TensorFlow · Stable Baselines3 · RLlib · Gymnasium · MuJoCo · BOPTEST · SINDy · scikit-learn · OpenCV · Hugging Face · Docker · HPC / SLURM · W&B · LaTeX · Linux
I'm a PhD researcher at Coventry University and a Research Software Developer at UCL. I develop learning and planning methods for energy-efficient building control, with a particular focus on Monte Carlo Tree Search, PPO and SAC. I'm also interested in the numerical details that can quietly make reinforcement learning unreliable, and in building experiment software that makes results easy to reproduce and audit. At UCL, that means reproducible multi-agent LLM experiments for management research.
Planning and reinforcement learning for building energy systems
Controllers that meet deadlines and comfort constraints without wasting energy.
| Project | Description | Stack | Link |
|---|---|---|---|
| Deadline-Aware Hot-Water Control | 📄 Paper, AAAI 2026 Workshop on AI for Energy & Sustainability (oral). Heat a domestic immersion tank to a target temperature by a deadline while using as little energy as possible. A custom Gymnasium environment models the tank with first-order thermal physics, and three controllers are compared under identical conditions: a bang-bang baseline, zero-shot Monte Carlo Tree Search planning, and a domain-randomised PPO agent. PPO is the most energy-efficient, saving 26–69% energy vs bang-bang depending on the prediction horizon; in a representative scenario it uses 54% less than bang-bang and 33% less than MCTS. | Python, Gymnasium, Stable Baselines3, MCTS | Repo · Paper |
| RL for HVAC Control | A Soft Actor-Critic agent controls a hydronic heat pump in BOPTEST's high-fidelity bestest_hydronic_heat_pump apartment under highly dynamic electricity pricing — model-free, with no building model or per-building tuning. SAC keeps the zone in comfort 84% of the time, the best of any controller tested (weather-compensated PI 81%, PID 71%, bang-bang 56%), using about 2% more energy than WC-PI. Compares hat-shaped, Huber-band and potential-based reward shaping (PBRS reaches 86.3% comfort), then stress-tests the best policy on extreme cold and 2× pricing, where it learns to trade comfort for cost. |
Python, Stable Baselines3, BOPTEST, Docker | Repo |
Skills: Model-free RL (PPO, SAC) · Monte Carlo Tree Search · Reward shaping · Building energy simulation · Deadline-constrained control · Gymnasium environment design
The numerical details that quietly break policy optimisation
Tracing a training failure down to a single rounded number, then proving it.
| Project | Description | Stack | Link |
|---|---|---|---|
| PPO + gSDE Tanh Replay Bug | A minimal reproduction of a numerical issue in Stable Baselines3's PPO when gSDE is combined with tanh squashing. In float32, tanh can round a sampled value to exactly ±1, so when SB3 inverts the stored action during the update it reconstructs the wrong Gaussian sample: a sample of 12.78 comes back as 8.32, and the loss gradient flips sign (−3.13 → +14.73) even though the PPO ratio is exactly 1. Checked against the analytic gradient and finite differences, with an unsaturated control case that agrees, and confirmed through SB3's own policy methods. Runs against unmodified SB3; the proposed fix (keep the original pre-squash sample) is raised upstream as an SB3 issue. |
Python, PyTorch, Stable Baselines3 | Repo · Issue |
Skills: Numerical debugging · Gradient verification (analytic & finite-difference) · Minimal reproductions · Controlled experiments · Upstream bug reports
Algorithms implemented from scratch to understand them properly
Tree search and policy optimisation, one moving part at a time.
| Project | Description | Stack | Link |
|---|---|---|---|
| MCTS for Gymnasium | Monte Carlo Tree Search from scratch for Gymnasium control tasks, tested on CartPole-v1 — UCB1 selection, child expansion, random rollouts and backpropagation, with parent-detaching to keep the tree's memory in check and a separate config for the exploration constant and search budget. | Python, Gymnasium, NumPy | Repo |
| MCTS Tic-Tac-Toe | Play Tic-Tac-Toe against a vanilla MCTS agent. Selection, expansion, simulation and backpropagation made concrete in a familiar game, with UCB1 exploration (c = 1.41, 1,000 iterations by default) and unit tests. | Python, NumPy | Repo |
| PPO CarRacing | PPO agent for Gymnasium's pixel-based CarRacing-v3 — 96×96 RGB observations, continuous steering, gas and brake — trained for 1M timesteps with Stable Baselines3, plus a video of the trained agent driving. | Python, Stable Baselines3, Gymnasium | Repo · Video |
Skills: Tree search · UCB1 · On-policy RL · Continuous control from pixels · Gymnasium
Tools that make research easier to run, reproduce, and share
Infrastructure around the research: sites, pipelines, and talks.
| Project | Description | Stack | Link |
|---|---|---|---|
| Research Website | Source for my research site: publications, software, teaching and research notes, plus a retrieval-augmented "research guide" served by a Cloudflare Worker. CI checks the strict build, internal links, accessibility (axe-core), responsive layouts (Playwright), and retrieval quality against a maintained evaluation set. | Jekyll, JavaScript, Cloudflare Workers, Playwright | Repo · Site |
| Meeting Transcriber | Transcribes meeting audio locally on CPU with faster-whisper, then sends the transcript to Gemini to produce an executive summary, notes, discussion points and action items, plus a running index of every meeting. Built for PhD supervision meetings; tested on Linux and macOS. | Python, Shell, faster-whisper, Gemini | Repo |
| PhD Progress Reviews | Marp slide decks for my PhD progress reviews. PRP 2 covers MCTS vs PPO for deadline-constrained heating, PPO instability with gSDE and tanh squashing, RL on the BOPTEST heat-pump benchmark, and MCTS with a SINDy-learned model on a physical testbed. | Marp, HTML, CSS | Repo |
Skills: Reproducible experiment software · Retrieval-augmented generation · CI & accessibility testing · Speech-to-text pipelines · Scientific communication
Happy to talk about planning, reinforcement learning, and reproducible AI research.