This is a hands-on tutorial about how to use Large Language Models and in particular, train one from scratch. Our goal here, is to train an LLM from scratch, with very low vocabulary, only dog barking.
While there might not be many applications for such a model (unless you want to build a robot dog), I always felt intimidated by LLMs and the fact that they are trained on vast amounts of datasets. Having tiny dataset, with small vocabulary, removes one of the complexities in the learning curves. Unsurprisingly understanding an LLM, even that small is still quite hard.
This project is heavily inspired on CatGPT, an AI that meows. I am a cat owner myself, although the market for meow AIs feels quite saturated with clear market dominance. I haven't seen any worthy dog GPTs, only mere amateur projects, so I think this project will just fit right in.
Try demo at BarkGPT
To get our BarkLLM from scratch, we need to do the following steps:
This repository is split into series of self-contained guides. Each one builds on the previous, but you can jump around if you want.
This is the foundation where we define a tiny vocabulary, build a GPT‑like transformer by hand, train it on synthetic data and generate tokens using pure PyTorch.
You’ll see very clearly how next‑token prediction works, why prompt contracts exist and how hallucinations naturally emerge.
If you only read one document, read this one.
Once the toy model makes sense, we move to something closer to reality. Here we introduce real text datasets and add a real language tokenizer
In this part, we stop thinking like ML engineers and start thinking like system builders. Here our model is ready to be used in chat interfaces, we wrap it in langchain, serve it with FastAPI and deep dive into why not using LLM /generate directly, and why is building extra chat assistant adapter layer.
This is essentially a miniature version of how real LLM systems are shipped.
In this part, we will examine how to get the ready language model and deploy it on AWS. For demonstration purposes, an actual small language model was used: Qwen. We will see how to work with Gguf files and package them reliably to serve the inferrence. As an extra mile, we will try to apply the same engineering approach to deploy our improved BarkGPT using improved compression with Gguf.
Even tiny BarkGPT can take a while to train. By moving to Mac MPS, tuning batch size, using gradient accumulation, and scaling the learning rate, epoch time dropped from ~250s to ~80s while keeping loss decreasing. Faster experiments, same learning—small tweaks, big difference!
A deep dive into BarkGPT’s internals. We break down embeddings, transformer layers, attention heads, and the forward pass, then show how to make the model smarter by tuning configuration - not rewriting code. This is where BarkGPT goes from "working" to "well-designed."