Fig. I — Filed by category
25 meditations filed under Machine Learning. Back to all categories orthe full archive.
LoRA Deep Dive · Part 3
Every LoRA config file has the same handful of lines — r, alpha, dropout, targetmodules, learning rate — and almost everyone (me included, at first) fills them in by copying a config that worked for someone else.
Read entryLoRA Deep Dive · Part 2
LoRA solved half the memory problem: with adapters, the trainable state — gradients and optimizer moments — collapses to almost nothing.
Read entryLoRA Deep Dive · Part 1
Full fine-tuning of a large language model is brutally expensive — not because of the forward pass, but because training multiplies memory.
Read entryML Foundations · Part 4
Of all the ideas in machine learning, this is the one that genuinely changed how I see the field: neural networks turn meaning into geometry.
Read entryML Foundations · Part 3
Most of machine learning research is messy and empirical — try something, see if the benchmark goes up. Then, in 2020, a team at OpenAI published a result with a completely different character.
Read entryML Foundations · Part 2
In 2017, a paper with the almost arrogant title "Attention Is All You Need" replaced decades of accumulated architecture research with one mechanism.
Read entryML Foundations · Part 1
Every impressive thing a neural network does — writing code, recognizing faces, translating languages — comes down to one shockingly simple loop repeated billions of times: make a guess, measure how wrong it was, nudge every parameter a tiny bit in the dire…
Read entry