🤖 Harold's Notes
Search
Search
Search
Explorer
ML
Biology
AlphaFold
Thoughts about AI in biology in 2024
Diffusion models
Architecture
DiT
Network architectures
Time embeddings
Frameworks - Theory
Discrete Diffusion
Discrete Diffusion (D3PM)
Resources
Score Entropy Discrete Diffusion Models (SEDD)
Diffusion Forcing
Guidance
KL-divergence
Score-based Generative models
Shortcut & Consistency models
VAEs
Variational diffusion
Training
Convergence in Diffusion Models
Evaluation metrics
Training diffusion models
Novel stuff
The diffusion process
Engineering
Coding
Abstract Syntax Tree
Einops && Einsum
Installing CUDA on RTX laptop
Local workflows
Neat tricks
Overmind
pdb& pdbplus
py-spy
Python project layout with uv
SLURM
trio
viztracer
CPUs
ggml
Quantization formats
GPU programming
Kernels implementation
How to implement kernel ops in Pytorch
Kernels 101
Mamba scan kernel
PMPP book
PMPP book
PMPP book - Heterogenous data parallel computing
PMPP book - Introduction
PMPP book - Multidimensional grids and data
Profiling
Nsight systems
Triton
Beginner steps into Triton
Implementing Flash Attention forward pass
Block Matrix Multiplication
CUDA Essentials
CUDA logical & physical layout
CUDA streams
Deriving Flash Attention forward pass
Fenwick Tree
Prefix Sum - Scan algorithm
Tensor Layouts
Tips
GPUs
Bandwidth
GPU architecture
GPU memory tiling
Hardware
Memory
Tensor cores
TFLOPS, MFU
Memory
Memory usage (VRAM)
ML instant math
Moving bits
Nanotron
Differential Distributed Primitives
Model definition
Nanotron code organisation
Parameter sharding
ZeRO-3 vs 3D Parallelism
Precision
FP8
FP8 training
FP8 Training Stability
FP8-LM Training FP8 Large Language Models
PyTorch FP8
Resources
e
Floating point numbers
Kahan summation
NVFP4 format
Pretraining Large Language Models with NVFP4
Rounding
Transformer Engine
Pytorch
ExecuTorch
Backends
XNNPACK
GEMMs
XNNPACK in Executorch
How to write a partitioner & backend delegate
QNN (Qualcomm)
Kernels
Custom C++ operator
Kernels
Llama implementation
Workflow
LLM workloads
Invoking the runtime
Utils for LLMs
Benchmarking
Core Aten ops
Edge Compilation
How it works
The 0-1 specialization problem in Pt2 export
torch.export()
Compiler stack
Launching a distributed training run
Pytorch Internals
torch.autograd
torch.compile
torch.distributed
Torchrun
Training
Networking
Inter-node networking
Intra-node networking
Parallelism
DeepSpeed
Deepspeed Notes
FSDP
DTensor
FSDP (Fully Sharded Data Parallel)
FSDP code
FSDP2
FSDP2 code
3D Parallelism
Distributed primitives
Expert Parallelism
Model Parallelism
NCCL
Pipeline Parallelism
Sequence Parallelism
Tensor Parallelism
YaFSDP
Zero Redundancy Optimizer
Profiling
Holisitic Trace Analysis (HTA)
MFU calculation
Pytorch memory profiling
Activation checkpointing
Backpropagation
Compilers
Diagnostics
Fused optimizer
Sharded Matmuls
General
Classical ML
Errors in ML (bias-variance tradeoff)
Generative vs discriminative modeling
Optimizers
Adam
Muon
Straight-Through Estimator
Theory
Linear Algebra
NTK
Representer theorem
Meta Learning
Random thoughts
LLMs
Architecture
Mixture of Experts (MoE)
Mixture of Experts (MoE) summary
Sequence-mixing operators
Comparing gated convolutions and attention
Compressed Convolutional Attention (CCA)
DeltaNet
Linear RNNs and State Space Models (SSMs)
Literature and ideas
Multi-Head Latent Attention (MLA)
Unified representation of sequence models
Transformers
Beyond GPT
Details of GPTs
Long Context Attention
Positional Embeddings
Self-Attention
Transformer
Batch Normalization
Beyond BPE tokenization; H-Nets
Encoder-decoder models
Multi-Token Prediction (MTP)
Physics of LLM - Canon layers
Data
SFT-IFT
Chain of Thought or Reasoning
Data
FLAN (Finetuned Language Net)
LIMA
Summary
Tricks
Synthetic Data
Code
Magicoder - OSS Instruct
OpenCoder
Qwen2.5-Coder Technical Report
WizardCoder
Autonomous Data Selection with Language Models
Evol-Instruct
MAmmoTH2 - Scaling Instructions from the Web
Text Tokenization
Evaluation
MCQ normalization
Metrics
Finetuning
Control Vectors
Spectrum - Target Training on Signal to Noise Ratio
Inference
Sampling
Basic sampling techniques
Chain-of-Thought (CoT) Paradigm
Entropy-based sampling
Looping prevention
nano-vllm
Paged Attention
Quantifying GEMM regimes - Arithmetic intensity
Speculative decoding
vLLM details
Multi-modality
Image-text
Isaac model (from Perceptron)
MANZANO
Molmo and PixMo
PaliGemma
Tokenization
Audio
Filter Banks and Log-Mel Spectrograms
How audio is represented on computers
mu-law
Neural audio codecs or tokenizers
Quantization, SNR, Bits, and Sample Rate
Video
3D Haar Wavelet Transform
How video is represented on computers
Neural video codecs
4M tokenization
Finite scalar quantization (FSQ) - VQ-VAE Made Simple
VQ-VAE
Video
Dreamer 4
Fréchet Video Distance (FVD)
3D positional embeddings
4M
Contrastive learning (CLIP, SigLIP)
Masked modeling
Mixed-modal generative modeling
MultiModN - Multimodal, Multi-Task, Interpretable Modular Networks
Recap of the multimodal landscape
Scaling Laws For Generative Mixed-Modal Language Models
Training multimodal models
ViT (Vision Transformers)
Quantization
llama.cpp
GGUF quantization
i-quants
Importance matrix
k-quants implementation
Post-training quantization (PTQ)
Broad overview of PTQ
EoRA - LoRA-style adapter quantization error compensation
GPTQ-like methods
Hadamard matrices
Quantization methods with rotations
Using orthogonal matrices for better quantization - The math
Quantization-aware training (QAT)
ParetoQ - Scaling Laws for Low-bit LLM Quantization
Quantization-aware training (QAT)
Literature
Pruning
Quantization basics
Sparsity
RL
Codebases
DPO
Reasoning RL Resources
Reward model
RL basics
Thoughts on verification
Scaling
mu-Transfer
Implementing u-mup
mu-Transfer
Resources
Spectral mup
Spectral Norm
Summary
u-mup
Unit scaling
Chinchilla
Golden rules for scaling deep neural networks
Scaling Laws
Scaling Laws for Batch Size
Scaling Laws for Neural Language Models
Scaling Laws for Transfer
T5
Tricks to reduce instabilities
Resources to read
Architecture
Distributed Training
GPU programming
Inference
Misc
Multi-modal
Must-reads
Post-training & test-time compute
Profiling
PyTorch
Research
Scaling
Training & optimization
Useful tools
Home
❯
ML
❯
LLMs
❯
Multi modality
❯
Tokenization
❯
Video
❯
How video is represented on computers
How video is represented on computers
Dec 06, 2025
1 min read
Topics to approach:
transcoding (see cosmos paper)
aspect ratios
Graph View
Backlinks
No backlinks found