Welcome to Harold's Notes

Hi, this is Harold. I’m a Member of Technical Staff at Liquid AI. I document here my learning notes. The content spans many aspects of ML research and engineering.

Good notes

• Multimodal
  • Recap of the multimodal landscape
  • Mixed-modal generative modeling
  • Training multimodal models
  • Scaling Laws For Generative Mixed-Modal Language Models
• Scaling
  • Golden rules for scaling deep neural networks
  • Scaling Laws
  • u-mup, Spectral Norm
• Diffusion
  • The diffusion process
  • VAEs, Variational diffusion
  • Training diffusion models
• Quantization
  • Using orthogonal matrices for better quantization - The math
  • Quantization methods with rotations
• Random engineering notes
  • Deriving Flash Attention forward pass
  • Fenwick Tree
  • Prefix Sum - Scan algorithm
  • Entropy-based sampling

Resources I keep pointing people to

• Machine Learning Engineering Open Book
• CUDA MODE youtube channel
• Understanding Diffusion Models: A Unified Perspective
• Sasha Rush youtube channel

My favorite researcher/engineers

• Zeyuan Allen-Zhu
• Jeremy Howard
• Tri Dao
• Jason Wei
• Horace He
• Noam Shazeer
• Songlin Yang
• Lucas Beyer