AI Inference learnings
Experiments (Threads/Blogs/Articles)
| # | Topic | Link |
|---|---|---|
| 1 | Inference comparison | PyTorch vs vLLM — Qwen2.5-1.5B-Instruct Comparison |
| 2 | kv cache | Why do we need KV cache? |
| 3 | prefill and decode | What actually happens inside an LLM when you send a prompt? |
Learning Resources
Prefill decode basics redis tech blog about p/d
Disaggregated infernce distServe - split p/d in diff gpus
Continuous Batching for LLM Inference — How batching works and how to optimize for better performance.
Quantization of LLMs — How to Quantize models for consumer gpus.
Profiling in pytorch Basic profiling of matmul
LangGraph Internals How LangGraph works