My personal bloghttps://astro-paper.pages.dev/https://astro-paper.pages.dev/posts/techniques-for-kv-cache-optimization/https://astro-paper.pages.dev/posts/techniques-for-kv-cache-optimization/This post explores techniques for optimizing the Key-Value (KV) cache in large language models, from Grouped-query attention to PagedAttention and distributed cache management.Sun, 25 Feb 2024 08:00:00 GMThttps://astro-paper.pages.dev/posts/understanding-how-llm-inference-works-with-llama-cpp/https://astro-paper.pages.dev/posts/understanding-how-llm-inference-works-with-llama-cpp/In this post we will understand how large language models (LLMs) answer user prompts by exploring the source code of llama.cpp, a C++ implementation of LLaMA, covering subjects such as tokenization, embedding, self-attention and sampling.Sat, 11 Nov 2023 16:00:00 GMThttps://astro-paper.pages.dev/posts/iops-the-silent-killer-of-cloud-databases/https://astro-paper.pages.dev/posts/iops-the-silent-killer-of-cloud-databases/Despite advancements in cloud infrastructure and storage technology, IOPS is still a significant bottleneck for cloud databases. This post explains the source of this bottleneck and techniques to solve it.Sun, 20 Aug 2023 08:00:00 GMT