惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

Apple Machine Learning Research
Apple Machine Learning Research
Google DeepMind News
Google DeepMind News
小众软件
小众软件
GbyAI
GbyAI
酷 壳 – CoolShell
酷 壳 – CoolShell
F
Fortinet All Blogs
博客园 - 三生石上(FineUI控件)
B
Blog
量子位
B
Blog RSS Feed
Vercel News
Vercel News
Blog — PlanetScale
Blog — PlanetScale
Last Week in AI
Last Week in AI
博客园 - 叶小钗
MongoDB | Blog
MongoDB | Blog
Cyber Security Advisories - MS-ISAC
Cyber Security Advisories - MS-ISAC
钛媒体:引领未来商业与生活新知
钛媒体:引领未来商业与生活新知
爱范儿
爱范儿
Jina AI
Jina AI
C
Check Point Blog
freeCodeCamp Programming Tutorials: Python, JavaScript, Git & More
IT之家
IT之家
H
Hackread – Cybersecurity News, Data Breaches, AI and More
云风的 BLOG
云风的 BLOG

Arpit Bhayani

Temporal Primer - Building Long-Running Systems What Matters in Production RAG Structure of Every LLM Chat How LLMs Really Work Your Monolith Is Already A Distributed System Databases Were Not Designed For This BM25 JOIN Algorithms Venting at Work Comes at a Reputation Cost Why Half Your Skills Expire Every Few Years Multi-Paxos - Consensus in Distributed Databases MySQL Replication Internals Bloom Filters When You Increase Kafka Partitions Product Quantization The Q, K, V Matrices The Day I Accidentally Deleted Production How LLM Inference Works What are Blocking Queues and Why We Need Them Heartbeats in Distributed Systems How Writes Work in Apache Cassandra Redis Replication Internals How to Handle Arrogant Colleagues at Work How Does a CDN Handle Content Replication You Can't Fix Everything on Day One When Emotions Spill Over at Work Why gRPC Uses HTTP2 Meetings With No Agenda Are a Waste of Time Career Longevity Beats Constant Job Hopping Stay Relevant at Higher Salary Levels
The Best Engineering Lessons Happen During Outages
Arpit Bhayani · 2024-05-29 · via Arpit Bhayani

I get excited whenever there is a production outage, because I know I will be learning something new and interesting very soon. I always hop on the call, even when I am not on call :)

Being part of stressful situations and seeing how seniors fixed the issues and operated with a calm head made me a better engineer. Reading old postmortem documents (RCAs) of outages at all my workplaces became a habit.

If you are an engineer early in your career, do not shy away from being on-call and spend some time reading RCAs. They are filled with interesting, practical, real-world insights about the systems, coding blunders, tuning parameters, etc.

To be very honest, you will learn more from being on-call than literally anything out there.

By the way, I have a massive playlist on my YouTube. I dissected 18 production outages of GitHub, Atlassian, Spotify, etc., and went deeper into the details not even mentioned in their blogs. The link to the playlist is in the comments, in case you want to check them out.

One interesting outage that happened at GitHub happened because their primary key went beyond the max value, and it had a ton of details on how they fixed it. It had a ton of details about running data migrations without downtimes.