惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

Engineering at Meta
Engineering at Meta
酷 壳 – CoolShell
酷 壳 – CoolShell
Forbes - Security
Forbes - Security
S
Schneier on Security
Cyberwarzone
Cyberwarzone
L
Lohrmann on Cybersecurity
NISL@THU
NISL@THU
T
Tenable Blog
Simon Willison's Weblog
Simon Willison's Weblog
G
GRAHAM CLULEY
Schneier on Security
Schneier on Security
K
Kaspersky official blog
AI
AI
Help Net Security
Help Net Security
Hacker News: Ask HN
Hacker News: Ask HN
Security Archives - TechRepublic
Security Archives - TechRepublic
T
Threatpost
Attack and Defense Labs
Attack and Defense Labs
I
Intezer
The Cloudflare Blog
博客园 - 叶小钗
Threat Intelligence Blog | Flashpoint
Threat Intelligence Blog | Flashpoint
Blog — PlanetScale
Blog — PlanetScale
freeCodeCamp Programming Tutorials: Python, JavaScript, Git & More
K
KPMG report finds enterprise disconnect between AI and its ROI | CIO
IT之家
IT之家
Recent Announcements
Recent Announcements
奇客Solidot–传递最新科技情报
奇客Solidot–传递最新科技情报
aimingoo的专栏
aimingoo的专栏
The GitHub Blog
The GitHub Blog
宝玉的分享
宝玉的分享
T
The Blog of Author Tim Ferriss
Y
Y Combinator Blog
O
OpenAI News
大猫的无限游戏
大猫的无限游戏
让小产品的独立变现更简单 - ezindie.com
让小产品的独立变现更简单 - ezindie.com
量子位
Microsoft Security Blog
Microsoft Security Blog
B
Blog
Jina AI
Jina AI
Hugging Face - Blog
Hugging Face - Blog
D
Docker
Cyber Security Advisories - MS-ISAC
Cyber Security Advisories - MS-ISAC
美团技术团队
Microsoft Azure Blog
Microsoft Azure Blog
H
Help Net Security
阮一峰的网络日志
阮一峰的网络日志
雷峰网
雷峰网
H
Hackread – Cybersecurity News, Data Breaches, AI and More
G
Google Developers Blog

ByteByteGo Newsletter

MCP vs A2A vs ACP: How AI Agents Actually Talk to Each Other A Guide to Multi-Tenancy: Benefits and Challenges AI Customer Support at Scale: The Travel Industry’s $Billion Bet How LLMs Learn to Be Helpful (RLHF vs DPO) How Microsoft Ships AI Agents at Enterprise Scale EP221: How Docker Works Under the Hood LAST CALL FOR ENROLLMENT: Become an AI Engineer - Cohort 7 Streaming vs Batch: Two Philosophies of Data Processing The Agent Loop: How AI Goes From Answering Questions to Doing Things ChatGPT vs Gemini vs Claude: How They Differ LAST CALL FOR ENROLLMENT: Become an AI Engineer - Cohort 7 Proof of Human: How to Verify a Person Is Real and Unique Multi-Region Architecture: Going Global Without Going Broke How OpenAI Delivers Low-Latency Voice AI for 900M Users Inside Thinking Machines’ Interaction Models How AI Agents Manage Memory and Avoid Forgetfulness EP220: RAG vs Graph RAG vs Agentic RAG Top Anti-Patterns to Avoid in Service Architecture Large Language Models vs Small Language Models An Ex-Meta L8’s Agentic Engineering Setup AI-Native Leaders: The Organizational Playbook for Engineering Transformation at Scale EP219: 12 Open-source LLMs Observability for Beginners: Logs, Metrics, Traces, and Everything Around Them LAST CALL FOR ENROLLMENT: Build with Claude Code - Cohort 2 How Open-Weight Models Changed the AI Landscape A Guide to AI Inference Engineering EP218: The Typical AI Agent Stack, Explained Must- Know Deployment Strategies: From Big-Bang to Progressive Delivery Love Teaching? ByteByteGo Is Hiring Part-Time AI & Engineering Instructors What Salesforce Learned from 20,000 Enterprise Agent Deployments Token Spend Out of Control? The Case for Smarter Routing EP217: Latency vs Throughput vs Bandwidth The Path of a Request: A Tour of Modern Web Architecture How OpenAI Built Its Data Agent A Practical Guide to Becoming an AI-Native Engineer How DoorDash Built a Testing System to Evaluate LLMs Must-Know Failure Modes in Distributed Systems How Airtable Built the Search Layer Behind Their AI Features How Vercel Cut Build Wait Times From 90 Seconds To 5 How CockroachDB Built Vector Indexing at Scale EP216: RAGs vs Agents 🚀 New cohort based course launch: Build with Claude Code A Guide to Async Patterns in API Design How Netflix is Using Multimodal AI to Power Video Search How Snapchat Serves a Billion Predictions Per Second How Grab is Using AI Agents to Boost Team Productivity EP215: The Anatomy of an AI Agent LAST CALL FOR ENROLLMENT: Become an AI Engineer - Cohort 6 A Guide To Event-Driven Architectural Patterns High Performance Rate Limiting at Databricks How Figma Upgraded Data Pipeline from Multi-Day Latency to Real-Time How Pinterest Built a Production MCP Ecosystem EP214: Claude Code vs. OpenClaw: 5 Design Dimensions Become an AI Engineer | Enrollment Ends Soon Container Design Patterns for Distributed Systems How Instacart Built a Search for Billions of Products Connecting LLMs to the Real World: Tool Use, Function Calling, and MCP EP213: MCP vs Skills, Clearly Explained A Beginner’s Guide to Kubernetes The Tech Stack Powering Wise How Stripe Detects Fraudulent Transactions Within 100 ms How Amazon Uses LLMs to Recommend Products B-Trees vs LSM Trees: Comparison and Trade-Offs How DoorDash Launches a New Country in One Week The Security Architecture of GitHub Agentic Workflow EP211: How the JVM Works A Guide to Relational Database Design Figma Design to Code, Code to Design: Clearly Explained How LinkedIn Feed Uses LLMs to Serve 1.3 Billion Users EP210: Monolithic vs Microservices vs Serverless Must-Know Cross-Cutting Concerns in API Development How Spotify Ships to 675 Million Users Every Week Without Breaking Things Nextdoor’s Database Evolution: A Scaling Ladder A Guide to Context Engineering for LLMs EP209: 12 Claude Code Features Every Engineer Should Know Our New Book on Behavioral Interviews Is Now Available on Amazon Database Performance Strategies and Their Hidden Costs How Datadog Redefined Data Replication How Meta Turned Debugging Into a Product How Roblox Uses AI to Translate 16 Languages in 100 Milliseconds EP208: Load Balancer vs API Gateway LAST CALL FOR ENROLLMENT: Become an AI Engineer - Cohort 5 How to Implement API Security How Anthropic’s Claude Thinks How Netflix Live Streams to 100 Million Devices in 60 Seconds How Agentic RAG Works? Last Chance to Enroll | Become an AI Engineer | Cohort-Based Course EP207: Top 12 GitHub AI Repositories Event Sourcing Explained: Benefits and Use Cases How OpenAI Codex Works
EP212: Data Warehouse vs Data Lake vs Data Mesh
ByteByteGo · 2026-04-25 · via ByteByteGo Newsletter

If slow QA processes bottleneck you or your software engineering team and you’re releasing slower because of it — you need to check out QA Wolf.

QA Wolf’s AI-native service supports web and mobile apps, delivering 80% automated test coverage in weeks and helping teams ship 5x faster by reducing QA cycles to minutes.

QA Wolf takes testing off your plate. They can get you:

  • Unlimited parallel test runs for mobile and web apps

  • 24-hour maintenance and on-demand test creation

  • Human-verified bug reports sent directly to your team

  • Zero flakes guarantee

The benefit? No more manual E2E testing. No more slow QA cycles. No more bugs reaching production.

With QA Wolf, Drata’s team of 80+ engineers achieved 4x more test cases and 86% faster QA cycles.

Schedule a demo to learn more

This week’s system design refresher:

  • Coding Agents Explained: How Claude Code, Codex & Cursor Actually Work (Youtube video)

  • Data Warehouse vs Data Lake vs Data Mesh

  • API Concepts Every Software Engineer Should Know

  • Polling vs Long Polling vs Webhooks vs SSE

  • SLA vs SLO vs SLI

  • Build with Claude Code — Course Direction Survey

Storing data is the easy part. Deciding where and how to organize it is the real challenge.

Image

A data warehouse is the traditional approach. It cleans and structures data before storing it. Queries run fast, and reports stay consistent. But adding a new data source takes effort because everything has to fit the schema first.

A data lake takes the opposite approach. It stores everything raw, like databases, logs, images, and video. Process it when you need it. The flexibility is great, but if rules around naming, formatting, and ownership are not properly set, you end up with duplicate, outdated, and undocumented data that is hard to manage.

Data mesh shifts data ownership from a central team to individual departments. For example, sales publishes sales data, and finance publishes finance data. Shared standards keep things compatible across teams.

It works well in larger organizations. But it requires every team to have the right people and processes to manage their data quality, documentation, and access, which is a challenge.

In practice, many companies use more than one approach. They'll use a warehouse for dashboards and reporting, a lake for machine learning workloads and start applying mesh principles as teams scale.

Most engineers use APIs every day. Sending a request and reading JSON is one thing. Designing an API that other people can rely on is something where things get complicated.

Image

A lot of problems begin with basic HTTP details that seem small at first. Methods, status codes, request formats, and response structure can make an API feel clear and predictable, or confusing and inconsistent.

Then there are the bigger design choices. REST, GraphQL, gRPC, webhooks, and WebSockets each make sense in different situations. The challenge is knowing what actually fits the system and the use case.

A lot of API problems also comes from design decisions that do not get enough attention early on. Naming, pagination, versioning, error responses, and backward compatibility often decide whether an API is easy to work with or frustrating to maintain.

Security is another area where weak decisions can cause real problems. API keys, OAuth, JWTs, scopes, and permissions are easy to mention. Getting them right is harder, and mistakes here can be costly.

Reliability matters too. Timeouts, retries, idempotency, rate limits, and caching are often easy to ignore until the system is under pressure.

And once an API starts growing, the supporting work matters too. Clear documentation, solid specs, observability, and contract testing make it much easier for teams to trust the API and use it without guessing how it works.

Over to you: What’s the most overlooked API concept in your experience?

Four ways to get updates from a server. Each one makes a different tradeoff between simplicity, efficiency, and real-time delivery.

Image

Here's how they compare:

  • Polling: The client sends a request every few seconds asking "anything new?" The server responds immediately, whether or not there's new data. Most of those requests come back empty, wasting client and server resources. For use cases like an order status page where a small delay is acceptable, polling is the simplest option to implement.

  • Long Polling: The client sends a request, and the server keeps the HTTP connection open until new data is available or a timeout occurs. This means fewer empty responses compared to regular polling. Some chat applications used this pattern to deliver messages closer to real-time communication.

  • Server-Sent Events (SSE): The client opens a persistent HTTP connection, and the server streams events through it as they're generated. It is one-way, lightweight, and built on plain HTTP. Many AI responses that appear token by token are delivered through SSE, streaming each chunk over a single open connection.

  • Webhooks: Instead of the client asking for updates, the service sends an HTTP POST to a pre-registered callback URL whenever a specific event occurs. Stripe uses this for payment confirmations. GitHub uses it for push events. The client never polls or holds a connection open, it just waits for the server to call.

Many systems don't rely on a single pattern. You may use polling for order status, SSE for streaming AI responses, and webhooks for payment confirmations.

These three terms are related, but they mean different things. Knowing the difference helps you define what to measure, aim for, and promise your customers.

Image

Here's how they actually connect:

  • SLI (Service Level Indicator): This is the metric you're measuring. For a login service, it could be the ratio of successful login requests to total valid requests. It tells you how your service is performing right now.

  • SLO (Service Level Objective): You take that SLI and define a target around it. Something like "login availability should stay above 99.9% over a rolling 28-day window." When you're missing your SLO, it’s a signal to find out what's failing before customers notice.

  • SLA (Service Level Agreement): This is what you promise your customers in a contract. It's usually set lower than the SLO, say 99.5% monthly availability. If you breach it, you owe service credits.

If your SLO and SLA are both set to 99.9%, then the moment your availability drops below 99.9%, you've already breached the agreement.

The SLI tells you where you stand. The SLO tells you where you should be. The SLA tells your customers what they can expect.

Over to you: How do you decide what the right SLO target is when you're launching a new service?

We’re building a new course, Build with Claude Code, and we’d love your input before we finalize it.

If you’re an engineer or engineering leader, we’d appreciate 3 minutes of your time. Your answers will directly shape what we cover. Thank you so much!

Take the short survey

No posts