惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

罗磊的独立博客
Apple Machine Learning Research
Apple Machine Learning Research
The Cloudflare Blog
WordPress大学
WordPress大学
钛媒体:引领未来商业与生活新知
钛媒体:引领未来商业与生活新知
奇客Solidot–传递最新科技情报
奇客Solidot–传递最新科技情报
博客园 - 叶小钗
博客园 - 聂微东
阮一峰的网络日志
阮一峰的网络日志
腾讯CDC
博客园 - 三生石上(FineUI控件)
V
V2EX
有赞技术团队
有赞技术团队
V
Visual Studio Blog
小众软件
小众软件
Jina AI
Jina AI
酷 壳 – CoolShell
酷 壳 – CoolShell
博客园 - Franky
量子位
T
Tailwind CSS Blog
cs.CV updates on arXiv.org
cs.CV updates on arXiv.org
P
Palo Alto Networks Blog
Cisco Talos Blog
Cisco Talos Blog
I
Intezer
Project Zero
Project Zero
A
Arctic Wolf
P
Privacy International News Feed
V
Vulnerabilities – Threatpost
L
Lohrmann on Cybersecurity
S
Securelist
C
Cybersecurity and Infrastructure Security Agency CISA
C
CXSECURITY Database RSS Feed - CXSecurity.com
T
Tor Project blog
Hacker News - Newest:
Hacker News - Newest: "LLM"
S
Security @ Cisco Blogs
cs.AI updates on arXiv.org
cs.AI updates on arXiv.org
Threat Intelligence Blog | Flashpoint
Threat Intelligence Blog | Flashpoint
Google DeepMind News
Google DeepMind News
N
News and Events Feed by Topic
TaoSecurity Blog
TaoSecurity Blog
L
LINUX DO - 热门话题
G
GRAHAM CLULEY
Help Net Security
Help Net Security
N
News | PayPal Newsroom
W
WeLiveSecurity
G
Google Developers Blog
Microsoft Security Blog
Microsoft Security Blog
Engineering at Meta
Engineering at Meta
MongoDB | Blog
MongoDB | Blog
C
Check Point Blog

Pierce Freeman

A browser for agents | Pierce Freeman The grey market of podcast appearances The way I travel | Pierce Freeman Fixing slow AWS uploads | Pierce Freeman Local tools should still use vaults We solved scratch content first Starting a podcast in 2025 Being late but still being early Automating our home video imports Adding my parents to tailscale A deep dive on agent sandboxes Language servers for AI | Pierce Freeman My simple home podcast studio We need centralized infrastructure | Pierce Freeman Coercing agents to follow conventions using AST validation My unified theory of social selling My personal backup strategy | Pierce Freeman July updates to the homelab How the KV Cache works httpx is the right way to do web requests in Python Reputation is becoming everything | Pierce Freeman Building a (kind of) invisible mac app Updated knowledge in language models Making an ascii animation | Pierce Freeman How speculative decoding works | Pierce Freeman Under the hood of Claude Code Doing things because they're easy, not hard Speeding up sideeffects with JIT in mountaineer Firehot for hot reloading in Python Misadventures in Python hot reloading How text diffusion works | Pierce Freeman The tenacity of modern LLMs The ergonomics of rails | Pierce Freeman How language servers work | Pierce Freeman Just add eggs | Pierce Freeman Unfortunately SEO still matters | Pierce Freeman The futility of human-only web requirements Setting up Input Leap | Pierce Freeman Checking in on Waymo | Pierce Freeman The react revolution | Pierce Freeman Speeding up many small transfers to a unifi nas Quick notes on swift libraries AI engineering is a different animal San Francisco | Pierce Freeman Debugging a mountaineer rendering segfault Local network config on macOS Building our home network | Pierce Freeman Introducing Envelope.dev | Pierce Freeman Legacy code and AI copilots Typehinting from day-zero | Pierce Freeman Generating database migrations with acyclic graphs Lofoten | Pierce Freeman Mountaineer v0.1: Webapps in Python and React Constraining LLM Outputs | Pierce Freeman Passthrough above all | Pierce Freeman Accuracy in kudos | Pierce Freeman How quick we are to adapt The curious case of LM repetition Costa Rica | Pierce Freeman Debugging chrome extensions with system-level logging Speeding up runpod | Pierce Freeman Inline footnotes with html templates Parsing Common Crawl in a day for $60 An era of rich CLI All or nothing with remote work The Next 10 Years | Pierce Freeman Adding wheels to flash-attention | Pierce Freeman LLMs as interdisciplinary agents | Pierce Freeman New Zealand | Pierce Freeman Representations in autoregressive models | Pierce Freeman Let's talk about Siri | Pierce Freeman Minimum viable public infrastructure | Pierce Freeman Reasoning vs. Memorization in LLMs Automatically migrate enums in alembic Greater sequence lengths will set us free On learning to ski | Pierce Freeman Dolomites | Pierce Freeman Using grpc with node and typescript Opportunity years | Pierce Freeman Buzzword peaks and valleys | Pierce Freeman Buenos Aires | Pierce Freeman Network routing interaction on MacOS Independent work: November recap | Pierce Freeman Debugging slow pytorch training performance The provenance of copy and paste Debugging tips for neural network training Patagonia | Pierce Freeman Santiago | Pierce Freeman My 2022 digital travel kit AWS vs GCP - GPU Availability V2 Independent work: October recap | Pierce Freeman Planning Patagonia | Pierce Freeman Relationship modeling | Pierce Freeman The power of status updates A new chapter | Pierce Freeman Give my library a coffee shop AWS vs GCP - GPU Availability V1 Switzerland | Pierce Freeman Headfull browsers beat headless | Pierce Freeman Webcrawling tradeoffs | Pierce Freeman
Go ahead, self-host Postgres | Pierce Freeman
2025-07-02 · via Pierce Freeman

Self-hosting a database sounds terrifying. That narrative has certainly been pushed over the last 10 years by the big cloud providers:

  • Hosting your own database is dangerous
  • How are you going to get all the 9s of reliability doing it yourself?
  • You'll have access to dedicated database engineers that you couldn't (or wouldn't want to) hire yourself

The rumors obscure the truth.

  • Most cloud hosts are just running a slightly modified version of the open source Postgres server anyway1
  • Database engineering is not a silver bullet if your queries are sub-optimal. Abstracting away your engine too much from your code doesn't let you benchmark what's going on and work around the otherwise reasonable constraints of how the engine is actually querying your code.

I've had data corruption when using a 3rd party vendor just the same as I've had when self-hosting. And with a serious markup, what's the point?

I've been running my own self-hosted postgres for the better part of two years now, serving thousands of users and tens of millions of queries daily2. I expected it would give me much more trouble than it has. It's caused me exactly 30mins of stress during a manual migration and that's all. Aside from that it's been fast, stable, and much cheaper.

I sleep just fine at night thank you.

Blue skies and the white cloud

Let me rewind for a second. The "database as a service" narrative wasn't always the dominant one. From the 80s to the early 2000s, everyone ran their own databases because there wasn't really an alternative. You had your application server and your database server, often on the same physical machine. It was pretty dang fast3 because it was communicating over localhost before forwarding the final payload over the network.

Amazon launched RDS in 2009. The pitch was compelling: we'll handle backups, patching, high availability, and monitoring. You just connect and go. The early pricing was reasonable too - a small RDS instance cost about the same as a dedicated server, but with less operational overhead. If you had to scale your database specs independent from your web service, it made some good sense to delegate responsibility.

The real shift happened around 2015 when cloud adoption accelerated. Companies started to view any infrastructure management as "undifferentiated heavy lifting"4. Running your own database became associated with legacy thinking. A new orthodoxy emerged of focusing on your application logic and letting AWS handle the infrastructure.

Fast forward to 2025 and I hope the pendulum might be swinging back. RDS pricing has grown considerably more aggressive. A db.r6g.xlarge instance (4 vCPUs, 32GB RAM) now costs $328/month before you add storage, backups, or multi-AZ deployment. For that price, you could rent a dedicated server with 32 cores and 256GB of RAM.

Unpacking the cloud

For the most part managed database services aren't running some magical proprietary technology. They're just running the same open-source Postgres you can download with some operational tooling wrapped around it.

Take AWS RDS. Under the hood, it's:

  • Standard Postgres compiled with some AWS-specific monitoring hooks
  • A custom backup system using EBS snapshots
  • Automated configuration management via Chef/Puppet/Ansible
  • Load balancers and connection pooling (PgBouncer)
  • Monitoring integration with CloudWatch
  • Automated failover scripting

None of this is technically complex. The value proposition is operational: they handle the monitoring, alerting, backup verification, and incident response. It's also a production ready configuration at minute zero of your first deployment. But the actual database engine? It's the same Postgres running the same SQL queries with the same performance characteristics.

I helped prove this to myself when I migrated off RDS. I took a pg_dump of my RDS instance, restored it to a self-hosted server with identical specs, and ran my application's test suite. Performance was identical. In some cases, it was actually better because I could tune parameters that RDS locks down.

The real operational complexity

I spent a weekend migrating to a dedicated server from DigitalOcean5: 16 vCPU / 32GB Memory / 400GB disk. The migration took about 4 hours of actual work and then a few weeks to get comfortable with its performance characteristics before migrating our live app.

For my products with high availability requirements, this stack takes me about half an hour per month. For the stacks that get less traffic I'm fully hands off - set it and forget it. This is roughly my cadence for my primary deploys:

Weekly tasks (10 minutes):

  • Check backup verification (automated, just reviewing alerts)
  • Review slow query logs
  • Check disk space trends

Monthly tasks (30 minutes):

  • Apply Postgres security updates
  • Review and rotate backup retention
  • Capacity planning based on growth trends

Quarterly tasks (optional) (2 hours):

  • Update monitoring dashboards
  • Review and optimize configuration parameters
  • Test disaster recovery procedures

The main operational difference is that you're responsible for incident response. If your database goes down at 3 AM, you need to fix it. But here's the thing: RDS goes down too6. And when it does, you're still the one getting paged, you just have fewer tools to fix the problem.

For the most part I've found that unless I'm actively messing with the database, it's really stable. After all you're just renting some remote machine in a data center somewhere. All updates are up to you - so you have a good idea when to schedule the most risky windows.

When self-hosting doesn't make sense

I'd argue self-hosting is the right choice for basically everyone, with the few exceptions at both ends of the extreme:

  1. If you're just starting out in software & want to get something working quickly with vibe coding, it's easier to treat Postgres as just another remote API that you can call from your single deployed app
  2. If you're a really big company and are reaching the scale where you need trained database engineers to just work on your stack, you might get economies of scale by just outsourcing that work to a cloud company that has guaranteed talent in that area. The second full freight salaries come into play, outsourcing looks a bit cheaper.
  3. Regulated workloads (PCI-DSS, FedRAMP, HIPAA, etc.) sometimes require a managed platform with signed BAAs or explicit compliance attestations.

Configurations to look out for

There are a lot of knobs and dials that let you tune Postgres exactly how you want it. At a point that power is helpful, but I find that degree of flexibility overwhelming early on. I've found the following to be the key things to bear in mind when self-hosting:

Memory Configuration: This is where most people mess up. Pulling the standard postgres docker image won't cut it. You have to configure memory bounds with static limits that correspond to hardware. I've automated some of these configurations. But whether you do it manually or use some auto-config, tweaking these params is a must.

The key parameters:

  • shared_buffers: Start around 25 % of RAM; modern PG happily uses tens of GB.
  • effective_cache_size: Set to 75% of system RAM (this tells Postgres how much memory the OS will use for caching)
  • work_mem: Be conservative here. Set it to total RAM / max_connections / 2, or use a fixed value like 32MB
  • maintenance_work_mem: Can be generous (1-2GB), only used during VACUUM and index operations

Connection Management: RDS enforces their own max connections, but when self hosting you get the opportunity to choose your own:

# Connection settings
max_connections = 200
shared_preload_libraries = 'pg_stat_statements'
log_connections = on
log_disconnections = on

Wahoo! More connections = more parallelism right?

No such free lunch I'm afraid. Making fresh connections in postgres has pretty expensive overhead, so you almost always want to put a load balancer on front of it. I'm using pgbouncer on all my projects by default - even when load might not call for it. Python asyncio applications just work better with a centralized connection pooler.

And yes, I've automated some of the config there too.

Storage Tuning: NVMe SSDs make having content on disk less harmful than conventional spinning hard drives, so you'll want to pay attention to the disk type that you're hosted on:

# Storage optimization for NVMe
random_page_cost = 1.1                 # Down from default 4.0
seq_page_cost = 1.0                    # Keep at default
effective_io_concurrency = 200         # Up from default 1

These settings tell Postgres that random reads are almost as fast as sequential reads on NVMe drives, which dramatically improves query planning.

WAL Configuration: Write-Ahead Logging is critical for durability and performance:

# WAL settings
wal_level = replica                     # Enable streaming replication
max_wal_size = 2GB                     # Allow larger checkpoints
min_wal_size = 1GB                     # Prevent excessive recycling
checkpoint_completion_target = 0.9      # Spread checkpoint I/O over 90% of interval

Conclusion

I'm not advocating that everyone should self-host everything. But the pendulum has swung too far toward managed services. There's a large sweet spot where self-hosting makes perfect sense, and more teams should seriously consider it.

Start small. If you're paying more than $200/month for RDS, spin up a test server and migrate a non-critical database. You might be surprised by how straightforward it is.

The future of infrastructure is almost certainly more hybrid than it's been recently: managed services where they add genuine value, self-hosted where they're just expensive abstractions. Postgres often falls into the latter category.

  1. They're either just hosting a vanilla postgres instance that's tied to the deployed hardware config, or doing something opaque with edge deploys and sharding. In the latter case they near guarantee your DB will stay highly available but costs can quickly spiral out of control. ↩

  2. Maybe up to billions at this point. ↩

  3. Even on otherwise absolutely snail speed hardware. ↩

  4. This was Jeff Bezos's favorite phrase during the early AWS days, and it stuck. ↩

  5. Similar options include OVH, Hetzner dedicated instances, or even bare metal from providers like Equinix. ↩

  6. AWS RDS & S3 has had several major outages over the years. The most memorable was the 2017 US-East-1 outage that took down half the internet. ↩