惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

K
Kaspersky official blog
Engineering at Meta
Engineering at Meta
D
DataBreaches.Net
Stack Overflow Blog
Stack Overflow Blog
Microsoft Security Blog
Microsoft Security Blog
Y
Y Combinator Blog
B
Blog RSS Feed
GbyAI
GbyAI
P
Proofpoint News Feed
aimingoo的专栏
aimingoo的专栏
MyScale Blog
MyScale Blog
D
Docker
阮一峰的网络日志
阮一峰的网络日志
Cyber Security Advisories - MS-ISAC
Cyber Security Advisories - MS-ISAC
Recorded Future
Recorded Future
美团技术团队
The Register - Security
The Register - Security
V
Visual Studio Blog
H
Hackread – Cybersecurity News, Data Breaches, AI and More
T
Tailwind CSS Blog
爱范儿
爱范儿
freeCodeCamp Programming Tutorials: Python, JavaScript, Git & More
T
The Blog of Author Tim Ferriss
博客园 - 司徒正美
量子位
B
Blog
F
Fortinet All Blogs
Martin Fowler
Martin Fowler
博客园 - 【当耐特】
MongoDB | Blog
MongoDB | Blog
A
About on SuperTechFans
I
InfoQ
钛媒体:引领未来商业与生活新知
钛媒体:引领未来商业与生活新知
有赞技术团队
有赞技术团队
雷峰网
雷峰网
大猫的无限游戏
大猫的无限游戏
J
Java Code Geeks
L
LangChain Blog
Latest news
Latest news
S
SegmentFault 最新的问题
Exploit-DB.com RSS Feed
Exploit-DB.com RSS Feed
Blog — PlanetScale
Blog — PlanetScale
Threat Intelligence Blog | Flashpoint
Threat Intelligence Blog | Flashpoint
Cisco Talos Blog
Cisco Talos Blog
F
Full Disclosure
C
Cisco Blogs
D
Darknet – Hacking Tools, Hacker News & Cyber Security
W
WeLiveSecurity
T
Tenable Blog
T
Tor Project blog

Stonecharioteer on Tech

I Traced My Traffic Through a Home Tailscale Exit Node What Was I Reading Last? In Three Not-So-Easy Pieces Dogfooding Is Hard Code blocks in your books, finally GoForGo v0.9.0 Merrilin - We built an app to read books I use a Macbook now Data Structures & Algorithms - Preparing for Interviews Using a local DNS namespace for local service discovery Direction KOllector - Publishing KOReader Highlights gbt: branches touched in the last 24 hours A Soiree into Symbols in Ruby Some Smalltalk about Ruby Loops Ruby Blocks Returning from Ruby Blocks, Procs and Lambdas My Linux Laptop Finally Works: How Claude Helped Me Fix Years of Annoyances TIL: Watchexec - Modern File Watching for Development Workflows A Less Busy Mind GoForGo - Learn Go through live examples Migrating My Old Blog to Hugo with Claude The Qtile Window Manager: A Python-Powered Tiling Experience Read the RFCs that Built the Internet Py-x-Protobuf - Or How I Learned to Stop Worrying and Love Protocol Buffers Python Reverse a List New Beginnings Leaving ChainSafe Systems Screen Lock for Cinnamon Desktop using Zenity and Terminal Commands Crews Not Teams A System for Getting Better at LeetCode So Far So Rust Retrying HTTP Requests with Rust A Primer on Control Charts Learning Rust Explicit is Better than Implicit: Rust for Pythonistas Using Custom Delimiters in Jinja Templates TIL: Creating Fixed Length Iterables in Python Documentation Without Assumption Vagrant Python - A Reflection in 2022 Learning Golang No, A Virtual Machine Is Not Enough: Why Developers Need Native Linux Empathy in Tech For Those Who Came in Late A Weekend With PostgreSQL TIL: Gooey and Python Fire for Quick GUIs and CLIs TIL: 2ality - Dr. Axel Rauschmayer's JavaScript Blog TIL: MassDNS - High-Performance Bulk DNS Lookups TIL: Matomo Analytics, Google Tech Writing, Memory Programming, and NES TV Signals TIL: MontyDB - MongoDB Implemented in Python Returning to the Craft of Programming TIL: CPUFetch, OneFetch, and Learn CSS TIL: DNS Performance Testing and Pi-hole with Unbound TIL: Eli Bendersky's Blog, Awesome By Example, NoCoDB, and Martin Kleppmann TIL: CRDTs, Extreme HTTP Performance, and BYTEPATH Game TIL: AutoInvent, ASGI, Python Packaging, RAPIDS GPU Computing, and FlaskCon TIL: MangaDesk - Terminal Client for MangaDex TIL: McFly - Smart Shell History Search TIL: Siege Load Testing and Awesome FastAPI Resources TIL: Ventoy Bootable USB and Justniffer Network Analysis TIL: CLI Code Review, Git Split Diffs, and Internal Combustion Engine TIL: Benford's Law, Web Security Headers, Event Sourcing, and Mozilla Security Guidelines How to Write Documentation - The README.md File The Importance of Documentation TIL: NNgroup UX Research, SponsorBlock, and Labella Python Library TIL: The Little Book of Rust Macros and Rust Performance Book TIL: Git-Bug Distributed Issue Tracker and Omni Kubernetes Monitoring TIL: Zellij - Modern Terminal Multiplexer TIL: How Discord Handles 2.5 Million Concurrent Voice Users TIL: Volumio - The Audiophile Music Player TIL: Areopagitica - Milton's Defense of Free Speech TIL: Fast Node Manager, Zoxide Smart CD, Technical Writing, PyO3, and Qubes OS TIL: Slurm Workload Manager for HPC Clusters TIL: Data Visualization Guide and Oso Authorization Academy TIL: CORS Deep Dive, Piku Tiny PaaS, Rust Strings, and Deno Standard Library TIL: Raspberry Pi OS Development, Vim Beginner Guide, Password Management, and QueryBook TIL: uBlock Origin Performance Optimization on Firefox TIL: Breaking PostgreSQL at Scale and LeetCode Problem Patterns TIL: Awesome Tmux Resources for Terminal Multiplexing TIL: Grit - A Multitree-Based Personal Task Manager TIL: Lens 4.2 Kubernetes IDE, Shell Scripting Guide, and Dark HTTP Server Do The Job You Hate So You Won't Hate The Job You Love TIL: Innernet VPN Solution and NoteCalc Calculator App TIL: Argo CD for GitOps and Lens Kubernetes IDE TIL: Modern Rust CLI Tools - System Monitoring, HTTP Requests, and DNS TIL: tz - A Time Zone Helper Tool TIL: Distributed Systems Education, Fallacies, and Self-Hosted Internet Archiving TIL: Real-Time Voice Cloning Technology TIL: ChartMuseum for Helm, AMD's Corporate Journey, and Kubernetes Pod Scaling TIL: Docker and Kubernetes Tools - Whaler, Descheduler, and Dive TIL: Post-Mortem Collection, Terminal Plotting, and Technical Twitter TIL: Dark Mode Toggle Web Component by Google Chrome Labs TIL: Python eval(), exec(), and compile() Functions TIL: Camelot PDF Tables, PostgreSQL Row Level Security, Zerodha Varsity, and Write Yourself a Git TIL: fuser Command for Process and File Investigation TIL: i Hate Regex - The Ultimate Regex Cheat Sheet TIL: Dolt - Git for Data and Database Version Control TIL: x86 Assembly Programming and SafeEyes Break Reminder TIL: Comprehensive Distributed Systems Reading List TIL: Cosmopolitan C Library, Distributed Systems Book, High Performance Browser Networking, and Rust Roguelike Tutorial
TIL: PostgreSQL Customization and Advanced Database Management
2020-11-08 · via Stonecharioteer on Tech

Today I discovered comprehensive techniques for customizing PostgreSQL environments and learned advanced database management practices that significantly improve developer and DBA productivity.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
-- ~/.psqlrc - PostgreSQL shell customization

-- Set better default formatting
\set QUIET 1
\pset null '[NULL]'
\pset linestyle unicode
\pset border 2
\pset format wrapped
\set COMP_KEYWORD_CASE upper
\set HISTSIZE 10000
\set PROMPT1 '%[%033[1m%]%M %n@%/%R%[%033[0m%]%# '
\set PROMPT2 '[more] %R > '
\unset QUIET

-- Timing for performance monitoring
\timing

-- Show expanded output for wide tables
\x auto

-- Better error handling
\set ON_ERROR_ROLLBACK interactive
\set ON_ERROR_STOP on

-- Helpful shortcuts and aliases
\set version 'SELECT version();'
\set extensions 'SELECT * FROM pg_available_extensions ORDER BY name;'
\set settings 'SELECT name, setting, unit, context FROM pg_settings ORDER BY name;'
\set locks 'SELECT * FROM pg_locks ORDER BY pid;'
\set activity 'SELECT pid, usename, application_name, client_addr, state, query_start, query FROM pg_stat_activity ORDER BY query_start DESC;'
\set blocking 'SELECT blocked_locks.pid AS blocked_pid, blocked_activity.usename AS blocked_user, blocking_locks.pid AS blocking_pid, blocking_activity.usename AS blocking_user, blocked_activity.query AS blocked_statement, blocking_activity.query AS current_statement_in_blocking_process FROM pg_catalog.pg_locks blocked_locks JOIN pg_catalog.pg_stat_activity blocked_activity ON blocked_activity.pid = blocked_locks.pid JOIN pg_catalog.pg_locks blocking_locks ON blocking_locks.locktype = blocked_locks.locktype AND blocking_locks.DATABASE IS NOT DISTINCT FROM blocked_locks.DATABASE AND blocking_locks.relation IS NOT DISTINCT FROM blocked_locks.relation AND blocking_locks.page IS NOT DISTINCT FROM blocked_locks.page AND blocking_locks.tuple IS NOT DISTINCT FROM blocked_locks.tuple AND blocking_locks.virtualxid IS NOT DISTINCT FROM blocked_locks.virtualxid AND blocking_locks.transactionid IS NOT DISTINCT FROM blocked_locks.transactionid AND blocking_locks.classid IS NOT DISTINCT FROM blocked_locks.classid AND blocking_locks.objid IS NOT DISTINCT FROM blocked_locks.objid AND blocking_locks.objsubid IS NOT DISTINCT FROM blocked_locks.objsubid AND blocking_locks.pid != blocked_locks.pid JOIN pg_catalog.pg_stat_activity blocking_activity ON blocking_activity.pid = blocking_locks.pid WHERE NOT blocked_locks.GRANTED;'

-- Database size information
\set dbsize 'SELECT datname, pg_size_pretty(pg_database_size(datname)) as size FROM pg_database ORDER BY pg_database_size(datname) DESC;'
\set tablesize 'SELECT schemaname, tablename, pg_size_pretty(pg_total_relation_size(schemaname||''.''||tablename)) as size FROM pg_tables ORDER BY pg_total_relation_size(schemaname||''.''||tablename) DESC;'

-- Index usage statistics
\set indexusage 'SELECT schemaname, tablename, attname, n_distinct, correlation FROM pg_stats WHERE schemaname = ''public'' ORDER BY n_distinct DESC;'
\set indexsize 'SELECT schemaname, tablename, indexname, pg_size_pretty(pg_relation_size(indexname::regclass)) as size FROM pg_indexes WHERE schemaname = ''public'' ORDER BY pg_relation_size(indexname::regclass) DESC;'

-- Connection information
\set conninfo 'SELECT usename, count(*) FROM pg_stat_activity GROUP BY usename ORDER BY count DESC;'

-- Slow queries (requires pg_stat_statements)
\set slowqueries 'SELECT query, calls, total_time, mean_time, rows FROM pg_stat_statements ORDER BY total_time DESC LIMIT 10;'

-- Replication status
\set replication 'SELECT * FROM pg_stat_replication;'

-- Vacuum and analyze status
\set vacuum_stats 'SELECT schemaname, tablename, last_vacuum, last_autovacuum, last_analyze, last_autoanalyze FROM pg_stat_user_tables ORDER BY last_vacuum DESC NULLS LAST;'
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
-- Query optimization and performance tuning

-- 1. Identify slow queries with detailed analysis
WITH slow_queries AS (
    SELECT
        query,
        calls,
        total_time,
        mean_time,
        max_time,
        stddev_time,
        rows,
        100.0 * shared_blks_hit / NULLIF(shared_blks_hit + shared_blks_read, 0) AS cache_hit_ratio,
        shared_blks_read,
        shared_blks_written,
        temp_blks_read,
        temp_blks_written
    FROM pg_stat_statements
    WHERE mean_time > 100  -- Queries averaging more than 100ms
    OR total_time > 10000  -- Queries with high total time
    ORDER BY total_time DESC
)
SELECT
    LEFT(query, 100) as query_snippet,
    calls,
    ROUND(total_time::numeric, 2) as total_time_ms,
    ROUND(mean_time::numeric, 2) as mean_time_ms,
    ROUND(max_time::numeric, 2) as max_time_ms,
    rows,
    ROUND(cache_hit_ratio::numeric, 2) as cache_hit_pct,
    CASE
        WHEN temp_blks_written > 0 THEN 'Uses temp files'
        WHEN cache_hit_ratio < 95 THEN 'Poor cache utilization'
        WHEN mean_time > 1000 THEN 'Very slow average'
        ELSE 'Review for optimization'
    END as optimization_hint
FROM slow_queries
LIMIT 20;

-- 2. Index recommendations based on query patterns
WITH table_scans AS (
    SELECT
        schemaname,
        tablename,
        seq_scan,
        seq_tup_read,
        idx_scan,
        idx_tup_fetch,
        n_tup_ins + n_tup_upd + n_tup_del as modifications
    FROM pg_stat_user_tables
    WHERE schemaname = 'public'
),
index_candidates AS (
    SELECT
        schemaname,
        tablename,
        seq_scan,
        seq_tup_read,
        CASE
            WHEN seq_scan > 0 THEN seq_tup_read / seq_scan
            ELSE 0
        END as avg_seq_read,
        idx_scan,
        modifications,
        CASE
            WHEN seq_scan > idx_scan AND seq_tup_read > 10000
            THEN 'High sequential scan activity - consider indexing'
            WHEN modifications > (idx_scan + seq_scan) * 10
            THEN 'High modification rate - index overhead may be significant'
            ELSE 'Normal access pattern'
        END as recommendation
    FROM table_scans
)
SELECT * FROM index_candidates
WHERE recommendation LIKE 'High%'
ORDER BY avg_seq_read DESC;

-- 3. Connection and lock analysis
WITH connection_analysis AS (
    SELECT
        datname,
        usename,
        application_name,
        state,
        COUNT(*) as connection_count,
        AVG(EXTRACT(EPOCH FROM (now() - query_start))) as avg_query_duration,
        MAX(EXTRACT(EPOCH FROM (now() - query_start))) as max_query_duration,
        SUM(CASE WHEN state = 'idle in transaction' THEN 1 ELSE 0 END) as idle_in_transaction
    FROM pg_stat_activity
    WHERE pid != pg_backend_pid()
    GROUP BY datname, usename, application_name, state
)
SELECT
    *,
    CASE
        WHEN idle_in_transaction > 0 THEN 'Idle transactions detected - may cause blocking'
        WHEN max_query_duration > 300 THEN 'Long-running queries detected'
        WHEN connection_count > 100 THEN 'High connection count'
        ELSE 'Normal'
    END as alert
FROM connection_analysis
ORDER BY connection_count DESC;

This comprehensive exploration of PostgreSQL customization and management demonstrates how proper configuration and monitoring can transform database administration from reactive troubleshooting to proactive optimization.

These PostgreSQL insights from my archive showcase the evolution from basic database usage to advanced administration practices, emphasizing the importance of proper tooling and systematic monitoring for database reliability and performance.