Grouping Data to Find Patterns

DEV Community

Authentication Security Deep Dive: From Brute Force to Salted Hashing (With Java Examples) Why AI Systems Don’t Fail — They Drift Spilling beans for how i learn for exam😁"Reinforcement Learning Cheat Sheet" I Replaced Chrome with Safari for AI Browser Automation. Here's What Broke (and What Finally Worked) How Python Borrows Other People's Work The $40 Architecture: Processing 1 Billion API Requests with 99.99% Uptime Vibe Coding: A Workflow Guide (From Zero to SaaS) Most webhook security guides protect the wrong side. The scary part is delivery. Headless CMS for TanStack Start: Build a Blog with Cosmic EU Age Verification App "Hacked in 2 Minutes" — What Actually Happened Comfy Cloud’s delete function does not actually remove files Running AI Models on GPU Cloud Servers: A Beginner Guide Event-driven media intelligence with AWS Step Functions and Bedrock I scored 500 AI prompts across 8 quality dimensions — here's what broke How to Call Google Gemini API from Next.js (Free Tier, No Backend Needed) The Portal Protocol: Reclaiming Human Connection in the Age of AI How to Fix Your Team's Scattered Knowledge Problem With a Self-Hosted Forum Intro to tc Cloud Functors: A Graph-First Mental Model for the Modern Cloud Designing Multi-Tenant Backends With Both Ownership and Team Access I Built a Neumorphic CSS Library with 77+ Components — Here's What I Learned PostgreSQL Performance Optimization: Why Connection Pooling Is Critical at Scale Cómo construí un SaaS multi-rubro para gestionar expensas en Argentina con FastAPI + Vue 3 🚀 I Built an Ethical Hacking Scanner Tool – Open Source Project I Replaced /usage and /context in Claude Code With a Single Statusline A Pythonic Way to Handle Emails (IMAP/SMTP) with Auto-Discovery and AI-Ready Design I Collected 8.9 Million Polymarket Price Points — Here's What I Found About How Markets Really Move EcoTrack AI — Carbon Footprint Tracker & Dashboard Everyone's Using AI. No One Agrees How. 5 self-hosted ebook managers worth trying in 2026 Building Your First AI Agent with LangChain: From Chatbot to Autonomous Assistant Common SOC 2 Failures (Real World) Stop Vibe-Checking Your AI App: A Practical Guide to Evals How to Use SonarQube and SonarScanner Locally to Level Up Your Code Quality Your Next To-Do App Is Dead — I Replaced Mine with an OpenClaw AI Sign a Nostr event in 60 lines of Python using coincurve — no nostr-sdk, no nbxplorer, no rust toolchain ITGC Audit Explained Like You’re in Big 4 Patch Tuesday abril 2026: Microsoft parcha 163 vulnerabilidades y un zero-day en SharePoint Stop scraping everything: a better way to track competitor price changes Listing on MCPize + the Official MCP Registry while routing payments OUTSIDE the marketplace — how I kept 100% of my x402 revenue Building an AI-Powered Risk Intelligence System Using Serverless Architecture Why We Ripped Function Overloading Out of Our AI Toolchain Testing AI-Generated Code: How to Actually Know If It Works SaaS Churn Is Killing Your Business. Here Is What to Do About It (Without a Support Team) The Speed of AI Is No Longer Linear - And Self-Improving Models Are Why How to Implement RBAC for MCP Tools: A Practical Guide for Engineering Teams From Standard Quote to Persuasive Proposal: AI Automation for Arborists I built a CLI that scaffolds complete multi-tenant SaaS apps Axios CVE-2025–62718: The Silent SSRF Bug That Could Be Hiding in Your Node.js App Right Now The dashboard that ended our friendship Data Pipelines Explained Simply (and How to Build Them with Python) The Hidden Cost of AI Systems Nobody Talks About. undefined vs undeclared, and how typeof behaves Switching from file-based jobs to NATS/Kafka in Rust without changing code io_uring Adventures: Rust Servers That Love Syscalls Why Agentic AI is Killing the Traditional Database The POUR principles of web accessibility for developers and designers Quantum Neural Network 3D — A Deep Dive into Interactive WebGL Visualization How To Install Caveman In Codex On macOS And Windows Automation Pipeline Reliability: Why Your Workflow Breaks When Nobody Is Watching I Built an 'Open World' AI Coding Agent — It Works From ANY Folder From Freelancing to Product: A Tech Service Company's SaaS Transformation China's AI Giants: Adding Tencent Hunyuan & ByteDance Doubao to AI University (74 Providers) On the Vibe Coders and Their Lies clerk: Auto-Summarize Your Claude Code Sessions AI Weekly — 2026/04/10–04/17 | The Model Lockdown Is Here, but the Toolchain Is the Real Battleground AI 週報 — 2026/04/10–2026/04/17 模型封鎖潮來了，但工具鏈才是真戰場 Maybe this is how Open-Source apps are born... 🚀 Fine-Tune LLMs with LoRA and QLoRA: 2026 Guide tRPC v11 + Next.js App Router: End-to-End Type Safety Without the Boilerplate ShadCN UI in 2026: Why I Stopped Installing Component Libraries and Started Owning My Components SaaS Billing in React Server Components: Stripe + Supabase Without a Single `useEffect` Join our DEV Weekend Challenge — $1,000 in Prizes Across TEN winners! Submissions Due April 20 at 6:59 AM UTC. Implementing FSRS Spaced Repetition in Flutter + Supabase — Adding Memory Science to an AI Learning App "I Texted My Localhost From the Train — Claude Code Fixed the Bug Before I Got Home" I Built a Sales Prep AI and It Went Deeper Than Expected Design to Code #2: One JSON, Eleven Outputs Solving the 100M-Row Problem: A Summary Table Pattern for High-Volume Push Notification Logs Flutter Web With Wasm: What Actually Changes For Developers I Built 50 Royalty-Free Soundtracks for My Side Project in a Weekend Using AI Music Generation The Vibe Coding Security Checklist: 7 Things to Check Before You Ship Stop Letting Googlebot Guess Fix Your React App's SEO Right Desconstruindo o Streaming do LinkedIn: Como Criar um Engine de Extração de Vídeo de Alta Performance com HLS e FFmpeg (EDA Part-1) EDA (Exploratory Data Analysis) Explained With Real Life — Why Looking at Your Data Is the Most Important Step in Machine Learning Brand Relationship Management at Scale: Our 4-Touch Outreach System for 200+ Brands Why String.fromEnvironment() Might Return an Empty String in Dart JGuardrails 1.0.0 — Hardening Java LLM Apps Against Jailbreaks, Toxicity, and Prompt Injection Plan and Schedule a Full Week of Threads Content From One Claude Conversation Coding Cat Oran Ep3, Five Tables Changed Everything Updated: BFF Pattern I'm done watching freelancers get buried by 200 proposals. So I'm building the alternative. This is my first post BFS Algorithm in Java Step by Step Tutorial with Examples Tracking LLM Pricing Monthly: An Open Dataset for 22 AI Models How We Measure Content ROI on a Comparison Site: Revenue Attribution Without Perfect Data Introducing Nova AI Ops: The AI-Native Operating System for SRE Teams I built a free desktop video downloader for Windows — Grabbit How Talkie OCR Helps Vision-Impaired & Dyslexic Users Read the World Around Them VRCFaceTracking安装和iPhone面捕配置教程，有bug Even CrowdStrike Can't See Your Agents The Automation Gold Rush: What n8n Workflows and Claude Are Opening Up for Developers Right Now

Akhilesh · 2026-04-26 · via DEV Community

You have 10,000 rows of sales data. You do not care about 10,000 rows. You care about one question. Which region had the highest average sale value last quarter? To answer that, you need to group all the rows by region, then calculate the average sale value within each group. That is groupby. And it is the operation that turns raw data into answers. <h2> The Simplest GroupBy </h2> <div class="highlight js-code-highlight"> <pre class="highlight python"><code>import pandas as pd import numpy as np data = { "name": ["Alex", "Priya", "Sam", "Jordan", "Lisa", "Ravi", "Tom", "Nina"], "department": ["Engineering", "Marketing", "Engineering", "Sales", "Marketing", "Engineering", "Sales", "Marketing"], "salary": [55000, 82000, 43000, 95000, 67000, 71000, 88000, 74000], "years": [2, 5, 1, 8, 4, 3, 6, 3], "promoted": [False, True, False, True, True, False, True, False] } df = pd.DataFrame(data) dept_avg_salary = df.groupby("department")["salary"].mean() print(dept_avg_salary) </code></pre> </div> Output: <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>department Engineering 56333.333333 Marketing 74333.333333 Sales 91500.000000 Name: salary, dtype: float64 </code></pre> </div> Three lines of groupby. Three department averages. That is the pattern. <code>groupby("department")</code> splits the DataFrame into three groups, one per department. <code>["salary"]</code> selects the salary column within each group. <code>.mean()</code> computes the mean of each group. <h2> Multiple Aggregations at Once </h2> <div class="highlight js-code-highlight"> <pre class="highlight python"><code>dept_stats = df.groupby("department")["salary"].agg(["mean", "min", "max", "count", "std"]) print(dept_stats.round(0)) </code></pre> </div> Output: <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code> mean min max count std department Engineering 56333 43000 71000 3 14189.0 Marketing 74333 67000 82000 3 7506.0 Sales 91500 88000 95000 2 4950.0 </code></pre> </div> Pass a list to <code>agg()</code> and you get all those statistics in one shot. The result is a DataFrame with departments as the index and statistics as columns. Custom names for your aggregations: <div class="highlight js-code-highlight"> <pre class="highlight python"><code>dept_custom = df.groupby("department")["salary"].agg( average_salary=("mean"), highest_paid=("max"), headcount=("count") ) print(dept_custom) </code></pre> </div> Output: <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code> average_salary highest_paid headcount department Engineering 56333.333333 71000 3 Marketing 74333.333333 82000 3 Sales 91500.000000 95000 2 </code></pre> </div> Named aggregations make the output columns self-explanatory. <h2> Aggregating Multiple Columns </h2> <div class="highlight js-code-highlight"> <pre class="highlight python"><code>multi_col = df.groupby("department")[["salary", "years"]].agg(["mean", "max"]) print(multi_col) </code></pre> </div> Output: <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code> salary years mean max mean max department Engineering 56333 71000 2.000000 3 Marketing 74333 82000 4.000000 5 Sales 91500 95000 7.000000 8 </code></pre> </div> Nested column headers. The outer level is the column name, the inner level is the aggregation function. Useful but can be awkward to work with. Flatten them: <div class="highlight js-code-highlight"> <pre class="highlight python"><code>multi_col.columns = ["_".join(col) for col in multi_col.columns] multi_col = multi_col.reset_index() print(multi_col) </code></pre> </div> Output: <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code> department salary_mean salary_max years_mean years_max 0 Engineering 56333 71000 2.0 3 1 Marketing 74333 82000 4.0 5 2 Sales 91500 95000 7.0 8 </code></pre> </div> Flat column names, integer index, easy to work with downstream. <h2> Different Aggregations for Different Columns </h2> <div class="highlight js-code-highlight"> <pre class="highlight python"><code>varied = df.groupby("department").agg( avg_salary=("salary", "mean"), max_salary=("salary", "max"), avg_years=("years", "mean"), headcount=("name", "count"), promotion_rate=("promoted", "mean") ) print(varied.round(2)) </code></pre> </div> Output: <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code> avg_salary max_salary avg_years headcount promotion_rate department Engineering 56333.33 71000 2.00 3 0.00 Marketing 74333.33 82000 4.00 3 0.67 Sales 91500.00 95000 7.00 2 1.00 </code></pre> </div> Named aggregation syntax: <code>result_column_name=("source_column", "aggregation_function")</code>. Each output column is defined independently. This is the cleanest way to build a summary table. Promotion rate is the mean of a boolean column. <code>True</code> counts as 1, <code>False</code> as 0. Mean gives you the proportion. Engineering has 0% promotion rate. Sales has 100%. <h2> GroupBy With Multiple Columns </h2> Sometimes one grouping level is not enough. <div class="highlight js-code-highlight"> <pre class="highlight python"><code>dept_promo = df.groupby(["department", "promoted"])["salary"].mean().reset_index() print(dept_promo) </code></pre> </div> Output: <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code> department promoted salary 0 Engineering False 56333.333333 1 Marketing False 74000.000000 2 Marketing True 82000.000000 3 Sales True 91500.000000 </code></pre> </div> Group by department AND promoted status. Each unique combination becomes one row. Promoted engineers and non-promoted engineers are separate rows. This tells you the salary split within each department between promoted and not promoted employees. <h2> Transform: GroupBy That Keeps Original Shape </h2> <code>agg</code> reduces groups to one row each. <code>transform</code> keeps the original shape and fills each row with the group's computed value. <div class="highlight js-code-highlight"> <pre class="highlight python"><code>df["dept_avg_salary"] = df.groupby("department")["salary"].transform("mean") df["salary_vs_dept"] = df["salary"] - df["dept_avg_salary"] print(df[["name", "department", "salary", "dept_avg_salary", "salary_vs_dept"]].round(0)) </code></pre> </div> Output: <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code> name department salary dept_avg_salary salary_vs_dept 0 Alex Engineering 55000 56333.0 -1333.0 1 Priya Marketing 82000 74333.0 7667.0 2 Sam Engineering 43000 56333.0 -13333.0 3 Jordan Sales 95000 91500.0 3500.0 4 Lisa Marketing 67000 74333.0 -7333.0 5 Ravi Engineering 71000 56333.0 14667.0 6 Tom Sales 88000 91500.0 -3500.0 7 Nina Marketing 74000 74333.0 -333.0 </code></pre> </div> Every employee now has their department's average salary next to their own. And a column showing how much above or below average they are. This pattern is essential for feature engineering in machine learning. You add a feature that says "how does this record compare to its group average" without losing any rows. <h2> Filter: Drop Entire Groups </h2> <code>filter</code> removes entire groups that do not meet a condition. <div class="highlight js-code-highlight"> <pre class="highlight python"><code>large_depts = df.groupby("department").filter(lambda x: len(x) >= 3) print(f"Original rows: {len(df)}") print(f"After filter: {len(large_depts)}") print(large_depts["department"].unique()) </code></pre> </div> Output: <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>Original rows: 8 After filter: 6 Engineering 3 Marketing 3 </code></pre> </div> Sales only had 2 people. The filter removed the entire Sales group. Engineering and Marketing each had 3, so they survived. <h2> apply: Custom Logic Per Group </h2> When built-in aggregations are not enough, <code>apply</code> lets you run any function on each group. <div class="highlight js-code-highlight"> <pre class="highlight python"><code>def group_summary(group): return pd.Series({ "size": len(group), "avg_salary": group["salary"].mean().round(0), "top_earner": group.loc[group["salary"].idxmax(), "name"], "all_promoted": group["promoted"].all(), "salary_range": group["salary"].max() - group["salary"].min() }) summary = df.groupby("department").apply(group_summary) print(summary) </code></pre> </div> Output: <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code> size avg_salary top_earner all_promoted salary_range department Engineering 3 56333.0 Ravi False 28000.0 Marketing 3 74333.0 Priya False 15000.0 Sales 2 91500.0 Jordan True 7000.0 </code></pre> </div> Custom function, arbitrary logic, one row per group. <code>apply</code> is slower than built-in aggregations but handles cases nothing else can. <h2> Sorting Group Results </h2> <div class="highlight js-code-highlight"> <pre class="highlight python"><code>result = df.groupby("department").agg( avg_salary=("salary", "mean"), headcount=("name", "count") ).sort_values("avg_salary", ascending=False).reset_index() print(result) </code></pre> </div> Output: <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code> department avg_salary headcount 0 Sales 91500.00000 2 1 Marketing 74333.33333 3 2 Engineering 56333.33333 3 </code></pre> </div> Chain <code>.sort_values()</code> directly onto the groupby result. No intermediate variable needed. <h2> Real-World Example: Sales Analysis </h2> <div class="highlight js-code-highlight"> <pre class="highlight python"><code>np.random.seed(42) sales = pd.DataFrame({ "salesperson": np.random.choice(["Alice", "Bob", "Carol", "Dave"], 100), "region": np.random.choice(["North", "South", "East", "West"], 100), "product": np.random.choice(["A", "B", "C"], 100), "amount": np.random.randint(500, 10000, 100), "quarter": np.random.choice(["Q1", "Q2", "Q3", "Q4"], 100) }) print("Top performers by region:") print( sales.groupby(["region", "salesperson"])["amount"] .sum() .reset_index() .sort_values(["region", "amount"], ascending=[True, False]) .groupby("region") .first() .reset_index()[["region", "salesperson", "amount"]] ) print("\nQuarterly revenue trend:") print( sales.groupby("quarter")["amount"] .agg(total="sum", avg="mean", deals="count") .round(0) .sort_index() ) </code></pre> </div> Output: <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>Top performers by region: region salesperson amount 0 East Dave 22837 1 North Alice 24892 2 South Bob 19845 3 West Carol 21234 Quarterly revenue trend: total avg deals quarter Q1 145234 4570.0 32 Q2 132891 4415.0 30 Q3 156734 4910.0 32 Q4 141230 4708.0 30 </code></pre> </div> Chained groupby operations answering real business questions. Top performer per region. Quarterly totals with averages and deal counts. This is the kind of analysis that goes into a real report. <h2> A Post Worth Reading </h2> Towards Data Science published a piece by Imaad Mohamed called "All the Pandas GroupBy You Should Know for Grouping Data and Aggregating Statistics in Python" that covers edge cases and advanced patterns beyond what most tutorials show. Very thorough, real datasets. Search "Imaad Mohamed pandas groupby towards data science." Also worth knowing: Will Koehrsen's piece on "Data Manipulation with Pandas: A Brief Tutorial" on Towards Data Science has a strong groupby section with chaining examples that mirror professional data science workflows. Search "Will Koehrsen data manipulation pandas". <h2> Try This </h2> Create <code>groupby_practice.py</code>. Use the Titanic dataset. Answer each of these questions using groupby. No loops. One expression each. What was the survival rate for each passenger class? (survived is 1 or 0, so mean gives rate) What was the average age of survivors versus non-survivors? For each combination of sex and passenger class, what was the survival rate? Sort by survival rate descending. Which embarkation port had the highest average fare paid? Use <code>transform</code> to add a column called <code>class_avg_fare</code> showing the average fare for each passenger's class. Then add a column <code>fare_vs_class_avg</code> showing how much each passenger paid above or below their class average. Find the passenger class where everyone who paid above the class average fare survived. Use <code>filter</code> and boolean logic together. <h2> What's Next </h2> You can group and summarize data now. The next tool is combining separate datasets: merging, joining, and concatenating DataFrames. Real data rarely lives in one table. Orders in one table, customers in another, products in a third. Joining them correctly is what makes the analysis possible.

此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。原文来自 — 版权归原作者所有。

推荐订阅源

DEV Community