惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

K
KPMG report finds enterprise disconnect between AI and its ROI | CIO
博客园 - Franky
V
V2EX
Last Week in AI
Last Week in AI
H
Help Net Security
J
Java Code Geeks
WordPress大学
WordPress大学
阮一峰的网络日志
阮一峰的网络日志
Hugging Face - Blog
Hugging Face - Blog
OSCHINA 社区最新新闻
OSCHINA 社区最新新闻
A
About on SuperTechFans
月光博客
月光博客
腾讯CDC
小众软件
小众软件
罗磊的独立博客
D
Docker
V
Visual Studio Blog
C
CXSECURITY Database RSS Feed - CXSecurity.com
Spread Privacy
Spread Privacy
博客园 - 叶小钗
F
Full Disclosure
Recent Announcements
Recent Announcements
让小产品的独立变现更简单 - ezindie.com
让小产品的独立变现更简单 - ezindie.com
L
LangChain Blog
T
The Exploit Database - CXSecurity.com
宝玉的分享
宝玉的分享
美团技术团队
Exploit-DB.com RSS Feed
Exploit-DB.com RSS Feed
奇客Solidot–传递最新科技情报
奇客Solidot–传递最新科技情报
L
LINUX DO - 热门话题
博客园 - 三生石上(FineUI控件)
T
Tailwind CSS Blog
www.infosecurity-magazine.com
www.infosecurity-magazine.com
cs.AI updates on arXiv.org
cs.AI updates on arXiv.org
S
Securelist
Latest news
Latest news
Project Zero
Project Zero
T
Threat Research - Cisco Blogs
NISL@THU
NISL@THU
K
Kaspersky official blog
O
OpenAI News
T
Tenable Blog
C
Cyber Attacks, Cyber Crime and Cyber Security
Cyberwarzone
Cyberwarzone
Vercel News
Vercel News
有赞技术团队
有赞技术团队
P
Proofpoint News Feed
爱范儿
爱范儿
B
Blog RSS Feed
U
Unit 42

DEV Community

Authentication Security Deep Dive: From Brute Force to Salted Hashing (With Java Examples) Why AI Systems Don’t Fail — They Drift Spilling beans for how i learn for exam😁"Reinforcement Learning Cheat Sheet" I Replaced Chrome with Safari for AI Browser Automation. Here's What Broke (and What Finally Worked) How Python Borrows Other People's Work The $40 Architecture: Processing 1 Billion API Requests with 99.99% Uptime Vibe Coding: A Workflow Guide (From Zero to SaaS) Most webhook security guides protect the wrong side. The scary part is delivery. Headless CMS for TanStack Start: Build a Blog with Cosmic EU Age Verification App "Hacked in 2 Minutes" — What Actually Happened Comfy Cloud’s delete function does not actually remove files Running AI Models on GPU Cloud Servers: A Beginner Guide Event-driven media intelligence with AWS Step Functions and Bedrock I scored 500 AI prompts across 8 quality dimensions — here's what broke How to Call Google Gemini API from Next.js (Free Tier, No Backend Needed) The Portal Protocol: Reclaiming Human Connection in the Age of AI How to Fix Your Team's Scattered Knowledge Problem With a Self-Hosted Forum Intro to tc Cloud Functors: A Graph-First Mental Model for the Modern Cloud Designing Multi-Tenant Backends With Both Ownership and Team Access I Built a Neumorphic CSS Library with 77+ Components — Here's What I Learned PostgreSQL Performance Optimization: Why Connection Pooling Is Critical at Scale Cómo construí un SaaS multi-rubro para gestionar expensas en Argentina con FastAPI + Vue 3 🚀 I Built an Ethical Hacking Scanner Tool – Open Source Project I Replaced /usage and /context in Claude Code With a Single Statusline A Pythonic Way to Handle Emails (IMAP/SMTP) with Auto-Discovery and AI-Ready Design I Collected 8.9 Million Polymarket Price Points — Here's What I Found About How Markets Really Move EcoTrack AI — Carbon Footprint Tracker & Dashboard Everyone's Using AI. No One Agrees How. 5 self-hosted ebook managers worth trying in 2026 Building Your First AI Agent with LangChain: From Chatbot to Autonomous Assistant Common SOC 2 Failures (Real World) Stop Vibe-Checking Your AI App: A Practical Guide to Evals How to Use SonarQube and SonarScanner Locally to Level Up Your Code Quality Your Next To-Do App Is Dead — I Replaced Mine with an OpenClaw AI Sign a Nostr event in 60 lines of Python using coincurve — no nostr-sdk, no nbxplorer, no rust toolchain ITGC Audit Explained Like You’re in Big 4 Patch Tuesday abril 2026: Microsoft parcha 163 vulnerabilidades y un zero-day en SharePoint Stop scraping everything: a better way to track competitor price changes Listing on MCPize + the Official MCP Registry while routing payments OUTSIDE the marketplace — how I kept 100% of my x402 revenue Building an AI-Powered Risk Intelligence System Using Serverless Architecture Why We Ripped Function Overloading Out of Our AI Toolchain Testing AI-Generated Code: How to Actually Know If It Works SaaS Churn Is Killing Your Business. Here Is What to Do About It (Without a Support Team) The Speed of AI Is No Longer Linear - And Self-Improving Models Are Why How to Implement RBAC for MCP Tools: A Practical Guide for Engineering Teams From Standard Quote to Persuasive Proposal: AI Automation for Arborists I built a CLI that scaffolds complete multi-tenant SaaS apps Axios CVE-2025–62718: The Silent SSRF Bug That Could Be Hiding in Your Node.js App Right Now The dashboard that ended our friendship Data Pipelines Explained Simply (and How to Build Them with Python) The Hidden Cost of AI Systems Nobody Talks About. undefined vs undeclared, and how typeof behaves Switching from file-based jobs to NATS/Kafka in Rust without changing code io_uring Adventures: Rust Servers That Love Syscalls Why Agentic AI is Killing the Traditional Database The POUR principles of web accessibility for developers and designers Quantum Neural Network 3D — A Deep Dive into Interactive WebGL Visualization How To Install Caveman In Codex On macOS And Windows Automation Pipeline Reliability: Why Your Workflow Breaks When Nobody Is Watching I Built an 'Open World' AI Coding Agent — It Works From ANY Folder From Freelancing to Product: A Tech Service Company's SaaS Transformation China's AI Giants: Adding Tencent Hunyuan & ByteDance Doubao to AI University (74 Providers) On the Vibe Coders and Their Lies clerk: Auto-Summarize Your Claude Code Sessions AI Weekly — 2026/04/10–04/17 | The Model Lockdown Is Here, but the Toolchain Is the Real Battleground AI 週報 — 2026/04/10–2026/04/17 模型封鎖潮來了,但工具鏈才是真戰場 Maybe this is how Open-Source apps are born... 🚀 Fine-Tune LLMs with LoRA and QLoRA: 2026 Guide tRPC v11 + Next.js App Router: End-to-End Type Safety Without the Boilerplate ShadCN UI in 2026: Why I Stopped Installing Component Libraries and Started Owning My Components SaaS Billing in React Server Components: Stripe + Supabase Without a Single `useEffect` Join our DEV Weekend Challenge — $1,000 in Prizes Across TEN winners! Submissions Due April 20 at 6:59 AM UTC. Implementing FSRS Spaced Repetition in Flutter + Supabase — Adding Memory Science to an AI Learning App "I Texted My Localhost From the Train — Claude Code Fixed the Bug Before I Got Home" I Built a Sales Prep AI and It Went Deeper Than Expected Design to Code #2: One JSON, Eleven Outputs Solving the 100M-Row Problem: A Summary Table Pattern for High-Volume Push Notification Logs Flutter Web With Wasm: What Actually Changes For Developers I Built 50 Royalty-Free Soundtracks for My Side Project in a Weekend Using AI Music Generation The Vibe Coding Security Checklist: 7 Things to Check Before You Ship Stop Letting Googlebot Guess Fix Your React App's SEO Right Desconstruindo o Streaming do LinkedIn: Como Criar um Engine de Extração de Vídeo de Alta Performance com HLS e FFmpeg (EDA Part-1) EDA (Exploratory Data Analysis) Explained With Real Life — Why Looking at Your Data Is the Most Important Step in Machine Learning Brand Relationship Management at Scale: Our 4-Touch Outreach System for 200+ Brands Why String.fromEnvironment() Might Return an Empty String in Dart JGuardrails 1.0.0 — Hardening Java LLM Apps Against Jailbreaks, Toxicity, and Prompt Injection Plan and Schedule a Full Week of Threads Content From One Claude Conversation Coding Cat Oran Ep3, Five Tables Changed Everything Updated: BFF Pattern I'm done watching freelancers get buried by 200 proposals. So I'm building the alternative. This is my first post BFS Algorithm in Java Step by Step Tutorial with Examples Tracking LLM Pricing Monthly: An Open Dataset for 22 AI Models How We Measure Content ROI on a Comparison Site: Revenue Attribution Without Perfect Data Introducing Nova AI Ops: The AI-Native Operating System for SRE Teams I built a free desktop video downloader for Windows — Grabbit How Talkie OCR Helps Vision-Impaired & Dyslexic Users Read the World Around Them VRCFaceTracking安装和iPhone面捕配置教程,有bug Even CrowdStrike Can't See Your Agents The Automation Gold Rush: What n8n Workflows and Claude Are Opening Up for Developers Right Now
The easiest Chinese platform to scrape in Python in 2026: Bilibili in under 30 lines
Sami · 2026-04-30 · via DEV Community

If you've ever tried to scrape RedNote (Xiaohongshu), you know the pain. Request signing that rotates monthly, TLS fingerprinting that blocks requests immediately, residential proxies required, the whole tour. Weibo isn't quite as bad but you still need the Sina Visitor System dance to even hit a public endpoint.

Bilibili is the outlier. No API key. No browser. No proxy. No request signing rotation worth worrying about. Pure HTTP. Runs in 256MB RAM.

If you only need to monitor one Chinese platform — say you're tracking a gaming brand launch, or doing creator research, or analyzing tech trends — Bilibili is where you should start. This post walks through how to scrape it from scratch in Python, what data you actually get, and when to switch from DIY to a hosted scraper.

Why Bilibili is unusually scrape-friendly

Bilibili (哔哩哔哩) is China's YouTube — 300M+ monthly active users, skewed Gen Z and millennials, dominant in anime, gaming, tech, and educational content. From a scraping perspective, three things make it different from RedNote and Weibo:

1. Internal HTTP APIs are mostly stable. Bilibili exposes JSON endpoints for search, video metadata, user info, popular/trending, and comments. Most don't require auth for public content. The endpoints change rarely (months between meaningful updates) compared to RedNote where signing rotates monthly.

2. No TLS fingerprinting. Plain httpx or requests works. You don't need curl_cffi or any Chrome impersonation library to get past anti-bot.

3. Generous rate limits from non-Chinese IPs. I've sustained 1-2 requests per second from a single datacenter IP without getting throttled. RedNote would have banned the IP within minutes at that rate.

There's one caveat I'll cover at the end: comments scraping is throttled from datacenter IPs. Everything else works fine.

A 30-line Bilibili scraper

Here's the smallest useful Bilibili scraper I'd actually run in production:

import httpx

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
    "Referer": "https://www.bilibili.com/",
}


def get_video_detail(bvid: str) -> dict:
    """Fetch full metadata for a Bilibili video by BVID."""
    url = "https://api.bilibili.com/x/web-interface/view"
    response = httpx.get(url, params={"bvid": bvid}, headers=HEADERS, timeout=30)
    response.raise_for_status()
    return response.json().get("data", {})


def get_popular(category_rid: int = 0, page_size: int = 20) -> list:
    """Fetch trending videos. category_rid=0 means all categories."""
    url = "https://api.bilibili.com/x/web-interface/popular"
    params = {"ps": page_size}
    if category_rid:
        params["rid"] = category_rid
    response = httpx.get(url, params=params, headers=HEADERS, timeout=30)
    response.raise_for_status()
    return response.json().get("data", {}).get("list", [])


# Use it
trending = get_popular(page_size=10)
for video in trending:
    stat = video.get("stat", {})
    print(f"{video.get('title')[:60]}")
    print(f"  Views: {stat.get('view'):,} | Coins: {stat.get('coin'):,} | "
          f"Favorites: {stat.get('favorite'):,} | Danmaku: {stat.get('danmaku'):,}")

Enter fullscreen mode Exit fullscreen mode

That's it. No auth, no proxy, no signing. You can paste this into a Python REPL and have working data in 5 seconds.

The five things you can scrape

Bilibili's web client exposes endpoints for five modes. Each is useful for different use cases:

Mode Endpoint Use case
Search /x/web-interface/wbi/search/type Find videos by keyword (Chinese or English). Filterable by sort order, duration, date range.
Video detail /x/web-interface/view Full metadata for a specific video including all engagement metrics
Comments /x/v2/reply/main Comments on a video (caveats below)
User videos /x/space/wbi/arc/search All recent uploads from a creator
Popular /x/web-interface/popular Trending feed, optionally filtered by category

The search endpoint requires a small WBI signing scheme — slightly more involved than the others. Open-source library bilibili-api on GitHub handles it if you don't want to reverse-engineer it yourself. For everything else, plain HTTP works.

What data you actually get back

Each video from Bilibili comes with three categories of engagement metrics:

Standard metrics (similar to YouTube):

  • View count
  • Like count
  • Share count
  • Reply count (comments)

Bilibili-specific metrics (these don't exist on YouTube):

  • Danmaku count (弹幕) — the count of real-time scrolling comments overlaid on the video as users watch
  • Coin count (投币) — Bilibili's tipping system. Users get a few coins per day and "throw" them at videos. Coins are scarce by design.
  • Favorite count (收藏) — equivalent to "save for later"

These extra metrics are non-trivial. A video with 1M views, 50k coins, 30k favorites is meaningfully different from 1M views, 1k coins, 5k favorites — even with identical view counts. The first has an audience that's actively engaged enough to spend their daily coin allocation; the second has passive viewers.

If you're doing creator analytics or content strategy, the engagement quality signals coins/favorites give you are worth the integration effort.

The comments caveat

There's one Bilibili-side restriction worth knowing: /x/v2/reply/main is throttled when called from datacenter IPs (AWS, GCP, Azure, etc.). You'll get the top ~3 pinned comments per video and then nothing. Full pagination requires either:

  • Authenticated session cookies, or
  • Residential IPs

If you need full comments at scale, this is the one part where you'll run into infrastructure cost. Other modes are unaffected.

Performance you can expect

Real numbers from running this on Apify's serverless infrastructure (256MB RAM, no proxy, no auth):

Mode Input Duration Throughput
Search max=50 ~7-8 seconds ~7 items/sec
Popular max=40 ~5 seconds ~8 items/sec
Video detail 10 BVIDs ~5 seconds ~2 items/sec (with tag enrichment)
User videos 3 users, max=15 ~4 seconds ~4 items/sec

Fast enough for monitoring use cases. For heavy bulk extraction (millions of videos) you'd want to parallelize across multiple workers — easy because there's no IP throttling on most endpoints.

DIY vs hosted

DIY is genuinely viable for Bilibili. The maintenance burden is low — endpoints don't change much, there's no signing rotation eating your time. If you only need Bilibili and you're already comfortable with Python HTTP, build it yourself.

But if you're already monitoring multiple Chinese platforms (RedNote + Weibo + Bilibili is the typical full-suite use case), a hosted scraper that handles all three with consistent output schema is worth the convenience. I built and maintain one on Apify Store: zhorex/bilibili-scraper. Honest current state:

  • 16 users
  • 9 monthly active
  • 100% success rate over 2,635 result extractions
  • Average issue response: 4.4 hours
  • $5 per 1,000 results (Apify free tier covers ~1,000/month at no cost)

It supports all five modes above with consistent output JSON, handles WBI signing for search internally, and is paired with companion scrapers for the other Chinese platforms — RedNote (Xiaohongshu) and Weibo. Same pricing model across the suite.

Using it from Python:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_API_TOKEN")

run = client.actor("zhorex/bilibili-scraper").call(run_input={
    "mode": "search",
    "searchQuery": "AI tutorial",
    "sortOrder": "click",       # most-viewed first
    "durationFilter": "medium", # 10-30 min videos
    "maxResults": 50,
})

for video in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(f"{video['title']}{video['viewCount']:,} views, {video['coinCount']:,} coins")

Enter fullscreen mode Exit fullscreen mode

When to use Bilibili specifically

Even if you're already covering YouTube, Twitter, etc., Bilibili captures things those platforms don't:

  • Chinese gaming and esports content — game launches, walkthroughs, and esports event reactions live here
  • Tutorial and educational content — Knowledge category is huge; replaces YouTube tutorials for Chinese audiences
  • Anime and otaku culture — central hub for the Chinese anime community
  • Creator economy in China — the coin/favorite metrics give better creator quality signals than subscriber counts (more on that in a follow-up post)

If your use case touches Chinese-market gaming, anime, tech, or education, Bilibili is irreplaceable.

FAQ

Is scraping Bilibili legal?

Public-data scraping legality varies by jurisdiction. Bilibili's ToS prohibits automated access. The scraping approach treats public web pages as accessible (the same content any logged-out browser visitor can see). Consult legal counsel for your specific use case. Not legal advice.

Do I need a Chinese IP?

For most endpoints, no. Bilibili is globally accessible. The exception is comments scraping (covered above). Some licensed video content (anime, dramas) may be geo-restricted but metadata and engagement metrics are accessible from any IP.

What's a BVID?

Bilibili's video ID format. Looks like BV1YXDfBUETP. Replaced the older numeric aid format. URLs use BVIDs: bilibili.com/video/BV1YXDfBUETP.

How does this compare to YouTube Data API?

YouTube has standardized API access for verified channel owners with quota limits. Bilibili has no equivalent for international developers. For Chinese-market analytics, scraping is the only practical option. The upside: more granular engagement signals (coins, favorites, danmaku) than YouTube exposes.

Why did bilibili-api (the Python lib) work for me a year ago and break now?

Probably the WBI signing scheme rotated. Check the library's commit history; if there's been a recent fix, update. If the project looks abandoned, consider switching libraries or going hosted.


If you're working on Chinese-market analytics, brand monitoring, or creator research and want to compare notes — drop a comment. I write about the build-vs-buy tradeoffs for Chinese platform scraping (RedNote, Weibo, Bilibili).

Hosted actor: apify.com/zhorex/bilibili-scraper