惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

B
Blog
Microsoft Security Blog
Microsoft Security Blog
Jina AI
Jina AI
博客园 - 叶小钗
J
Java Code Geeks
博客园 - 聂微东
博客园 - 司徒正美
大猫的无限游戏
大猫的无限游戏
阮一峰的网络日志
阮一峰的网络日志
V
V2EX
美团技术团队
WordPress大学
WordPress大学
M
MIT News - Artificial intelligence
雷峰网
雷峰网
酷 壳 – CoolShell
酷 壳 – CoolShell
GbyAI
GbyAI
罗磊的独立博客
T
The Blog of Author Tim Ferriss
aimingoo的专栏
aimingoo的专栏
T
Tailwind CSS Blog
The Cloudflare Blog
Stack Overflow Blog
Stack Overflow Blog
N
Netflix TechBlog - Medium
小众软件
小众软件

Common Crawl

Common Crawl - Blog - Introducing the AI Visibility Audit Common Crawl - Blog - Host- and Domain-Level Web Graphs March, April, and May 2026 Common Crawl - Blog - May 2026 Crawl Archive Now Available Common Crawl - Blog - April 2026 Crawl Archive Now Available in a Hugging Face Storage Bucket Common Crawl - Blog - You can now build directly on Common Crawl from the browser Common Crawl - Blog - Host- and Domain-Level Web Graphs February, March, and April 2026 Common Crawl - Blog - April 2026 Crawl Archive Now Available Common Crawl - Blog - April 2026 Common Crawl Newsletter Common Crawl - Blog - Announcing a Change to Common Crawl Dataset Size Reporting Common Crawl - Blog - Host- and Domain-Level Web Graphs January, February, and March 2026 Common Crawl - Blog - March 2026 Crawl Archive Now Available Common Crawl - Blog - IPv6 Adoption Across the Top 100K Web Hosts Common Crawl - Blog - Web Graph Statistics Gets a Proper Upgrade Common Crawl - Blog - Measuring Web Accessibility from Crawl Archives Common Crawl - Blog - Announcing the Whirlwind Tour of Common Crawl's Datasets Using Java Common Crawl - Blog - Introducing the New Examples & Resources Browser Common Crawl - Blog - February 2026 Crawl Archive Now Available Common Crawl - Blog - AI Plumbers at FOSDEM’26 Common Crawl - Blog - CC-Citations: A Visualization of Research Papers Referencing Common Crawl Common Crawl - Blog - CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data Common Crawl - Blog - Host- and Domain-Level Web Graphs November/December 2025 and January 2026 Common Crawl - Blog - January 2026 Crawl Archive Now Available Common Crawl - Blog - Web Archives for Social Sciences Datathon, Bristol Common Crawl - Blog - How SEOs Are Using Common Crawl's Web Graph Data for AI Ranking Signals Common Crawl - Blog - GneissWeb Annotations Examples Common Crawl - Blog - Common Crawl at the Mozilla Festival 2025 Common Crawl - Blog - Host- and Domain-Level Web Graphs October, November, December 2025 Common Crawl - Blog - December 2025 Crawl Archive Now Available Common Crawl - Blog - A Sampling of 2025 Research Referencing Common Crawl Common Crawl - Blog - Host- and Domain-Level Web Graphs September, October, and November 2025
Common Crawl - Blog - Host- and Domain-Level Web Graphs D...
2026-02-24 · via Common Crawl

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of December, January, and February 2025-26. The crawls used to generate the graphs were CC-MAIN-2025-51, CC-MAIN-2026-04, and CC-MAIN-2026-08. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior Web Graph Releases. You may also visit the projects cc-webgraph and cc-pyspark which include all scripts and tools required to construct the graphs. Instructions to explore the graphs in the webgraph format are given in our collection of Web Graph Notebooks. You may also wish to explore our Web Graph Statistics page for more information on ranking.

Host-level Graph

The host-level graph consists of 288.6 million nodes and 12.4 billion edges.

There are 219.3 million dangling nodes (76.00%) and the largest strongly connected component contains 40.4 million (14.00%) nodes. Dangling nodes stem from:

  • Hosts that have not been crawled, yet are pointed to from a link on a crawled page
  • Hosts without any links pointing to a different host name
  • Hosts which did only return an error page (eg. HTTP 404).

Host names in the graph are in reverse domain name notation such that: www.example.com becomes com.example.www.

You can download the graph and the ranks of all 288.6 million hosts from AWS S3 on the path s3://commoncrawl/projects/hyperlinkgraph/cc-main-2025-26-dec-jan-feb/host/ (this requires an account on AWS). Alternatively, you can use https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2025-26-dec-jan-feb/host/ as prefix to access the files from everywhere.

Please note that the text representation of the host-level graph is shipped in 240 gzip-compressed files listed in two path listings - one for the nodes (vertices), one for the edges (arcs). First, download the paths listing and decompress it using gzip -d or gunzip. By adding the prefix s3://commoncrawl/ or https://data.commoncrawl.org/ to each line in the path listing you get the list of URLs to download the entire graph.

Download files of the Common Crawl December, January, and February 2025-26 host-level Web Graph

Domain-level Graph

The domain graph is built by aggregating the host graph on the level of pay-level domains (PLDs) based on the public suffix list maintained on publicsuffix.org. Version (commit) b3b6791 of the public suffix list was used (commit date 2026-01-29T19:28:48Z).

The domain-level graph has 134.2 million nodes and 5.4 billion edges. 65.2% or 87.6 million nodes are dangling nodes, the largest strongly connected component covers 33.2 million or 24.73% of the nodes.

All files related to the domain graph are available on AWS S3 under s3://commoncrawl/projects/hyperlinkgraph/cc-main-2025-26-dec-jan-feb/domain/ or on https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2025-26-dec-jan-feb/domain/.

Download files of the Common Crawl December, January, and February 2025-26 domain-level Web Graph

Credits

Thanks to the authors of the WebGraph framework, whose software made the computation of graph properties and ranks possible. We hope the data will be useful for you to do any kind of research on ranking, graph analysis, link spam detection, etc.

Let us know about your results via Common Crawl's Google Group, or on our Discord Server.