惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

雷峰网
雷峰网
博客园 - 聂微东
酷 壳 – CoolShell
酷 壳 – CoolShell
宝玉的分享
宝玉的分享
奇客Solidot–传递最新科技情报
奇客Solidot–传递最新科技情报
罗磊的独立博客
Hugging Face - Blog
Hugging Face - Blog
T
Tailwind CSS Blog
OSCHINA 社区最新新闻
OSCHINA 社区最新新闻
IT之家
IT之家
博客园_首页
博客园 - 三生石上(FineUI控件)
博客园 - 叶小钗
Apple Machine Learning Research
Apple Machine Learning Research
freeCodeCamp Programming Tutorials: Python, JavaScript, Git & More
量子位
钛媒体:引领未来商业与生活新知
钛媒体:引领未来商业与生活新知
人人都是产品经理
人人都是产品经理
美团技术团队
小众软件
小众软件
Jina AI
Jina AI
S
SegmentFault 最新的问题
博客园 - Franky
让小产品的独立变现更简单 - ezindie.com
让小产品的独立变现更简单 - ezindie.com

Common Crawl

Common Crawl - Blog - Introducing the AI Visibility Audit Common Crawl - Blog - Host- and Domain-Level Web Graphs March, April, and May 2026 Common Crawl - Blog - May 2026 Crawl Archive Now Available Common Crawl - Blog - April 2026 Crawl Archive Now Available in a Hugging Face Storage Bucket Common Crawl - Blog - You can now build directly on Common Crawl from the browser Common Crawl - Blog - Host- and Domain-Level Web Graphs February, March, and April 2026 Common Crawl - Blog - April 2026 Crawl Archive Now Available Common Crawl - Blog - April 2026 Common Crawl Newsletter Common Crawl - Blog - Announcing a Change to Common Crawl Dataset Size Reporting Common Crawl - Blog - Host- and Domain-Level Web Graphs January, February, and March 2026 Common Crawl - Blog - March 2026 Crawl Archive Now Available Common Crawl - Blog - IPv6 Adoption Across the Top 100K Web Hosts Common Crawl - Blog - Web Graph Statistics Gets a Proper Upgrade Common Crawl - Blog - Measuring Web Accessibility from Crawl Archives Common Crawl - Blog - Announcing the Whirlwind Tour of Common Crawl's Datasets Using Java Common Crawl - Blog - Host- and Domain-Level Web Graphs December 2025 and January/February 2026 Common Crawl - Blog - Introducing the New Examples & Resources Browser Common Crawl - Blog - February 2026 Crawl Archive Now Available Common Crawl - Blog - AI Plumbers at FOSDEM’26 Common Crawl - Blog - CC-Citations: A Visualization of Research Papers Referencing Common Crawl Common Crawl - Blog - CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data Common Crawl - Blog - Host- and Domain-Level Web Graphs November/December 2025 and January 2026 Common Crawl - Blog - January 2026 Crawl Archive Now Available Common Crawl - Blog - Web Archives for Social Sciences Datathon, Bristol Common Crawl - Blog - How SEOs Are Using Common Crawl's Web Graph Data for AI Ranking Signals Common Crawl - Blog - GneissWeb Annotations Examples Common Crawl - Blog - Common Crawl at the Mozilla Festival 2025 Common Crawl - Blog - December 2025 Crawl Archive Now Available Common Crawl - Blog - A Sampling of 2025 Research Referencing Common Crawl Common Crawl - Blog - Host- and Domain-Level Web Graphs September, October, and November 2025
Common Crawl - Blog - Host- and Domain-Level Web Graphs O...
2025-12-23 · via Common Crawl

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of October, November, and December 2025. The crawls used to generate the graphs were CC-MAIN-2025-43, CC-MAIN-2025-47, and CC-MAIN-2025-51. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior Web Graph Releases. You may also visit the projects cc-webgraphand cc-pyspark which include all scripts and tools required to construct the graphs. Instructions to explore the graphs in the webgraph format are given in our collection of Web Graph Notebooks. You may also wish to explore our Web Graph Statistics page for more information on ranking.

Host-level Graph

The host-level graph consists of 250.8 million nodes and 10.9 billion edges.

There are 191.6 million dangling nodes (76.40%) and the largest strongly connected component contains 38.0 million (15.15%) nodes. Dangling nodes stem from:

  • Hosts that have not been crawled, yet are pointed to from a link on a crawled page
  • Hosts without any links pointing to a different host name
  • Hosts which did only return an error page (eg. HTTP 404).

Host names in the graph are in reverse domain name notation and a leading www. is stripped: www.subdomain.example.com becomes com.example.subdomain.

You can download the graph and the ranks of all 250.8 million hosts from AWS S3 on the path s3://commoncrawl/projects/hyperlinkgraph/cc-main-2025-oct-nov-dec/host/ (this requires an account on AWS). Alternatively, you can use https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2025-oct-nov-dec/host/ as prefix to access the files from everywhere.

Please note that the text representation of the host-level graph is shipped in 114 gzip-compressed files listed in two path listings - one for the nodes (vertices), one for the edges (arcs). First, download the paths listing and decompress it using gzip -d or gunzip. By adding the prefix s3://commoncrawl/ or https://data.commoncrawl.org/ to each line in the path listing you get the list of URLs to download the entire graph.

Download files of the Common Crawl October, November, and December 2025 host-level Web Graph

Domain-level Graph

The domain graph is built by aggregating the host graph on the level of pay-level domains (PLDs) based on the public suffix list maintained on publicsuffix.org. Version (commit) ea600e4 of the public suffix list was used (commit date 2025-12-04T07:50:00Z).

The domain-level graph has 121.3 million nodes and 6.2 billion edges. 61.8% or 75.0 million nodes are dangling nodes, the largest strongly connected component covers 32.8 million or 27.01% of the nodes.

All files related to the domain graph are available on AWS S3 under s3://commoncrawl/projects/hyperlinkgraph/cc-main-2025-oct-nov-dec/domain/ or on https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2025-oct-nov-dec/domain/.

Download files of the Common Crawl October, November, and December 2025 domain-level Web Graph

Credits

Thanks to the authors of the WebGraph framework, whose software made the computation of graph properties and ranks possible. We hope the data will be useful for you to do any kind of research on ranking, graph analysis, link spam detection, etc.

Let us know about your results via Common Crawl's Google Group, or on our Discord Server.