推荐订阅源

http://zh.wikipedia.org/wiki/Robots.txt

http://zh.wikipedia.org/wiki/%E7%B6%B2%E7%AB%99%E5%9C%B0%E5%9C%96

http://www.sitemaps.org/zh_CN/

http://www.qq.com/robots.txt

User-agent: *
Disallow:  
Sitemap: http://www.qq.com/sitemap_index.xml

http://www.qq.com/sitemap_index.xml

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap><loc>http://news.qq.com/news_sitemap.xml.gz</loc><lastmod>2011-07-24</lastmod></sitemap>
  ...
</sitemapindex>

http://news.qq.com/news_sitemap.xml.gz

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url><loc>http://news.qq.com/a/20110724/000721.htm</loc><lastmod>2011-07-24T12:08:38+08:00</lastmod><changefreq>daily</changefreq><priority>0.8</priority></url>
  ...
</urlset>