惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

P
Privacy International News Feed
A
Arctic Wolf
Security Latest
Security Latest
雷峰网
雷峰网
V2EX - 技术
V2EX - 技术
钛媒体:引领未来商业与生活新知
钛媒体:引领未来商业与生活新知
S
Schneier on Security
WordPress大学
WordPress大学
J
Java Code Geeks
宝玉的分享
宝玉的分享
T
The Exploit Database - CXSecurity.com
T
Troy Hunt's Blog
Scott Helme
Scott Helme
爱范儿
爱范儿
罗磊的独立博客
Apple Machine Learning Research
Apple Machine Learning Research
Application and Cybersecurity Blog
Application and Cybersecurity Blog
I
Intezer
博客园 - 【当耐特】
T
Threat Research - Cisco Blogs
L
LINUX DO - 最新话题
W
WeLiveSecurity
K
Kaspersky official blog
Google Online Security Blog
Google Online Security Blog
IT之家
IT之家
N
News and Events Feed by Topic
The Hacker News
The Hacker News
Know Your Adversary
Know Your Adversary
小众软件
小众软件
博客园 - 叶小钗
Latest news
Latest news
P
Proofpoint News Feed
G
GRAHAM CLULEY
Schneier on Security
Schneier on Security
T
Tor Project blog
cs.CL updates on arXiv.org
cs.CL updates on arXiv.org
T
Tailwind CSS Blog
让小产品的独立变现更简单 - ezindie.com
让小产品的独立变现更简单 - ezindie.com
Recent Commits to openclaw:main
Recent Commits to openclaw:main
Project Zero
Project Zero
The Cloudflare Blog
美团技术团队
大猫的无限游戏
大猫的无限游戏
S
Secure Thoughts
量子位
奇客Solidot–传递最新科技情报
奇客Solidot–传递最新科技情报
博客园_首页
Jina AI
Jina AI
V
Visual Studio Blog
Help Net Security
Help Net Security

博客园 - 涤生

Enterprise Library Caching Application Block(CAB)源代码研究之缓存条目清理后台线程篇 了解ASP.NET底层架构(网络上的翻译) EditPlus中的正则表达式使用--如何使用查找到的原字符 Spring.Net学习笔记一——入门、第一个例子 CodeSmith4.0.2连接SqlServer2005――不允许远程连接 many2many的写法与注意点 Nhibernate继承类的写法 One2One主键关联的实现 NHibernate基础学习时遇到的问题 NHibernate学习之一:Many2One遇到的问题 权限学习--BlueDavy之技术Blog漫谈权限系统之结尾篇(开源产品、个人观点、知识体系) 权限学习--BlueDavy之技术Blog漫谈权限系统之基于ACL的实现 权限设计学习篇--BlueDavy之技术Blog - 漫谈权限系统之技术策略以及基于RBAC ... 权限设计-之学习篇--来源于BlueDavy之技术Blog Oracle数据库发生Ora-12162错误 Linux下Oracle文件的自动备份与ftp自动上传 BugFree安装手记 CodeSmith3.0x中Oracle的访问 Asp.Net StartKits之Asp.Net Issue Tracker Starter Kit-数据库
自动提取网页的信息 ,并分析之
涤生 · 2006-09-04 · via 博客园 - 涤生

本文是参照摩诘的Blog
今天遇到这样一个问题,从政府网站中,根据一个关键数据KeyData,提取相关数据。
这个问题可分为三部分解决:
1)取得政府网站交互的方法;
2)按照合适的方法,用HttpWebResponse,取得相关数据
3)分析取回来的数据

第一部分:获取网站交互信息,采用工具ieHTTPHeadersSetup.exe
得到的数据如下:
GET /search.asp?key=2006002995&ys_type=hy&imageField2.x=32&imageField2.y=20 HTTP/1.1
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */*
Accept-Language: zh-cn
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)
Host: www.suzhou-logistics.com
Connection: Keep-Alive

可以看出,
url: http://www.suzhou-logistics.com/search.asp?
Data:key=2006002995&ys_type=hy&imageField2.x=32&imageField2.y=20

也可以直接作为url:http://www.suzhou-logistics.com/search.asp?key=2006002995&ys_type=hy&imageField2.x=32&imageField2.y=20

第二部分:根据第一部分的分析,通过HttpWebResponse取HTML
在此就给出一个通用的函数

public static string GetPage(string url, string postData,string encodeType,out string err) 

        


            Stream outstream 
= null

            Stream instream 
= null

            StreamReader sr 
= null

            HttpWebResponse response 
= null

            HttpWebRequest request 
= null

            Encoding encoding 
= Encoding.GetEncoding(encodeType); 

            
byte[] data = encoding.GetBytes(postData); 

            
// 准备请求 

            
try 

            
{    

                
// 设置参数 

                request 
= WebRequest.Create(url) as HttpWebRequest; 

                CookieContainer cookieContainer 
= new CookieContainer(); 

                request.CookieContainer 
= cookieContainer; 

                request.AllowAutoRedirect 
= true

                request.Method 
= "POST"

                request.ContentType 
= "application/x-www-form-urlencoded"

                request.ContentLength 
= data.Length; 

                outstream 
= request.GetRequestStream(); 

                outstream.Write(data,
0,data.Length); 

                outstream.Close(); 

                
//发送请求并获取相应回应数据 

                response 
= request.GetResponse() as HttpWebResponse; 

                
//直到request.GetResponse()程序才开始向目标网页发送Post请求 

                instream 
= response.GetResponseStream(); 

                sr 
= new StreamReader( instream, encoding ); 

                
//返回结果网页(html)代码 

                
string content = sr.ReadToEnd(); 

                err 
= string.Empty; 

                
return content; 

            }
 

            
catch(Exception ex) 

            


                err 
= ex.Message; 

                
return string.Empty; 

            }
 

        }
 

第三部分:分析Html数据,有两个开源软件
SgmlReader与HtmlAgilityPack20,由于本人机器上只有vs2003,无法使用vs2005版本HtmlAgilityPack20。所以下面用SgmlReader来分析。SgmlReader可以将Html解析成格式完整的类似XML数据,可以采用Xpath进行查询,获取我们想要的数据。
取得完整的xml数据后的分析,根据post页面数据格式的不同而有区别。我取的这个页面,主要用了两个DataTable,一个保存一行基本数据,另一个保存多行的状态数据。

public static DataSet ParsePage(string pageContent, string xclpath,string xrpath,out string err)
        
{

            err 
= string.Empty;
            DataSet ds 
= new DataSet();
            DataTable table 
= new DataTable("QueryResult1");
            DataTable table1 
= new DataTable("QueryResult2");

            

            
            StringWriter strWriter 
= null;
            SgmlReader sgmlReader 
= null;
            XmlTextWriter xmlWriter 
= null;

            
try 
            
{
                sgmlReader 
= new SgmlReader();
                sgmlReader.DocType 
= "HTML";
                sgmlReader.InputStream 
= new StringReader(pageContent);
                strWriter 
= new StringWriter();
                xmlWriter 
= new XmlTextWriter(strWriter);
                xmlWriter.Formatting 
= Formatting.Indented;

                sgmlReader.Read();
                
while (!sgmlReader.EOF) 
                
{                    
                    xmlWriter.WriteNode(sgmlReader, 
true);
                    
                }
 
                xmlWriter.Flush();
                xmlWriter.Close();

                
string wellFormedHTML = strWriter.ToString();


                
if(xclpath.Trim().Length == 0)
                    
return ds;
                
                
                XPathDocument doc 
= new XPathDocument(new StringReader(wellFormedHTML));
                XPathNavigator nav 
= doc.CreateNavigator();
                XPathNodeIterator nodes 
= nav.Select(xclpath);

                
int i = 0;
                
while (nodes.MoveNext()) 
                
{
                    
                    
string sNodeText = nodes.Current.Value;
                    
if( i < nodes.Count - 5)
                    
{
                        
if( i< 17)
                        
{
                            
if(table.Columns.Contains(sNodeText))
                            
{
                                sNodeText 
= sNodeText + i.ToString();
                            }

                            table.Columns.Add(sNodeText ,
typeof(string));
                        }

                        
                    }

                    
else
                    
{
                        
if(table1.Columns.Contains(sNodeText))
                        
{
                            sNodeText 
= sNodeText + i.ToString();
                        }

                        table1.Columns.Add(sNodeText ,
typeof(string));
                    }

                    i 
++;
                    
                }


                ds.Tables.Add(table);
                ds.Tables.Add(table1);



                
bool bNext = false;
                nodes 
= nav.Select(xrpath);

                DataRow row 
= table.NewRow();
                table.Rows.Add(row);

                DataRow row1 
= null;                
                
int j = 0;
                
int k = 0;
                
while (nodes.MoveNext()) 
                
{
                    
string nodetext = nodes.Current.Value;

                    
if(table.Columns.Contains(nodetext) || table1.Columns.Contains(nodetext))
                    
{
                        
continue;
                    }


                    
if(!bNext)
                    
{
                        
if ( nodetext == "正在预录入")
                        
{
                            bNext 
= true;
                        }

                    }

                    
if(!bNext)
                    
{                        
                        
if( j < 17)
                        
{
                            row[j] 
= nodetext;
                            j
++;
                        }

                    }

                    
else
                    
{
                        
                        
if( k == 0)
                        
{
                            row1 
= table1.NewRow();
                            table1.Rows.Add(row1);
                        }


                        row1[k] 
= nodetext;
                        k 
= (k + 1% 5;

                    }
    
                }


                err 
= string.Empty;

                
return ds;
            }
 
            
catch (Exception exp) 
            
{
                
                err 
= exp.Message;
                
return ds;
            }


        }

有了上面的代码就可以采用如下方法调用了

private void Button1_Click(object sender, System.EventArgs e)
        
{    

            
string sHtml = string.Empty;
            
string sErr = string.Empty;
            
string sUrl = @"http://www.suzhou-logistics.com/search.asp?key=2006002995&ys_type=hy&imageField2.x=32&imageField2.y=20";
            DataSet ds;

            sHtml 
= WebForm1.GetPage(sUrl , string.Empty, "GB18030",out sErr);

            
if(sErr == string.Empty)
            
{
                
string xcpath = @"/html/table/tr/td/table/tr/td/font/strong";
                
string xrpath = @"/html/table/tr/td/table/tr/td";
                ds 
= WebForm1.ParsePage(sHtml, xcpath, xrpath,out sErr);

                
if((sErr == string.Empty) && (ds.Tables.Count == 2))
                
{
                    
if(ds.Tables[0].Rows.Count > 0)
                    
{
                        DataGrid1.DataSource 
= ds.Tables[0];
                        DataGrid1.DataBind();
                    }


                    
if( ds.Tables[1].Rows.Count > 0)
                    
{
                        DataGrid2.DataSource 
= ds.Tables[1];
                        DataGrid2.DataBind();
                    }

                }

            }


            
        }

其实SgmlReader可以直接完成从URl抓取数据的功能,即将第二部分与第三部分合并。

string SgmlReaderTest(Uri baseUri, string url, TextWriter log, bool upper, bool formatted)
        
{
            
string inputUri = url;
            
            
try 
            
{
                SgmlReader r 
= new SgmlReader();
                r.SetBaseUri(Server.MapPath(
"."));
                r.DocType 
= "HTML";
                r.Href 
= url;
                
if (upper) r.CaseFolding = CaseFolding.ToUpper;
                StringWriter sw 
= new StringWriter();
                XmlTextWriter w 
= new XmlTextWriter(sw);

                
if (formatted) 
                
{
                    w.Formatting 
= Formatting.Indented;
                    r.WhitespaceHandling 
= WhitespaceHandling.None;
                }

                r.Read();
                
while (!r.EOF) 
                
{
                    
                    w.WriteNode(r, 
true);
                }

                w.Flush();
                w.Close();
                
return sw.ToString();
            }
 
            
catch (Exception e) 
            
{
                
return e.ToString();
            }

        }