Web Mining（网路探勘）.ppt

* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * Concurrency A crawler incurs several delays: Resolving the host name in the URL to an IP address using DNS Connecting a socket to the server and sending the request Receiving the requested page in response Solution: Overlap the above delays by fetching many pages concurrently Source: Bing Liu (2011) , Web Data Mining: Exploring Hyperlinks, Contents, and Usage Data * Architecture of a concurrent crawler Source: Bing Liu (2011) , Web Data Mining: Exploring Hyperlinks, Contents, and Usage Data * Concurrent crawlers Can use multi-processing or multi-threading Each process or thread works like a sequential crawler, except they share data structures: frontier and repository Shared data structures must be synchronized (locked for concurrent writes) Speedup of factor of 5-10 are easy this way Source: Bing Liu (2011) , Web Data Mining: Exploring Hyperlinks, Contents, and Usage Data * Universal crawlers Support universal search engines Large-scale Huge cost (network bandwidth) of crawl is amortized over many queries from users Incremental updates to existing index and other data repositories Source: Bing Liu (2011) , Web Data Mining: Exploring Hyperlinks, Contents, and Usage Data * Large-scale universal crawlers Two major issues: Performance Need to scale up to billions of pages Policy Need to trade-off coverage, freshness, and bias (e.g. toward “important” pages) Source: Bing Liu (2011) , Web Data Mining: Exploring Hyperlinks, Contents, and Usage Data * Large-scale crawlers: scalability Need to minimize overhead of DNS lookups Need to optimize utilization of network bandwidth and disk throughput (I/O is bottleneck) Use asynchronous sockets Multi-processing or multi-threading do not scale up to billions of pages Non-blocking: hundreds of network connections open simultaneously Polling socket to monitor completion of network transfers Source: Bing Liu (2011) , Web Data Mining: Exploring Hyperlinks, Con