Crawling The Web For Fun And Profit
|
|
|
||||||||||||
Description: With over a couple of billion web pages on the Internet, it is but tempting to see how one can mine much of this information for fun or for profit. In this video, i run you through how to program a web crawler which will fetch pages and parse their content, so it can be converted into a useful format.
The web crawler which we create in this tutorial, consists of mainly 2 parts:
- Document fetching engine : This fetches the raw HTML page data from a website
- Document parsing engine : This uses an HTML DOM Parser to parse the page and derive useful input from it.
We shall use the Python language and the BeautifulSoup DOM parser to pull this off. The video is very interactive and i use a "type as you go" methodology to help you understand the programming techniques.
The code for this tutorial is available for download.