Mohan Ganesan

One of the most significant applications of Web Scraping in retail and e-commerce is in monitoring competitor price movements. This, when done well, can mean extra revenue and also will allow the retailer to ensure that they are always in the game and are not taken by surprise by anything…

--

--

Are you getting IP blocked repeatedly when web scraping at scale?

Is this happening even after passing it through a couple of free proxy servers?

It might be because the target website’s algorithm might be picking up on who you are by the User-Agent-String signature that your curl request or any other library you might be using is sending. Even the absence of it is telling.

But if you do it well, you do get a lease of life, albeit only for a while.

--

--

One of the biggest applications of Web Scraping is in scraping hotel listings from various sites. This could be to monitor prices, create an aggregator, or provide better UX on top of existing hotel booking websites.

Here is a simple script that does that. We will use BeautifulSoup to help…

--

--

Here is a list of places that your web crawler will probably fail at. You will need to build in checks for each and also expect them to happen. Send yourself alerts by having portions of the scripts check for unexpected behavior.

--

--