Web scraping nightmare — Can you scrape these websites?

# -*- coding: utf-8 -*-
from bs4 import BeautifulSoup
import requests
headers = {'User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.106 Safari/537.36'}
url = 'https://pixabay.com/images/search/crazy/'
headers = {
'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36',
'referrer': 'https://google.com',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
'Accept-Encoding': 'gzip, deflate, br',
'Accept-Language': 'en-US,en;q=0.9',
'Pragma': 'no-cache',
cookies = {
'is_human': '1',
'__cfduid' : 'dd0dbf52b454e1cbbc4f46fecc72f997f1581756368',
'anonymous_user_id' : 'e9a26f0a-d5f4-4e21-9af0-8c0dc1ddd441',
curl "http://api.proxiesapi.com/?key=API_KEY&render=true&url=https://example.com"




Founder @ ProxiesAPI.com

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Scaling up containers with Dynamic Local Cache

College Decider: Your College Application Companion

January 2022 Newsletter

Some interesting git commands

Convert Php/Symfony web app into a Windows Desktop App

Get familiar with layout, concern, and helper in rails

How to create a static HTML web app by using Azure Cloud Shell

Learn python with me

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Mohan Ganesan

Mohan Ganesan

Founder @ ProxiesAPI.com

More from Medium

What Is a Credible Source? The Facts & Fat of the Matter

How to calculate ROI using predictive or real LTV

5 Crystals for the Lunar New Year — Year of the Tiger

5 Crystals for the Lunar New Year

Turn me into a cyborg