Web scraping means reading a page with code instead of your eyes. In Python it is two libraries and about ten lines: requests to fetch the HTML and BeautifulSoup to pull values out of it. The harder parts are doing it politely, handling pages that change, and knowing when scraping is the wrong tool.
Before you write anything#
Three checks, in order:
- Is there an API? Many sites publish one. It returns structured data, will not break when the design changes, and is explicitly permitted. Always prefer it.
- What does robots.txt say? Visit the site’s
/robots.txt. It lists paths that automated clients are asked not to fetch. - What do the terms of service say? Some sites prohibit automated access outright. Respect that.
Python can check robots.txt for you:
from urllib.robotparser import RobotFileParser
rp = RobotFileParser()
rp.set_url("https://example.com/robots.txt")
rp.read()
print(rp.can_fetch("*", "https://example.com/products"))
Installing the tools#
python -m venv .venv
source .venv/bin/activate # .venv\Scripts\activate on Windows
pip install requests beautifulsoup4
Fetching a page#
import requests
headers = {"User-Agent": "LearningBot/1.0 (contact: you@example.com)"}
response = requests.get("https://example.com", headers=headers, timeout=10)
response.raise_for_status() # raises on 404, 500 and friends
print(response.status_code)
print(len(response.text), "characters")
Two habits worth adopting immediately. Always set a timeout — without one, a slow server hangs your script indefinitely. Always call raise_for_status(), or you will happily parse a 404 error page as if it were data.
Setting an honest User-Agent that identifies your script is good manners and makes it easy for a site owner to contact you rather than simply block you.
Parsing the HTML#
from bs4 import BeautifulSoup
soup = BeautifulSoup(response.text, "html.parser")
print(soup.title.string)
# One element
heading = soup.find("h1")
print(heading.get_text(strip=True))
# Every match
for link in soup.find_all("a"):
print(link.get("href"), "-", link.get_text(strip=True))
CSS selectors are usually more precise than find arguments:
soup.select_one("h1.product-title")
soup.select("div.product .price")
soup.select("table#results tr td:nth-child(2)")
soup.select("a[href^='https://']")
Finding the right selector#
Open the page in your browser, right-click the value you want and choose Inspect. Look at the element and its parents for a class or id that is specific to what you want and stable across pages.
Prefer selectors that describe meaning over ones that describe position:
# Fragile - breaks if a div is added anywhere above
soup.select("body > div:nth-child(3) > div > span")
# Sturdier - describes what the thing is
soup.select_one("span.price")
soup.select_one("[data-testid='price']")
A complete example#
import csv
import time
import requests
from bs4 import BeautifulSoup
HEADERS = {"User-Agent": "LearningBot/1.0 (contact: you@example.com)"}
def fetch(url):
response = requests.get(url, headers=HEADERS, timeout=10)
response.raise_for_status()
return BeautifulSoup(response.text, "html.parser")
def parse_quotes(soup):
rows = []
for quote in soup.select("div.quote"):
text = quote.select_one("span.text")
author = quote.select_one("small.author")
if text and author:
rows.append({
"text": text.get_text(strip=True),
"author": author.get_text(strip=True),
})
return rows
def scrape(base, pages=3):
all_rows = []
for page in range(1, pages + 1):
try:
soup = fetch(f"{base}/page/{page}/")
except requests.RequestException as error:
print("skipping page", page, "-", error)
continue
rows = parse_quotes(soup)
if not rows:
break # ran out of pages
all_rows.extend(rows)
time.sleep(1) # be polite
return all_rows
rows = scrape("https://quotes.toscrape.com")
with open("quotes.csv", "w", newline="", encoding="utf-8") as f:
writer = csv.DictWriter(f, fieldnames=["text", "author"])
writer.writeheader()
writer.writerows(rows)
print(len(rows), "rows written")
Note the shape: fetching, parsing and saving are three separate functions. That means you can test the parser on a saved HTML file without making a single request.
Being a good citizen#
- Sleep between requests. One second is a reasonable floor. Your convenience is not worth degrading someone’s server.
- Cache while developing. Save the HTML to a file the first time and parse from disk while you get the selectors right. You will run the parser fifty times; the site only needs to serve it once.
- Fetch only what you need. Do not crawl an entire site to get one table.
- Back off on errors. If you get a 429 or 503, wait longer, do not retry immediately in a loop.
import pathlib, hashlib
def cached_fetch(url, folder="cache"):
pathlib.Path(folder).mkdir(exist_ok=True)
key = hashlib.sha256(url.encode()).hexdigest()[:16]
path = pathlib.Path(folder) / (key + ".html")
if path.exists():
return path.read_text(encoding="utf-8")
response = requests.get(url, headers=HEADERS, timeout=10)
response.raise_for_status()
path.write_text(response.text, encoding="utf-8")
time.sleep(1)
return response.text
When the page is built by JavaScript#
If response.text does not contain the data you can see in the browser, the page is rendering it with JavaScript after loading. requests only fetches the initial HTML and does not run scripts.
Two options, in order of preference:
- Find the underlying request. Open the browser’s Network tab, filter to XHR or Fetch, and reload. Very often the page is calling a JSON endpoint you can call directly — simpler and far faster than rendering the page.
- Drive a real browser. Playwright or Selenium load the page properly, scripts and all. This is much slower and heavier, so treat it as the fallback.
# Often this is all you need
data = requests.get("https://example.com/api/products?page=1", timeout=10).json()
Questions people ask#
Is web scraping legal?
It depends on the site, the data and where you are. Public factual data is generally treated differently from personal data or copyrighted content, and terms of service matter. Check robots.txt and the terms, avoid personal data, and if the project is commercial, take proper advice.
Why does my scraper work once and then get blocked?
Usually rate. Requests arriving faster than a human could click look automated, and many sites throttle or block on that. Add delays, cache, and identify yourself honestly.
Should I use Scrapy instead?
Scrapy is worth it for large crawls — it handles queuing, retries, concurrency and pipelines. For a handful of pages, requests plus BeautifulSoup is less to learn and less to configure.
How do I handle a login?
Use a requests.Session() so cookies persist across requests. Be aware that logging in usually means agreeing to terms that restrict automated access, so check first.
Where to go next#
- Python file handling — for saving what you scrape.
- Python try/except explained — the error handling this needs.
- What is an API? — the thing to look for before scraping at all.
