This workshop introduces the fundamentals of scraping user-generated content from the web using R or Python, taught over one 2-hour practical session. Prior to the practical session, pre-reading has been assigned and will be shared with you ahead of time to best prepare for the class. Pre-reading material focuses on the legal and ethical dimensions of web scraping, including questions around terms of service, copyright, privacy, consent, and the boundary between public and private data. The pre-reading material will further cover controversial research cases, consider why they were problematic, and reflect on what responsible web scraping might look like in practice. Alongside these important reflections, participants will receive a short introduction to HTML structures and the tools available for scraping in R and Python. The practical class session will be hands-on, working through the process of scraping a selected static website step by step, before concluding with a discussion of the limitations of scraping and the challenges posed by more complex dynamic sites.
Participants should have a basic familiarity with R or Python.
You should be comfortable installing and loading packages, working with data frames, and running simple code chunks.
For those working with R, some prior exposure to the Tidyverse syntax (e.g. using the pipe %>% and functions like mutate() or filter()) will make the hands-on parts easier to follow, though it is not strictly required.
No prior experience with web scraping or HTML is assumed.
- Make informed decisions about when web scraping is appropriate, and when alternative approaches (e.g., APIs, open data sources) should be used.
- Explain the main legal and ethical considerations in web scraping, including terms of service, copyright, privacy, and consent.
- Identify and interpret the HTML structure of a static web page.
- Use R or Python tools to scrape and organise content from a simple static website.
By attending this workshop, you will become familiar with the following skills:
- Understanding and working with basic HTML structures.
- Using the R package rvest or the Python library Beautiful Soup to scrape static websites.
