Definition:
Scraping is the automated extraction of data from another system’s output or interface. When the source is a website or web application, it is called web scraping: a program retrieves content, locates relevant elements, and transforms their values into data that can be stored or analyzed.
The process may retrieve a few pages or operate at a larger scale, but it does not necessarily collect millions of records. Nor does it inherently mean crawling an entire site, bypassing its controls, or freely reusing the information. Its scope depends on the source consulted, the data extracted, and the purpose.
How web scraping works
The process begins with a request to the source. A scraper may download HTML over HTTP, use an automated browser when JavaScript generates the content, or query a structured response. The technique used depends on how the system delivers its data.
A parser then interprets the document and makes it possible to select particular fields. CSS selectors, regular expressions, or XPath may locate titles, prices, links, tables, or other elements within the structure.
Extracted values are commonly cleaned, normalized, and stored in a table, database, or file. This step may correct date formats, currencies, whitespace, duplicates, and characters. Subsequent preparation is necessary because visible text does not always represent a complete or comparable value.
On dynamic sites, some information may arrive through internal requests or appear after an interaction. This dynamic content may require an automated browser. When a documented and suitable API exists, it provides a more stable and structured interface, although its availability, permissions, coverage, and limits may differ from what the website displays.
Applications of scraping
Automated extraction can support observation, integration, and research. Its common applications include:
- Prices and catalogues: Collect prices, availability, features, or references to compare changes among sources and points in time.
- Market research: Organize public information about products, companies, job listings, or activity within a sector.
- Content monitoring: Detect changes in pages, notices, publications, regulations, or accessible datasets.
- Integration and migration: Retrieve owned or authorized content when no adequate structured export is available.
- Research and analysis: Build a set of observations for later study while documenting the source, date, and method.
Scraping obtains data but does not guarantee that it is correct, complete, or representative. A page may display outdated, personalized, or location-specific information. Conclusions need to account for the sample’s actual coverage and the rules under which it was generated.
Differences from other processes
A crawler follows links to discover or update resources, whereas a scraper focuses on extracting fields from retrieved sources. The processes can be combined, but crawling determines what to visit and extraction determines the data to collect.
An API defines operations and formats for exchanging information between systems. Scraping works on a representation primarily created for another purpose, such as a visible page. An API therefore commonly provides greater contractual stability, while a scraper depends on an external structure that may change without notice.
Web mining uses content, structure, or usage data to find patterns and produce knowledge. Scraping may provide some of that data, but it does not perform the analysis by itself. Data mining likewise covers sources and methods that are not limited to the web.
A parser, CSS selector, or XPath is a technical component, not a synonym for the complete process. Extraction may also be manual or semi-automated. Scraping describes structured retrieval, not a particular tool, programming language, or format.
Legal and privacy conditions
Data being visible online does not mean that it is unprotected or available for every form of reuse. Access terms, copyright or database rights, trade secrets, personal data protection, competition rules, and other laws may apply. Assessment depends on the particular case and jurisdiction.
When personal data is collected, the legal basis, purpose, minimization, retention, security, and individual rights require consideration. The ICO statement on data scraping notes that publicly accessible personal information remains subject to privacy and data protection laws.
Terms of service and licences may establish permitted uses or contractual restrictions. Authentication, access controls, and technical limits also need to be respected. Circumventing a barrier or accessing information not intended for the public creates a different context from retrieving open pages.
A robots.txt file communicates crawling rules to compatible agents, but it is not an authorization or security mechanism. The Robots Exclusion Protocol standard states that these rules are not a substitute for access controls. Allowing or excluding a path does not by itself settle whether processing is lawful.
Quality, protection, and limitations
A scraper’s reliability depends on both the source and its maintenance. Common quality controls include:
- Validate selectors: Check that each field still points to the correct element after design or template changes.
- Control frequency: Adjust requests, concurrency, and retries to avoid overloading the source or duplicating results.
- Record provenance: Retain the URL, date, version, and method so that the resulting dataset can be interpreted or reproduced.
- Detect anomalies: Identify missing values, blocks, error pages, personalized content, and format changes.
- Limit data: Collect only necessary fields and apply the relevant access, retention, and deletion rules.
Sites may use rate limits, authentication, automation detection, or a CAPTCHA. These measures reduce certain access but do not provide absolute protection and may also affect legitimate users, search engines, or assistive technologies. Defence requires proportionate layers.
Scraping is sensitive to HTML changes, dynamic content, geolocation, A/B tests, and personalization. Its results describe what the system could retrieve under specific conditions. They do not prove that the entire site was observed or that the data represents a whole population.