Data Validation

TLDR: Data validation checks collected records against expected rules for type, format, range, and completeness, so broken data is caught before it reaches anything that depends on it.

Data validation is the quality gate between collection and use. In web data pipelines it matters more than in most systems, because the source is outside your control and can change without warning. A site redesign does not produce an error. It produces records that look structurally fine and contain nothing useful.

Levels of Validation:

1. Field level: Does each value have the right type, format, and range? A price should be a positive number, a currency should be a valid ISO code.
2. Record level: Are the fields consistent with each other? A sale price above the list price is individually valid and jointly wrong.
3. Batch level: Does this run resemble previous runs? A sudden drop in record count or a spike in nulls indicates a problem regardless of whether individual records pass.
4. Cross source: Do independent sources agree? A product priced at 40 on four sites and 4 on a fifth suggests a parsing error, not a bargain.

Common Validation Rules:

Implementing Field Validation:

from datetime import datetime, timezone

VALID_CURRENCIES = {"USD", "EUR", "GBP", "JPY", "ILS"}

def validate(record):
    errors = []

    title = record.get("title")
    if not title or not title.strip():
        errors.append("title_empty")

    price = record.get("price")
    if price is None:
        errors.append("price_missing")
    elif not isinstance(price, (int, float)):
        errors.append("price_not_numeric")
    elif price <= 0 or price > 1000000:
        errors.append("price_out_of_range")

    currency = record.get("currency")
    if currency not in VALID_CURRENCIES:
        errors.append("currency_invalid")

    url = record.get("url", "")
    if not url.startswith(("http://", "https://")):
        errors.append("url_not_absolute")

    ts = record.get("scraped_at")
    if ts and ts > datetime.now(timezone.utc):
        errors.append("timestamp_future")

    return errors

Collecting all errors rather than returning on the first one matters. Knowing a record failed on four rules is far more diagnostic than knowing it failed on one, and it avoids the fix one, rerun, discover the next cycle.

Batch Validation Catches What Record Validation Misses:

The most damaging failures are the ones where every individual record is valid.

If a selector breaks and returns null for price on every page, each record passes a schema that allows null prices. Nothing raises an error. The dataset is simply empty in the column that mattered, and the problem surfaces days later in a report.

Batch checks catch this by comparing the run against history rather than against rules. Useful signals include record count compared with the previous run, null rate per field, distribution of values, and the proportion of records that changed. A field going from 2 percent null to 97 percent null is unambiguous, and no per record rule would have flagged it.

Deciding What to Do With Failures:

Validation is only useful if failure triggers a defined action, and the right action varies by severity.

Reject the record. Appropriate when a required field is unusable. The record is quarantined rather than deleted, so it can be inspected and reprocessed.

Accept with a flag. Appropriate when an optional field fails. The record is stored with a quality marker so downstream consumers can filter if they care.

Repair. Appropriate for deterministic fixes such as trimming whitespace or normalizing casing. Repair should never guess at missing values.

Halt the batch. Appropriate when batch level checks fail. Publishing a run where 97 percent of prices are null is worse than publishing nothing.

Validation and Trust:

A dataset’s value depends on whether its consumers believe it. One pricing report built on a broken parse does more lasting damage than a day of missing data, because it changes how every subsequent report is received.

This is why validation belongs inside the pipeline rather than in the analysis layer. By the time an analyst notices something looks wrong, the bad data has already been queried, charted, and quoted.

Bright Data’s Datasets and Scraper APIs apply validation before delivery, so records arrive already checked for completeness and type consistency rather than requiring a verification layer on receipt.