How We Validate Macro Data Accuracy Before Serving It

When a central bank releases a policy rate decision or a statistics bureau publishes a new inflation print, the raw announcement often arrives with noise: encoding artifacts, partial pages, missing fields, or revisions that contradict the prior month. Serving that data naively — as many aggregators do — passes the problem straight to the developer.

At FXMacroData we treat data quality as a first-class product concern. Every indicator from every currency goes through a deterministic validation pipeline before it is written to Firestore and exposed through the API. This post walks through that pipeline layer by layer — from the moment a fetcher downloads a raw response to the moment a value becomes queryable at endpoints like /v1/announcements/{currency}/{indicator}.

Stage 1 — Ingest: Structured Source Fetchers

Validation starts before a single value is extracted. Every currency has a dedicated fetcher class that targets an official primary source — the central bank website, the national statistics bureau, or a government data portal. We deliberately avoid secondary aggregators in the ingest path: their lag, licensing terms, and occasional silent revisions introduce uncertainty we cannot control.

Fetchers are asynchronous Python classes implementing an async with context manager pattern. On entry they open an aiohttp.ClientSession with a realistic User-Agent and controlled timeouts; on exit they close cleanly regardless of whether the fetch succeeded or raised. Inside each fetcher, parsing is strict: HTML is parsed with lxml or BeautifulSoup using exact element selectors rather than regex fallbacks against raw markup, and JSON APIs are accessed via typed accessor keys that raise immediately if a field is absent or renamed upstream.

{
    "date": "2026-03-31",          # ISO-8601 date string
    "val": 3.5,                    # float — never string
    "announcement_datetime": "..." # UTC ISO-8601 if available
}

The output contract is enforced at the fetcher boundary: any record missing date or val is discarded before it reaches the next stage. announcement_datetime is optional at ingest but required for publication-facing endpoints that expose event timing to API consumers.

Stage 2 — Schema Check: Type and Completeness Validation

Raw fetcher output is handed to a schema validator that applies four checks on every record:

Schema failures are surfaced as structured Cloud Logging entries tagged with severity=ERROR, stage=schema_check, and the originating fetcher name. This makes cross-run diffing straightforward in the GCP console.

Stage 3 — Range & Outlier Filter

Structural validity is necessary but not sufficient. A value of 250.0 for USD CPI is syntactically valid but obviously wrong. Stage 3 applies two complementary checks to catch these semantic errors.

Hard range bounds

Each indicator has a catalogue entry that includes optional min_val and max_val bounds derived from plausible historical ranges plus a generous safety margin. Policy rates, for example, are bounded between -5.0 and 30.0 percent. Year-over-year inflation is bounded between -30.0 and 300.0 percent — wide enough to accommodate hyperinflationary episodes without constraining legitimate emerging-market data. Values outside these bounds are quarantined pending manual review.

Rolling z-score outlier detection

For indicators with at least 24 months of history in Firestore, the pipeline computes a 36-month rolling mean and standard deviation and flags any new record whose z-score exceeds |4.0|. Unlike hard bounds, z-score flags do not automatically discard records — they create a review entry and attach an outlier_flag: true field to the Firestore document so that API consumers can optionally filter outlier-flagged records in their own workflows.

Stage 4 — Cross-Source Reconciliation

For a subset of high-importance indicators — central bank policy rates, headline CPI, and unemployment — the pipeline maintains a secondary source to cross-reference against. This is not a live fallback at request time (all data served to users comes exclusively from Firestore); it is an ingest-time consistency check.

When the primary and secondary values for the same (currency, indicator, date) diverge by more than a configurable tolerance, an alert is raised and the primary value is held pending investigation. For policy rates the tolerance is 5 basis points; for CPI it is 0.1 percentage points. The tolerances are intentionally narrow for these indicators because even small discrepancies often indicate a parsing error, a reporting lag, or a preliminary-vs-final revision conflict.

Beyond per-record cross-checks, the pipeline also runs a month-over-month continuity check: if a new record represents a change of more than N standard deviations from the trailing 12-month average change, it is treated as a candidate revision conflict. Preliminary releases frequently differ from final revisions; the pipeline logs both values and exposes a revised flag when a date's value is updated after initial publication.

Stage 5 — Business-Day Integrity

The final validation stage addresses a subtle but important constraint: every announcement_datetime must fall on a valid business day in the market timezone of the currency being published. Statistical bureaus and central banks do not publish announcements on weekends or public holidays — so if the pipeline produces a timestamp that lands on a Saturday in Tokyo or a bank holiday in Sydney, something went wrong upstream.

The validator calls is_valid_announcement_date(currency, local_date), which checks the date against per-currency timezone definitions and complete holiday calendars maintained in the codebase. Every currency served by the API — AUD, EUR, GBP, JPY, USD, CAD, CHF, NZD, and all others — has its own independent timezone and holiday table. Currencies do not inherit from their FX session; a Friday in New York can be Saturday in Sydney, and the validator handles this precisely.

def is_valid_announcement_date(currency: str, local_date: date) -> bool:
    tz = CURRENCY_TIMEZONE[currency]
    # Reject weekends
    if local_date.weekday() >= 5:
        return False
    # Reject public holidays
    if local_date in _build_holiday_set(currency, local_date.year):
        return False
    return True

When a computed date fails this check, next_valid_announcement_date advances it to the next business day — for example, rolling a Christmas announcement to the following Monday. This ensures the release calendar endpoint served to API consumers always contains dates that can be used directly in trading calendars without manual cleaning. These business-day rules are also enforced by a CI test suite that fails the build if any currency in the catalogue is missing timezone or holiday data.

Continuous Monitoring and Alerting

Passing the validation pipeline once is not enough. The pipeline runs on a schedule — triggered by Cloud Tasks and backfill workflows — and each run produces structured telemetry that feeds a monitoring layer.

When a fetcher fails to return data — network timeout, upstream site change, or response structure change — the pipeline does not fall back to live upstream calls at request time. Instead it emits a validation failure and returns an empty result or a structured DataUnavailableError to the caller. This prevents stale or partially-validated data from reaching the API layer, even temporarily.

Handling Revisions and Restatements

Macro data revisions are a fact of life. Initial GDP estimates are revised two and three times. Payrolls get significantly restated. The pipeline handles revisions explicitly rather than silently overwriting:

• First-print storage: The pipeline stores the first value for a given (currency, indicator, date) with a revised: false flag.
• Revision detection: On subsequent ingest runs, if the value for a date has changed by more than the indicator's revision threshold, the document is updated and revised: true is set.
• History preservation: The original first-print value is preserved in a prior_val field for auditing and comparison purposes.
• API transparency: The revised field is exposed in API responses so consuming applications can distinguish preliminary from final readings.

This matters most for indicators like Non-Farm Payrolls, where the preliminary print and the subsequent revision can differ by tens of thousands of jobs — a meaningful signal in its own right for FX traders tracking the USD employment narrative via the non-farm payrolls endpoint.

What This Means for API Consumers

The practical outcome of this pipeline for anyone querying the API:

Query any indicator — policy rate, headline CPI, unemployment — and the response you receive has already passed all five stages. The indicator catalogue documents exactly which sources feed each series, so you can verify the provenance of any data point independently.