Data Quality & Monitoring

Abstract

Every data vendor says their data is good. This page documents how FlashAlpha verifies it: a six-layer architecture in which chains are reconciled against an independent second source contract by contract several times a day, serving endpoints refuse with machine-readable errors rather than degrade, pipelines across the fleet report into a central health registry, the historical archive is never silently restated, and responses carry data_as_of - the delivery timestamp of each upstream feed behind it - so freshness is something you read, not something you take on trust. The design principle throughout is fail closed and say why: scheduled monitors alert operators when a check fails, and an endpoint that cannot answer well refuses with a named reason rather than answering anyway.

In the spirit of the methodology document, this page describes capabilities and guarantees at the level a due-diligence reviewer needs. Parameter-level detail - tolerances, schedules, source pairings and escalation practice - is available to enterprise and systematic customers under NDA via /institutional. The narrative version, with worked code, is the article Why You Can Trust FlashAlpha Data.

The six layers

Layer 1 - Ingest integrity

Market data arrives from exchange-grade feeds with instrument mapping handled before anything is stored, so a contract's identity is stable through symbol reuse and rolls. Each feed - equity and index options, futures and futures options, spot - is tracked on its own liveness row for record throughput and session health, evaluated every minute, so a feed that stops delivering is visible on its own row rather than hidden behind a live process. The daily OPRA open-interest publication is confirmed before every equity open, and CME futures open interest is reconciled on the late-morning pass once the exchange publication lands. Open-interest snapshots are written idempotently - re-running a snapshot cannot double-count - expired contracts are evicted on schedule so chains do not carry dead weight, and the listed universe of more than 6,000 option roots is refreshed against the exchange listings so new roots are added and delisted ones retired. Where a feed's own semantics create known edge cases (the daily OI publication burst, session reconnects), the ingest layer is built around those semantics.

Layer 2 - Cross-source reconciliation

The core discipline: check the data against sources that have no reason to agree, and do it several times every trading day.

• Chain reconciliation. Option chains are compared contract by contract against an independent second source - coverage, open interest and price agreement scored per symbol. The sweep of 2026-09-14 compared 70,858 contracts across 44 symbols, equity, index and CME futures roots included, at 99.6% open-interest agreement on comparable contracts (how the figures are measured). A reference source that returns nothing comparable on consecutive passes raises its own alert, so the check cannot quietly stop checking.
• Parity re-derivation. Index spot is recovered from the option chain itself via put-call parity - a per-expiry regression of call-put differences across strikes with outlier trimming and a reliability gate - and compared with the vendor print. Equity index futures (ES, NQ, RTY) are anchored against their cash indexes on every pass: a future is a near-exact forward on the same underlying, so its basis must sit inside a configured band. A quote stream can drift; an arbitrage identity cannot.
• Session volume, two layers. The live classified trade tape is reconciled against an independent vendor's session totals on every pass, and inside the ingest every contract's volume accumulator is checked against its own tape as a conservation identity.
• Served-layer conservation. The open interest the exposure endpoints actually serve is summed per strike and compared with the truth source per strike, so the check covers what customers consume, not only what the feed delivered.
• Reference-data cross-checks. Earnings dates are verified across two independent providers daily, with drift surfaced for review.

How the published figures are measured

The same aggregates the strip shows are published live, without a key, at /v1/dq/public-summary. contractsCompared counts contracts with positive open interest on both sources; oiAgreementPct is the share of those within the configured relative-error tolerance, weighted by contract count across the latest result for each symbol and source pair; lastSweepUtc dates the newest result included. Coverage - a contract one side lists and the other does not - is scored separately, so it can never hide inside an agreement figure. Uptime comes from /v1/status, the same feed that drives the public status page.

Layer 3 - Serving guards: fail closed

Verification would be theatre if degraded data were served anyway. When a scheduled check fails, operators are alerted; when an endpoint cannot answer well, it refuses, with machine-readable faults a pipeline can branch on:

• A price series that is not fresh enough to model on is refused with HTTP 409 and a stale_data fault - including the date of the last good bar and its age in weekdays - rather than serving a frozen price:

HTTP/1.1 409 Conflict
{
  "error": "stale_data",
  "message": "The historical price archive's most recent bar is 2026-07-28 (5 trading days old). Data is stale; refusing to serve a frozen price or fit models on it. Re-hydrate the symbol's history.",
  "stale_since": "2026-07-28",
  "stale_trading_days": 5,
  "endpoint_version": "2026.09.11"
}

• One-sided, crossed and settled ghost quotes (a bid with no ask, an ask below the bid, a contract past its settlement cutoff) are excluded from IV solving, from the Greeks, and from the exposure aggregates built on them, rather than solved into a fictitious surface.
• A gamma flip is published as certified only after it passes local-coverage, expiry-day quote-quality and single-strike sensitivity checks; gamma_flip_status reports available or names the check, and the exposure regime is reported as unknown rather than derived from an uncertified level. The policy was set on 4,575 historical option books and 436,507 stress variants - see the gamma flip certification article.
• Analytics requests compute on request-owned market data, so the numbers inside one response are internally consistent and data_as_of reports the feed deliveries observed while that response was built.
• Same-day contracts are handled through their settlement semantics, so a settled expiry drops out of the live exposure profile.
• The strategy scorers return an explicit insufficient_data decision instead of a recommendation when a symbol's history coverage cannot support one.
• A fleet-wide availability control can withdraw serving for an asset class or a single symbol across the API fleet, with the standard no_data envelope - the platform prefers absence over wrongness at every scale.

Layer 4 - Fleet & job health

Pipelines across every host in the fleet - feed sessions, ingest jobs, nightly writers, reconciliation monitors, alert delivery and serving loops - report into a central job-health registry with status, last-run time, staleness and consecutive-failure tracking, and the API and ingest subset is published live at /v1/dq/public-summary. Around it sit dedicated monitors for the failure modes that matter most: per-feed liveness every minute; served index and equity spot checked for movement every five minutes during their session hours; the OPRA open-interest publication confirmed before every equity open; nightly and external processors tracked by heartbeat. Alerts are trading-calendar aware, and a digest goes to the operators twice each trading day whether or not anything is wrong. Independently of all of this, external probes exercise the platform from outside its own infrastructure, and availability is reported publicly at /status.

Layer 5 - Point-in-time integrity

Stored inputs are point-in-time and never silently restated: what replay sees at any minute is what a live consumer could have known at that minute, and any correction to the archive is documented in the changelog rather than applied invisibly under a backtest. Successful historical responses carry archive_as_of - the vintage of the archive rows actually replayed for the timestamp you asked for - so the provenance of a replay is as readable as the provenance of a live call. A look-ahead guard in the replay engine and a freshness guard on the archive back both properties. Analytics recompute through the same engine that serves live traffic, with live-historical parity pinned by test and sampled daily against production - the full computation model is documented at Historical Data & Computation.

Layer 6 - Provenance on every successful response

Successful JSON responses carry two fields, stamped by the response envelope in one place rather than endpoint by endpoint: endpoint_version, the version label of the deployed build that produced the answer, and data_as_of, the delivery timestamp of each upstream feed the response used. Feeds are stamped by physical pipe, not by asset class, because spot and options arrive over different connections and a single timestamp cannot describe both.

"endpoint_version": "2026.09.11",
"data_as_of": {
  "node": "fa4",
  "equity_feed":          "2026-09-14T15:14:22.543Z",
  "equity_options_feed":  "2026-09-14T15:14:21.935Z",
  "index_feed":           null,
  "index_options_feed":   null,
  "futures_feed":         null,
  "futures_options_feed": null,
  "flow_feed":            null,
  "oi_feed":              null,
  "macro_feed":           null
}

The example is a SPY gamma-exposure call captured at 15:14 UTC on 2026-09-14: the request read equity spot and equity options, so those two slots are stamped a second or two before the response was built, and the slots it did not read are null. Read each feed against its own cadence. Endpoints that return a top-level array carry the same object in the X-Data-As-Of and X-Endpoint-Version headers. The public /health, /livez and /readyz endpoints on lab.flashalpha.com return the node-level view of the same object with no API key, with every streaming feed the node consumes stamped, so freshness can be checked before a single data call is made. The Python SDK (flashalpha on PyPI) types it as DataAsOf. The full specification is in the API reference.

Alongside provenance, the 0DTE responses attach machine-readable quality signals a pipeline can gate on: a composite data-quality score built from valid-IV share, valid-Greek share and snapshot freshness; a Greek-smoothness score that measures how smoothly implied volatility moves across neighbouring strikes; and the snapshot age and contract count behind every aggregate. Every strategy decision carries its own data_quality object with a score and warnings. The exact score construction is published in methodology §7. If you would rather gate on our numbers than trust them, that is exactly the intended use.

Independent verification

An external monitor probes the platform from outside its own infrastructure every 30 seconds: the API over the public path a customer uses, with a real API key on a real analytics endpoint, so a response that is well-formed but stale or incoherent counts as a failure; the historical service; and the market-data ingest, including its per-feed liveness rows. On the probed API response the monitor reads data_as_of itself and treats a spot or options feed that falls behind inside its own session as a failure, and it checks through put-call parity that the served SPY option book prices the same spot the platform serves. Results drive the public status page, bucketed per calendar day around the clock, with per-component uptime over a rolling 90-day window - above 99.9% on every component at the time of writing.

The monitoring stack itself is under test. Reconciliation, parity validation, staleness monitors, the job-health registry, the response envelope and every serving guard carry dedicated automated suites, inside a codebase of more than 5,000 automated tests across the API, ingest, historical, volatility-surface and backtest projects, and engine parity between live and historical computation is pinned by fixture. For security posture and operational due diligence, see /security; for licensing and firm-wide terms, /institutional.

Frequently asked questions

How does FlashAlpha know its options data is right?

By checking it against sources that have no reason to agree, several times every trading day: contract-level reconciliation against an independent second source (70,858 contracts across 44 symbols at 99.6% open-interest agreement in the sweep of 2026-09-14), index spot re-derived from the chain via put-call parity, equity index futures anchored to cash, session volume reconciled on two layers, served open interest checked as a conservation invariant, and earnings dates cross-verified across two providers. Disagreement outside tolerance raises an alert.

How can I verify freshness myself?

Read data_as_of. Successful responses carry the delivery timestamp of each feed the response used, plus the node name and endpoint_version. The same object is on the public /health endpoint with no key, and successful historical responses add archive_as_of with the vintage of the rows actually replayed. Compare each feed with its cadence: options feeds move in seconds during the session; settled open interest is a prior-close vintage.

What happens when a data source degrades?

The API refuses rather than degrades: 409 stale_data instead of a frozen price, ghost quotes excluded from IV solving, a gamma flip certified before it is published with gamma_flip_status naming the result, the strategy scorers returning an insufficient_data decision on thin coverage, and a fleet-wide availability control that can withdraw a class or symbol. Every failure mode is explicit and machine-readable, and scheduled reconciliation breaches alert operators.

Do you restate historical data?

No silent restatement. The archive replays the inputs as they were known at the requested minute through the live engine, successful historical responses carry archive_as_of, and any correction is documented in the changelog. That property is what makes point-in-time backtesting on this platform valid.

How is data quality monitored across the fleet?

Pipelines across every host report into one job-health registry - status, staleness, consecutive failures - and the API and ingest subset is published live at /v1/dq/public-summary, with feed liveness evaluated every minute, spot staleness every five minutes in session, the OPRA open-interest publication confirmed before each equity open, and digests to the operators twice a day. External probes exercise the real customer path every 30 seconds, read data_as_of on what they get back, and publish availability at /status.

Can we review the monitoring configuration during due diligence?

Yes. Parameter-level detail - tolerances, schedules, source pairings, escalation practice and SLA riders - is shared with enterprise and systematic customers under NDA as part of commercial due diligence, via /institutional.