Data contracts

This reference design follows an event from an application into several analytical and operational consumers. I assume the producer controls publication, each event carries a stable identifier and schema version, and accepted history remains available for a defined recovery period. Consumer requirements differ, so the contract states which guarantees are shared.

The producer owns field meaning, units, identifier scope, event-time semantics, and valid absence. An absent amount is not automatically zero; a user identifier is not necessarily global. The platform owns enforcement and transport. Consumers document the versions and meanings their processing depends on.

Registration checks the proposed schema against an explicit compatibility policy. Backward compatibility lets a newer reader consume older data; transitive checking extends that comparison beyond the immediately preceding version. The allowed changes depend on the serialization format and original field definitions.

Those checks cannot establish that a duration still means milliseconds or that an account identifier has the same scope. Semantic changes need review, examples, and a migration plan. Prefer a new field or event version when meaning changes. Specify consumer rollout order, the supported overlap period, and when the old representation can be retired.

The producer validates against a pinned contract before publishing. An ingress gate independently checks the declared version, required fields, bounds, and cross-field rules. Successful serialization alone is insufficient. Validation failure returns a precise reason the producer can act on.

Apply a documented privacy policy before the event reaches a shared log or additional consumers. Prefer omitting unnecessary sensitive fields at collection; use approved transformations where collection cannot be avoided. An allowlist prevents new fields from silently spreading. Transformation rules are versioned, and pseudonymous identifiers retain access and retention controls because they may still be linkable.

If delivery has already been accepted, persist rejected events in a restricted quarantine before acknowledging progress. Store the contract version, reason code, source position, and permitted payload or protected reference. Quarantine has its own retention and access rules; a validation error must not copy sensitive content into ordinary logs.

Route the failure to the producer owner and define whether other events may continue. An ordered entity stream may need to pause that entity while unrelated work proceeds. Monitor storage headroom. If quarantine cannot persist the record, apply backpressure or fail publication rather than silently dropping it.

An approved repair records what changed and links the correction to the rejected event. Replay uses a bounded range, pinned transformation versions, and the current publication gate, including privacy checks. A new schema passing validation does not mean every historical record can be reinterpreted safely.

Backfills write a separate output version or staging destination, then compare counts, missing intervals, duplicates, and relevant aggregates before publication. Define how concurrent arrivals join the result. Preserve stable identifiers for transport retries; use explicit correction or rebuild versions for changed effects. A Kafka offset records transport progress, not successful application of every downstream effect.

Track rejected-event rate and age, unknown versions, field completeness, source freshness, quarantine capacity, and consumer lag. Group failures by producer and contract version. Consumer acknowledgements and retirement dates make obsolete dependencies visible; a registry entry alone does not identify who will fix a broken integration.

This design costs coordination, retention, and migration work. A small system may need only a versioned schema, producer tests, and one receiving validator. A shared registry becomes useful when independent teams evolve on different schedules. In either case, operational ownership and semantic examples are part of the contract.