Data Encoding

Encoding is the durable contract between software versions. Bytes written today may be read by a rolling-deploy peer, a delayed queue consumer, a restored backup, or code written years later in another language. Choosing JSON, Protocol Buffers, or Avro therefore decides more than payload size: it decides how a reader identifies fields, skips what it does not know, applies defaults, frames messages, and proves compatibility.

A durable encoding contract covers record and message representation, framing, schema authority, compatibility, parser safety, and format migration. Column-Oriented Storage owns analytical page encodings such as dictionary, run-length, and bit packing. API Design Patterns owns public resource and protocol semantics, while Change Data Capture owns database-log envelopes and sink application.

Boundary and workload contract

Inventory every boundary separately: public HTTP, internal RPC, event stream, database blob, WAL, snapshot, cache value, and analytical file. For each, record:

• producer and consumer languages, ownership, and deployment independence;
• message rate and size distribution, including maximum and adversarial sizes;
• retention and replay horizon, and how many schema generations may coexist;
• latency, CPU, allocation, and network/storage budgets;
• random field access versus full materialization, streaming, and batching;
• unknown-field, missing-field, numeric, null, ordering, and canonicalization semantics;
• confidentiality, signing, tenant isolation, and regulatory deletion requirements.

A short-lived request between lockstep components may tolerate generated-code coupling. A ten-year event log must retain writer schema and support all legal historical versions. A signed message needs one canonical byte representation. A zero-copy game asset assumes trusted generated bytes; an Internet endpoint needs bounded parsing and verification before following offsets.

Encoding does not define business meaning. Changing timeout_ms from milliseconds to seconds, reusing enum 3, or changing an absent boolean from “inherit” to false can be wire-compatible and still break every consumer. The schema contract includes units, identity, cardinality, default meaning, normalization, and lifecycle, not just types.

Envelope state and invariants

A durable record should be self-identifying enough to find its decoder and delimit its bytes. One general envelope is:

magic | envelope_version | schema_id | flags | payload_length
      | producer/tenant metadata | encoded payload | checksum/auth tag

Not every protocol needs every field. A container file may store one schema in its header for millions of records; an event stream may place a small registry ID in each message; an RPC method already identifies its request type. The invariant is that the reader can unambiguously determine framing, schema, and integrity before trusting payload offsets or allocating unbounded memory.

The control plane stores immutable schema definitions, IDs or fingerprints, subject ownership, compatibility policy, generated-code/toolchain versions, deprecation state, and audit history. The data plane caches schemas, encodes and frames messages, transports bytes, and decodes them. Schema deletion is constrained by retained data and offline clients, not by the age of the registry entry.

Useful invariants are:

1. One schema ID forever identifies the same canonical schema bytes.
2. A producer emits only a schema registered and authorized for that subject or method.
3. A consumer either decodes under explicit writer/reader rules or rejects; it never guesses from payload shape.
4. Unknown fields survive any intermediary that promises transparent forwarding.
5. Removed wire identities (field numbers, enum numbers, names where positional resolution uses them) are not repurposed.
6. Frame length, nesting, allocation, and decompression are bounded before expensive work.
7. Compatibility gates cover every retained version required by the rollout and replay horizon.
8. Integrity and signature verification use the exact declared bytes or a standardized canonical form.

Three representation models

JSON: names and types in every record

JSON carries member names and a small set of value kinds. It is widely interoperable and inspectable, making it a strong external boundary. It does not carry a schema, integer width, timestamp type, byte string, or canonical member order.

RFC 8259 permits implementations to limit numeric precision; JavaScript’s common binary64 number representation exactly represents integers only through 2^53 - 1. Send larger identifiers and exact decimals as strings with explicit format rules, or use a consumer that preserves arbitrary-precision numbers. Decide whether absent, null, empty, and zero differ. Reject or define duplicate object names; different parsers may retain the first value, the last, or all values.

Signing or hashing arbitrary serialized JSON is unsafe because whitespace, escaping, number spelling, and member order can vary without changing the abstract value. Use a canonicalization scheme such as JCS (RFC 8785), or sign the original framed bytes and never parse/re-serialize them before verification. JSON Schema can validate structure, but compatibility policy and tolerant-reader behavior still require governance.

Protocol Buffers: numbered fields on the wire

A protobuf message is a sequence of field keys and values. The key combines a field number with a wire type; length-delimited fields carry their own size. Names exist in source schemas, but the numeric tag is the durable identity.

message Account {
  string account_id = 1;
  optional string display_name = 2;
  reserved 3, 7;
  reserved "legacy_status";
}

Unknown field numbers can be skipped because the wire type says how. Generated runtimes usually retain unknown fields when parsing and serializing the same message, but transformations to JSON, hand-built objects, or another message type may discard them. Test every intermediary if forward compatibility depends on preservation.

Adding a fresh optional field is normally safe. Removing a field requires reserving its number and preferably its name. Never reuse a tag, and do not casually change types: two protobuf types can share a wire type yet interpret the same bytes differently. Presence also matters. An optional scalar distinguishes absent from present-with-default; an implicit proto3 scalar often does not. oneof, maps, packed repeated fields, and enum additions each have documented cross-version constraints.

Protobuf fits internal RPC and streams where generated types, compact messages, and unknown-field skipping are valuable. It is not self-describing: the receiver still needs the correct message type and schema revision family.

Avro: writer schema resolved against reader schema

Avro binary records omit per-field tags and follow the writer schema’s structure. A consumer obtains that exact writer schema from a container header or registry ID, then resolves it against its own reader schema. Fields match by name and aliases; writer-only fields are skipped; a reader field absent from the writer needs a default in the reader schema; permitted numeric promotions are explicit.

Defaults are read-time resolution values, not values automatically written by producers. Adding a field with a default lets a new reader consume old records. Removing a field can break an old reader if that old reader expects the field and its schema provides no default when reading new data. Union branch order, aliases, logical types, and defaults are wire semantics and deserve compatibility tests rather than intuition.

Avro works particularly well for event streams and data files where schema distribution is already part of the architecture. Positional encoding is compact, but losing the writer schema makes the payload uninterpretable.

Offset-based and schema-light formats

FlatBuffers and Cap’n Proto place tables and offsets so generated code can access selected fields without constructing a complete object graph. They trade compactness and ergonomic mutation for low allocation and partial access. Validate an untrusted buffer before following offsets, and keep it alive for every view that references it.

CBOR and MessagePack encode JSON-like values more compactly and include useful binary/numeric types. They improve representation efficiency but do not create a compatibility contract by themselves. CBOR offers deterministic-encoding rules for protocols that need canonical bytes. The decisive axis is schema and evolution model, not “text versus binary” alone.

Encode and decode paths

On production, a producer selects the registered schema and method, validates semantic constraints, encodes the payload, frames it, optionally compresses it, and applies checksum, encryption, or authentication in a documented order. Compression, encoding, and encryption are independent layers. Compressing after encryption is ineffective; signing decoded objects rather than canonical bytes invites disagreement.

A consumer first authenticates the peer and checks the outer frame’s magic, version, declared length, flags, and integrity. It enforces compressed and decompressed size limits, recursion depth, collection counts, and deadlines. It resolves the writer schema from a bounded cache, decodes into its reader model, applies semantic validation and authorization, and only then performs business work. A schema registry is normally off the hot path after cache warm-up, but a cold process still needs a defined behavior when the registry is unavailable.

Streams need framing because TCP and byte files do not preserve message boundaries. A fixed-width or varint length prefix is common, but parse the prefix under a strict maximum before allocating. Checksums detect corruption; they do not authenticate an attacker. AEAD encryption authenticates the exact ciphertext and associated header fields, but key ID and nonce rules become part of the envelope version.

Compatibility and rollout mechanics

Use precise direction names:

backward compatible: new reader can read old writer data
forward compatible:  old reader can read new writer data
full compatible:     both directions
transitive:           checked against every supported historical version,
                      not only the immediately previous schema

The required direction follows deployment and replay. Deploying consumers before producers relies on backward compatibility. Deploying producers while old consumers remain relies on forward compatibility. Long-retained topics and database blobs usually need transitive checks because version 8 may still read version 1 data even if each adjacent pair passed.

A safe additive rollout introduces a field as optional with a documented default meaning, releases readers that tolerate both states, then releases writers. After observability shows old readers and old data are outside the support horizon, a later migration may require the field semantically. Wire-level required fields turn deployment timing into a permanent constraint and should be avoided.

Breaking changes use a new field identity or message version. Producers may dual-write old and new representations, or a bridge may translate at one explicit boundary. Compare both decode paths, move consumers in cohorts, stop old production, wait through retention and retry horizons, then retire the old schema. Do not mutate a registered schema in place or let a registry ID resolve differently by environment.

Semantic evolution needs extra gates. Units, enum behavior, normalization, privacy classification, and identifier scope belong in API review and contract fixtures. An enum reader should define what an unknown numeric value does. An intermediary should not convert it to a familiar default and erase evidence of the new state.

Specialized failure traces

Protobuf tag reuse silently changes identity

Version 1 writes string phone = 5; version 2 removes it; version 6 defines bytes encryption_key = 5. Old stored records now populate the new field with phone bytes. The payload may parse successfully, so no alert fires. Reserve removed tags forever and gate schemas against the complete history.

Transparent gateway drops unknown fields

A new producer adds field 12. An old gateway parses the message, maps known fields into a new object, and re-serializes it; field 12 vanishes. A later new consumer treats absence as “disabled.” Either proxy raw framed bytes, use a runtime path proven to preserve unknowns, or version the transformation contract.

JSON identifier rounds into another key

A 64-bit database ID crosses a JSON client that parses numbers as binary64. It is rounded, then sent back on an update and addresses the wrong or nonexistent row. Encode exact large integers as decimal strings and validate their canonical form at the boundary.

Cold consumer cannot resolve schema

After a regional restart, consumers have empty caches and the registry is unavailable. They can read message bytes but not the Avro writer schema, so lag grows. Replicate registry metadata, prewarm schemas referenced by assigned partitions, or bundle an immutable schema cache with the deployment; never guess the latest schema.

Length prefix becomes an allocation attack

An unauthenticated frame declares a 20 GiB payload or a tiny compressed body expands by orders of magnitude. Allocating or decompressing before enforcing limits exhausts memory. Bound wire bytes, decoded bytes, nesting, collection elements, and CPU independently; reject before allocation where possible.

Wire-compatible unit change corrupts behavior

Both schemas declare int64 timeout = 4, but a new producer changes milliseconds to seconds. Every compatibility checker passes and old consumers wait a thousand times too long. Encode the unit in the field name or a semantic type and include meaning in cross-version contract tests.

Capacity and cost model

Let message rate be Q, mean framed bytes before compression be B, compression ratio be r = compressed/uncompressed, encode and decode CPU costs be ce and cd seconds/message, and average number of decode/encode hops be H:

network bytes/s        ~= Q * (header_bytes + B * r)
producer CPU cores     >= Q * ce / target_utilization
consumer CPU cores     >= Q * cd / target_utilization
pipeline codec CPU     ~= Q * H * (ce + cd)
retained bytes         ~= network_bytes/s * retention_seconds * replication

Use distributions, not means, for allocation and tail latency. A few multi-megabyte messages can dominate garbage collection and queue residence. Batching amortizes headers, syscalls, compression dictionaries, and registry lookups, but adds fill delay and makes one corrupt batch affect more records. Measure payload bytes, allocations, copies, encode/decode time, compression CPU, and downstream storage, not a microbenchmark of one object.

Schema-cache memory is roughly active schema count times parsed-schema and generated-decoder footprint. High-cardinality per-tenant schemas can exhaust it or turn registry lookup into a denial of service. Bound schemas per subject and tenant, use eviction that respects hot assignments, and separate cache-miss latency from decode latency.

Security, governance, and observability

Treat parsers and registries as security-sensitive infrastructure. Authenticate schema registration and reads, namespace subjects by environment and tenant, require review for sensitive-field additions, and keep an immutable audit trail. A schema reveals field names and business structure even if payloads are encrypted. Limit who can enumerate it.

Sensitive fields persist in queues, dead-letter stores, logs, traces, fixtures, and old schema versions. Classify fields, redact debug rendering, encrypt at the appropriate envelope or field layer, and propagate retention/deletion policy. Per-tenant size, rate, schema-count, nesting, and decode-CPU quotas prevent one workload from exhausting shared consumers.

Observe encode/decode latency and allocation by schema ID, framed/compressed/decoded size distributions, parse and semantic-validation failures, unknown-field and unknown-enum rates, defaulted-field frequency, registry cache hit and fetch latency, compatibility rejections, producer use of deprecated versions, oldest retained schema reference, decompression-limit rejects, and poison-message age. A rise in defaults can be the first evidence that producers stopped sending a field.

Verification includes golden byte fixtures for each supported version and language, an old-reader/new-writer matrix, round trips that preserve unknowns, deterministic-canonicalization tests, and schema-registry compatibility tests against full history. Property and fuzz tests truncate every byte position, mutate lengths and tags, generate deep nesting and duplicate JSON names, and compare independent implementations. Restore tests decode historical queue segments, snapshots, and WAL with the release candidate, not only fresh data generated by it.

Decision framework

Use JSON for external, human-inspectable boundaries where universality matters and exact numeric/canonical rules are documented. Use Protobuf for typed RPC and streams where numbered-field evolution and generated code fit. Use Avro when writer-schema resolution and registry-backed long-lived records are central. Use FlatBuffers or Cap’n Proto for measured partial-access/allocation bottlenecks with controlled buffers. Use CBOR or MessagePack when a schema-light value model is intentional, not as a substitute for governance.

The best format is the one whose identity and evolution rules match the lifetime and ownership of the data. Payload size matters; the ability to deploy, replay, audit, and reject hostile bytes without ambiguity matters more.

Primary references

• IETF. RFC 8259: The JavaScript Object Notation (JSON) Data Interchange Format.
• Rundgren, A., Jordan, B., and Erdtman, S. RFC 8785: JSON Canonicalization Scheme.
• Google. Protocol Buffers Encoding and Proto Best Practices.
• Apache Avro. Specification: Binary Encoding and Schema Resolution.
• IETF. RFC 8949: Concise Binary Object Representation (CBOR).
• Google. FlatBuffers Internals and Buffer Verification.
• Confluent. Schema Evolution and Compatibility.