At Amazon Global Store I owned the service that moved product data from the AWS global partition into the AWS China partition. A full run was about 2TB. The link was Direct Connect rather than the public internet, with a 150 ms round trip. Explicit rules governed which fields could cross the partition boundary. Every decision below is a trade between security overhead and long-haul throughput.

Written in 2026, four years after the work, entirely from memory and without consulting any internal document. Operational numbers are kept to the order of magnitude that decided each choice; the worked arithmetic uses public facts about TCP, KMS and S3.

Sketch: the shape of a full run
steady state   incremental sync, a fraction of a full run
full run       ~2TB, rare: only when incrementals aren't enough
link           Direct Connect, RTT ~150 ms, effective throughput in the hundreds of Mbps
duration       one overnight window
batch          one business-entity shard each, single-digit GB, on the order of a thousand per run
part           8–16MB, S3 multipart

1. The shape of the job

2TB was not the daily number. Steady state was incremental sync, a fraction of a full run. The full 2TB happened only when incrementals were not enough and the destination had to rebuild the whole catalog. So the service was not optimized for sustained throughput. It was optimized for two things. An infrequent bulk run must not fail partway through, and afterwards every byte must be accountable. Most of what follows comes from that. The complexity went into recovery and evidence, and the throughput only had to be enough to finish one run in one night.

2. Why it had to be built

AWS global and AWS China are two separate partitions, and the split is written into every identifier:

Evidence: the two partitions as they appear in ARNs
arn:aws:s3:::<bucket>        us-west-2     source
arn:aws-cn:s3:::<bucket>     cn-north-1    destination

The two IAM systems are not connected, so a role cannot be assumed across partitions. S3 cross-region replication does not cross partitions, and neither does DataSync. Every managed copy AWS offers works within a single partition. Even where a generic tool could run, two more reasons ruled them out. They move files, and we needed a batch of product entities to arrive together and retry per entity. And they cannot produce the record we had to keep for every batch that crossed: which key was used, who approved it, and which fields were removed before transfer. The Data Security Law and the Personal Information Protection Law both took effect in the second half of 2021, during this work, and that record was a hard requirement.

3. How the keys get across

The encryption is envelope encryption. Each batch asks KMS for one data key. Objects are encrypted locally with that key under AES-GCM, and KMS does not handle the data itself. Calling KMS per object instead would hit API throttling long before 2TB, and throughput would drop with it. Batches were cut by business entity, one category shard per batch, which kept the call count under the limits:

Sketch: KMS traffic for one full run
per batch    2TB ÷ single-digit GB each    ≈  data-key requests in the thousands, well under KMS limits
per part     8–16MB, S3 multipart          →  never touches KMS

The hard part is getting the key to the other side. The ciphertext of a symmetric data key can only be decrypted by the KMS that produced it, and the destination had no way to call the source's KMS, because the two IAM systems are not connected. So the two sides shared no symmetric secret at all.

Sketch: how one batch's key gets across. Reconstructed from memory, field names illustrative
cn-north-1 KMS   create an RSA asymmetric CMK; publish the public key to us-west-2
us-west-2        ask KMS for an AES data key, encrypt the whole batch with it (AES-GCM),
                 then wrap that AES key with the RSA public key
envelope         { key_id, wrapped_data_key, ciphertext }   →   travels with the batch
cn-north-1       KMS Decrypt(wrapped_data_key) with its own private key  →  AES key  →  data

Three properties follow. First, the source only ever holds the public key. For that to hold, two more things are needed. The plaintext AES key is kept only in process memory until the batch is encrypted, and the ciphertext copy returned by the source's KMS is not retained. With both in place, a compromised source cannot decrypt anything it has sent. Second, no cross-account trust is needed, and none could have been built anyway. Third, rotation is routine. The destination creates a new key pair and republishes the public key. The envelope carries the key id, so old and new can coexist. Both KMSs are regional and independent of each other, and every call goes through a VPC endpoint.

4. The limits of the link

I do not remember the exact sysctl values, so I will not quote them. The arithmetic below is enough to show the problem:

Evidence: bandwidth-delay product at 150 ms
1 Gbps × 150 ms    =  150 Mbit in flight  ≈  18.75 MB window per stream
default tcp_rmem ceiling                 ≈  6 MB
6 MB ÷ 150 ms                            ≈  320 Mbps max per stream

The service ran many streams with medium-sized parts instead of tuning kernel receive windows host by host. Larger parts give better steady-state throughput. But retransmits are common on a long fat pipe, and a retry resends the whole part, so 8–16MB was the compromise. Concurrency filled the link and isolated failures. If one part fails, only that part is resent.

5. Resume

Resumption is a state machine, not a retry loop. Each batch carries a manifest listing objects, sizes and checksums. Each object has a row in DynamoDB in one of three states. Restart means reading the checkpoint and applying one rule per state:

Sketch: what happens to each object on restart
VERIFIED    skip
UPLOADED    re-verify the checksum
PENDING     take the multipart upload ID, call ListParts, upload only the missing parts

Object keys are derived deterministically from the entity, so running a batch twice writes the same keys. The whole pipeline is therefore idempotent. A rerun produces no duplicates and leaves no half-written objects.

6. Verification

Verification ran at three levels. Per object, a SHA-256 of the content. Per batch, the manifest, with object count, total bytes and a rolling hash. Above that, a periodic job pulled S3 Inventory, diffed it against the source list, and backfilled whatever was missing. The problem to watch for was at the bottom level:

Evidence: why a multipart ETag is not a content hash
ETag(multipart)  =  MD5( MD5(part 1) ‖ MD5(part 2) ‖ … ‖ MD5(part N) ) + "-" + N

The ETag depends on where the parts were cut, not only on the bytes, so reconciling against it gives wrong answers with no error. What made the top layer hold was S3 Object Lock. The destination bucket was write-once, and a verified object could not be overwritten or deleted for the retention period. The cost was deletion flexibility and storage. Both were accepted, because compliance required that transferred data could not be altered, and a reconciliation result has to remain valid after the job that produced it has ended.

What it cost

What I took away

Decide first whether you are optimizing for the steady state or the worst case. 2TB sounds like a throughput problem, and it was mostly a recovery problem. When a security primitive does not fit in the hot path, move it out of the path rather than dropping it. One KMS call per batch reduced nothing in security. And the constraint that the two partitions could not trust each other also gave a useful property. The sender could never read what it sent.