At Amazon Global Store I owned a cross-Pacific data-transfer service. A single task ran to roughly 2TB, moving data from the AWS China region to the United States, across a long fat pipe with a 150ms round trip and across a legal border. Nearly every design decision in that system came down to the same axis: security overhead on one side, long-haul throughput on the other, and which one you give ground to.
This is about several specific decisions on that axis, and what each of them actually cost.
What 2TB actually means here
2TB is the size of a single task, not daily throughput. The steady state is incremental sync, tens of gigabytes a day. 2TB is the full-load case: onboarding a new category, a complete catalogue refresh before a major sale, or an upstream schema change that forces a backfill.
That distinction decided the architecture. If the goal were sustained high throughput you would optimise the steady state. The real goal was different: the infrequent large batch must not fall over, and it must be auditable. Those two goals produce different systems. The first optimises average latency; the second optimises failure recovery and the chain of evidence. Almost all the complexity in this system went into the second.
What moved was Global Store business data. Which categories is not the interesting part. The property that mattered is this: within a single record, some fields could cross the border and some had to be stripped before they did. China's Data Security Law and PIPL both landed in the second half of 2021, so that filtering was not a nice-to-have. It was a hard constraint.
Why off-the-shelf tools did not fit
The hardest reason is AWS partition isolation. The China regions and the global regions are two separate partitions. The ARN prefixes differ (aws-cn against aws), the IAM account systems do not interoperate, you cannot do a cross-account assume-role across the boundary, and S3 cross-region replication does not cross it either. AWS's own managed replication, which works within one partition, is simply unavailable here. So is DataSync. This is not a configuration problem; it is what the boundary means.
The second reason is business semantics. A general-purpose tool moves files. What we needed was consistent synchronisation of a batch of product entities. Those look identical while everything works and diverge sharply when something fails: recovery has to retry and reconcile at the granularity of a business entity, not a file. One entity might map to a dozen objects, and half an entity is worth nothing.
The third reason is audit. Which key encrypted this batch, who approved the export, which fields were stripped: no off-the-shelf tool produces that chain. In a compliance review that chain is very nearly the whole conversation. Saying you filtered the data and being able to prove you filtered it are different things.
The first tradeoff: get KMS out of the data path
The mechanism is envelope encryption. Each batch requests one data key from KMS, the objects are encrypted locally with AES-GCM, and KMS only ever handles keys. It never touches the data.
This is not an optimisation, it is a requirement. Calling KMS once per object would run straight into the service's API rate limits; with 2TB of small objects the quota is exhausted and throughput collapses. Envelope encryption moves the call volume from the order of the object count to the order of the batch count, three orders of magnitude apart.
The cost is that the key's blast radius gets coarser. One data key covers a whole batch rather than one object, so a leaked data key exposes the batch rather than a single file. I accepted that, because batches are cut along business entities anyway. The exposure boundary is the same as the boundary of one business synchronisation, rather than some new range invented by the encryption scheme.
The part that took longest: moving a key across a partition
The ciphertext of a symmetric data key can only be decrypted by the KMS that generated it. Since the two partitions' IAM systems do not interoperate, the destination has no way to call back to the source's KMS. The symmetric route is a dead end.
What we did instead was asymmetric. The destination creates an RSA asymmetric CMK in KMS in us-west-2 and publishes the public key to the China side. The source wraps its AES data key with that public key, and the wrapped key travels inside the envelope. The destination decrypts it with the KMS in its own partition and recovers the data key.
Several properties of this arrangement are why I like it. The source only ever holds a public key, so its compromise carries no risk. Crossing the partition requires no shared symmetric secret and no cross-account trust relationship at all: there is no trust between the two partitions, only a public key. Rotation is straightforward too. The destination creates a new CMK pair and republishes the public key, and because the envelope carries a key id, old and new coexist without a cutover.
Chunking and concurrency: why we did not tune a single stream
The round trip across the Pacific is about 150ms. A single TCP stream cannot fill the bandwidth on a pipe like that, and the bandwidth-delay product says why. At 150ms, saturating 1Gbps on one stream needs a window approaching 19MB, while the default ceiling for tcp_rmem on Linux is typically around 6MB, which caps a single stream somewhere in the low hundreds of megabits.
Given that gap there are two roads: tune kernel parameters until one stream is big enough, or use concurrency to fill the pipe. We took the second, with many parallel streams and medium-sized chunks.
Not just because it was easier. Large chunks give better steady-state throughput, but retransmission is not rare on a long fat pipe, and the cost of one retry scales with chunk size. The bigger the chunk, the more you forfeit each time something fails. Medium chunks plus concurrency give up a little in the steady state and buy failure isolation, which is exactly the design goal from earlier: the infrequent large batch must not fall over. Within an object, parts of 8 to 16MB go through S3 multipart, and that layer never touches KMS.
Resumption is manifest-driven
Every batch produces a manifest recording the object list, sizes and checksums. Every object has a state machine in DynamoDB: PENDING, UPLOADED, VERIFIED.
On restart the job reads its checkpoint. VERIFIED objects are skipped. Objects marked UPLOADED but not yet verified are re-verified. Objects that never finished are resumed through their multipart upload ID: call ListParts, upload only the missing parts. Object keys are generated deterministically, so the whole pipeline is idempotent. Running it twice produces no dirty data, and that property matters more than any other when something retries itself at three in the morning.
Consistency in three layers, and one trap
At the object layer we use SHA-256, not the ETag. This trap deserves its own paragraph. For a multipart upload the ETag is the MD5 of the concatenated part MD5s, with the part count appended. It is not the hash of the content. Reconciling across two sides with it gives you wrong answers, and it gives them quietly.
At the batch layer we compare against the manifest: object count, total bytes, rolling hash.
Above both there is a periodic reconciliation job that diffs S3 Inventory against the source-side listing and automatically re-sends whatever is missing. That layer exists because the first two are only true at the moment of transfer, and the data has to keep sitting there afterwards.
Object Lock: flexibility traded for immutability
The destination bucket has S3 Object Lock enabled, so objects cannot be overwritten or deleted for the duration of the retention period. The cost is direct: deletion flexibility is gone and storage costs go up.
We accepted it, because immutability of exported data is precisely what the compliance position requires. It is also what makes the periodic reconciliation meaningful. If an object could still be modified after it passed verification, the reconciliation result would have no lasting validity; you would only know the data was correct at one instant.
Where the boundaries are drawn: VPC endpoints and IAM
Every call to S3, KMS and DynamoDB goes through a VPC endpoint and never touches the public internet. This is the other half of the same idea as Direct Connect. Direct Connect keeps the trans-Pacific traffic off the public internet; VPC endpoints keep the in-region traffic to AWS services off it as well. Together they mean no segment of this data path runs over the public internet at all.
On the IAM side, each partition has its own entirely separate set of roles, and they do not know about each other. The source role can read the business data store, call KMS in its own region to generate a data key, and write to the staging location. The destination role can call KMS in its own region to decrypt, and write to the Object Lock bucket. No single role holds permissions on both sides, because crossing the partition is impossible in the first place. That impossibility is the property you want here: it caps the blast radius of a stolen credential at one partition.
How the system actually runs
Threading the pieces above into one path:
- Trigger. Full-load tasks are scheduled into an overnight low-traffic window rather than started on demand. Incremental tasks run on their own schedule.
- Batching. Batches are cut by business entity rather than by size, one category shard per batch, landing between 1 and 4GB. A full 2TB run is on the order of a thousand batches, so the whole task makes only around a thousand KMS calls and sits comfortably inside the quota.
- Filtering. Fields that may not leave the country are stripped from the records, and the result goes into the audit log along with the version of the rules that produced it.
- Encryption. Request a data key from KMS in the local region, encrypt objects locally with AES-GCM, then wrap the data key with the destination's RSA public key and put the wrapped key in the envelope.
- Transfer. Parallel streams, 8 to 16MB parts within each object through S3 multipart, over Direct Connect throughout. Every object's state is written to DynamoDB.
- Landing. The destination decrypts the data key with KMS in its own partition, decrypts the objects, and writes them into the Object Lock bucket.
- Verification. SHA-256 at the object layer, manifest comparison at the batch layer, then the object is marked VERIFIED.
- Reconciliation. Afterwards the periodic job diffs S3 Inventory against the source listing and re-sends any difference.
A failure at any step resumes from the checkpoint rather than from the beginning. That is the most expensive part of the design, and the part that earned its cost most clearly.
The general version
Three things from this are worth carrying elsewhere.
Work out whether you are optimising the steady state or the tail. "2TB" sounds like a throughput problem and is actually a recovery problem. Once that is clear, most of the design decisions answer themselves.
Move an expensive security primitive off the hot path instead of going around it. Envelope encryption does not weaken anything. It puts KMS where it belongs, once per batch rather than once per object. The genuinely bad move is to drop encryption because it is slow, which trades away security rather than trading away design effort.
What a boundary refuses to do is sometimes exactly what you want. Two AWS partitions cannot assume roles into each other, which was pure friction early in the project. But the asymmetric key scheme it eventually forced is safer than a design built on mutual assume-role would have been, because the source never has the ability to decrypt what it sends.