KMS keys and who may use them

Lesson 3 of 5 in Encryption and Masking.

The deep mechanics of AWS KMS — key policy evaluation, grants and their constraints, multi-Region keys, CloudHSM and external key stores — are taught on the security path at key types, key policies, and grants. This lesson is the data engineer's slice: how many keys, where their permissions bite in a pipeline, and which mistakes break a job rather than a security property.

Start with how many keys. The useful unit is a key per trust boundary, not a key per bucket. A lake with raw, curated and restricted zones usually wants three keys, because the key policy then becomes a second gate on each zone — a role with s3:GetObject on the restricted prefix and no grant on its key reads nothing. One key for everything removes that gate; a key per table makes key administration the bottleneck and multiplies the cost of getting a policy wrong. Keys are also the simplest blast-radius control available: disabling a key makes one zone unreadable in seconds without touching a single IAM policy.

Where the permissions actually bite, in order of how often each one causes an outage:

The pipeline role needs kms:Decrypt to read and kms:GenerateDataKey to write. A job that reads encrypted input and writes encrypted output needs both, on possibly two different keys. "It can read but not write" is almost always a missing GenerateDataKey on the destination key.

The catalog is separate from the data. With Data Catalog encryption on, every client needs key permission simply to resolve a table, so a job can fail before it touches an object — and the error points at the catalog.

Cross-account reads need the key policy. The reader can list and describe and then fail on content. And an AWS managed key cannot be used at all for cross-account access, because its policy is not editable.

Service principals need their own grant. Firehose writing to an encrypted destination, CloudTrail writing to an encrypted bucket, Redshift unloading to an encrypted prefix, Athena writing encrypted results, Backup copying into an encrypted vault — each needs the service's access to the key, usually with a kms:ViaService condition narrowing the grant to calls arriving through that service. "The pipeline works when I run it and fails on the schedule" is this.

Two constraints worth stating because they change designs. Cross-Region replication of encrypted objects needs a key in the destination Region — a key is Regional, so either the replication configuration names a destination key or you use a multi-Region key. And key deletion is irreversible and takes effect after a waiting period of 7 to 30 days, during which the key can be cancelled; after it elapses, every object encrypted under that key is permanently unreadable. Rotation is unrelated to this and safe: old material is retained so old ciphertext still decrypts.

A job that was working has started failing with AccessDenied. Which key permission is missing?

Interactive decision tree — outcomes:

  • Data Catalog encryption was enabled and this role has no key permission

    Metadata encryption makes every catalog client a KMS caller. The role needs kms:Decrypt on the catalog key to read a table, and kms:Encrypt on the connection-password key if it creates or updates a connection. The giveaway is that the job fails before it opens any data file.

  • The role is missing `kms:Decrypt` on the source key

    Both sides have to allow it: the key policy must admit the principal (directly, or by delegating to IAM), and the role’s identity policy must allow kms:Decrypt on that key ARN. A role scoped to the wrong key ARN — the curated key when the data is on the raw key — produces exactly this.

  • Missing `kms:GenerateDataKey` on the destination key

    Writing needs a new data key, so kms:Decrypt alone is not enough — and when the destination is a different zone it is a different key. This is the specific shape of "the job reads fine and cannot write", which gets misdiagnosed as a bucket permission because the S3 error is the one the caller sees.

  • A service principal needs access to the key

    Firehose, Athena, CloudTrail, Redshift UNLOAD, AWS Backup and S3 replication act on your behalf and need their own key access, typically granted in the key policy with a kms:ViaService condition. "Works interactively, fails on the schedule" is this nearly every time, because the interactive run used your permissions and the scheduled one did not.

  • The key is disabled, or pending deletion

    If nobody can read — administrators included — look at the key state rather than at policies. A disabled key denies every operation, and a key pending deletion behaves the same way; it can be cancelled during the 7-to-30-day window, and after that every object under it is permanently unreadable. Alarm on DisableKey and ScheduleKeyDeletion for every key that protects a dataset.

  • The key policy does not admit the external account

    Cross-account use needs the key policy to name the external principal and that principal’s own identity policy to allow the action — the two-sided rule again. If the data is on an AWS managed key, stop: that policy cannot be edited, and the only route is re-encrypting under a customer managed key.

Key terms: KMS key policy, kms:ViaService, KMS key rotation, Envelope encryption, Encryption context

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.