Identities for pipelines and people
Lesson 1 of 4 in Authentication and Authorization over Data.
Two populations touch a data platform and they need opposite designs. Pipelines run unattended, thousands of times a day, and must never hold a password. People arrive through a browser or a SQL client, change teams, join and leave, and must be traceable individually. Most of the security faults in a data platform come from treating one like the other — a human sharing the pipeline's role, or a pipeline carrying a credential someone typed in.
For pipelines the rule is one line long: an AWS-to-AWS call uses a role, not a stored credential. Every compute surface in the data stack has one. An AWS Glue job runs with a service role; the equivalent on Amazon EMR is the EC2 instance profile, or better, a per-job runtime role; Lambda has an execution role; an Amazon Managed Workflows for Apache Airflow environment has one, and so does a Redshift cluster when it reads from Amazon S3. The design discipline is one role per purpose, scoped to exact prefixes and table ARNs — not one shared role that holds the union of everything the platform ever needed, which is what happens by default and is impossible to narrow later because nobody knows what still depends on it.
The role that cannot be avoided is the one handed to a service. Creating a Glue job with a role attached requires iam:PassRole for that specific role, and that permission is the gate that stops an engineer escalating by attaching an over-privileged role to a job they control. Scope it to the role ARNs that are legitimately passable; iam:PassRole on * is an administrator in disguise.
A few credentials genuinely cannot be roles, and that is what AWS Secrets Manager is for: the password for a self-managed database the pipeline reads over JDBC, a partner's API key, an OAuth client secret. Where the target can authenticate with IAM, prefer that — Amazon RDS and Aurora support IAM database authentication, Amazon Redshift supports IAM-generated credentials, and Amazon MSK supports IAM access control — because a credential that does not exist cannot leak, be committed to a repository or need rotating.
Three authentication choices come up often enough that the shape of each is worth holding.
The warehouse. A pipeline reaching Amazon Redshift has three paths. GetClusterCredentials mints a temporary database user and password from an IAM identity, with the DbUser and DbGroups values restrictable by condition key and AutoCreate able to create the database user on first use — this is the answer for AWS-native callers, because there is no stored password at all. Identity Center integration carries the human identity through, which is what you want for analysts so that grants and logs name people. A password in Secrets Manager remains correct for the third-party tool that can only speak JDBC with a static credential, and for that case rotation is the control.
The relational source. IAM database authentication on Amazon RDS and Aurora generates a token that is used where a password would go. Two properties decide whether it fits: the token is valid for 15 minutes, so a connection pool must regenerate rather than cache forever; and the connection must use TLS, since the token travels as the password. AWS documents a limit on new IAM-authenticated connections per second, which is why a high-churn workload is better served by a pooled credential from Secrets Manager.
The stream. Amazon MSK offers three client authentication paths and they divide on who owns the credential. IAM access control authorises topic-level actions by IAM policy — nothing to store. SASL/SCRAM keeps a user name and password in Secrets Manager, and three constraints are examinable: the secret name must begin with AmazonMSK_, it must live in the same account and Region as the cluster, and it cannot use the default aws/secretsmanager key — a customer managed key is mandatory. Mutual TLS authenticates with client certificates from AWS Private CA, which is what an existing Kafka estate usually already has; authorization for the TLS and SCRAM paths is by Kafka ACL rather than IAM.
| Surface | What it authenticates with | Scope of the credential | The trap |
|---|---|---|---|
AWS Glue job or crawler | A service role, passed at job creation | Whatever the role allows — one role per job is the discipline | A shared "glue-role" accumulates every permission the platform ever needed and cannot be narrowed later |
Amazon EMR step | The EC2 instance profile, or a per-job runtime role | Instance profile = the whole cluster; runtime role = that one job | With an instance profile the profile must hold the union of every job on the cluster, so a shared cluster is shared access |
Amazon Redshift reading S3 | An IAM role associated with the cluster, named in | The prefixes the role allows | Several roles can be associated; |
Analyst in Athena or a SQL client | Identity Center, ideally with identity propagation | The person, carried through to grants and logs | Without propagation every analyst appears as the shared role they assumed, in grants and in CloudTrail |
Self-managed database over JDBC | A password in Secrets Manager, or an IAM auth token where supported | The database user | A password in a job parameter or a Glue connection property that was never encrypted — see connection password encryption |
Kafka client on Amazon MSK | IAM access control, SASL/SCRAM, or mutual TLS | Cluster, topic and group actions | A SCRAM secret on the default |
Key terms: Amazon EMR runtime role, Redshift IAM database authentication, RDS and Aurora IAM database authentication, MSK IAM access control, Trusted identity propagation
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.