Troubleshooting authentication

Lesson 4 of 4 in Authentication Strategies.

Authentication is examined in two halves: designing it, and troubleshooting it. The exam asks the second half as a method question — given a symptom and a log line, what do you check next? The method starts with one split that candidates routinely skip.

Authentication failure means AWS does not accept that you are who you claim. Nothing was authorized because no principal was established. The event is an AssumeRole, AssumeRoleWithSAML, AssumeRoleWithWebIdentity, or ConsoleLogin that failed, and the fix lives in a trust policy, an IdP configuration, a certificate, or a clock.

Authorization failure means AWS knows exactly who you are and will not let you do that. The event is the API call you attempted, with an AccessDenied error, and the fix lives in the policy chain — covered in Policy Evaluation.

Confusing them wastes the whole investigation: no amount of policy editing fixes an expired IdP signing certificate, and no amount of IdP work fixes a permissions boundary. Walk the split first.

A user cannot get in. Triage it.

Interactive decision tree — outcomes:

This is an authorization problem
The principal was established, so authentication worked. Stop looking at the IdP. Decode the authorization failure message (sts decode-authorization-message on the encoded message in the error), then walk the chain: SCP or RCP, resource policy, permissions boundary, session policy, identity policy — and the KMS key policy if a key is involved.
The request never reached AWS
No CloudTrail event means no API call arrived. Look earlier: DNS or network egress from the client, a proxy, an endpoint policy on a VPC endpoint that dropped the request, or an IdP that failed before redirecting. For Identity Center, check the portal URL and whether the user has any assignment at all — a user with zero assignments sees an empty portal, not an error.
The both-sides rule was broken
AssumeRole needs two allows: the role’s trust policy must name the caller as a principal for sts:AssumeRole, and the caller’s own identity policy must allow sts:AssumeRole on that role ARN. Missing either produces the same AccessDenied. A trust policy naming the account root delegates the decision to that account’s administrators — it does not remove the identity-policy requirement.
A trust-policy condition is unmet
The usual suspects: a third-party integration not passing ExternalId; aws:PrincipalOrgID on a caller that has moved out of the organization; an MFA condition on a session that did not carry MFA details; aws:SourceArn pinned to a resource that was recreated with a new ARN. The CloudTrail event records the request parameters — compare them with the condition literally.
SAML attribute or role mapping mismatch
The assertion’s https://aws.amazon.com/SAML/Attributes/Role attribute must carry role-ARN,provider-ARN pairs that exist in the account, and the IAM SAML provider must be the one named. Group-to-role mapping rules in the IdP are the usual culprit after a group rename. Also check SAML:aud conditions against the relying-party identifier the IdP actually sent.
IdP certificate expired or rotated
AWS validates the assertion signature against the metadata document uploaded to the IAM SAML provider (or the Identity Center external IdP configuration). Rotating the IdP signing certificate without re-uploading metadata breaks every sign-in at once — the signature is no longer verifiable. The symptom is organization-wide and sudden, which is the tell. The same shape of failure comes from clock skew: assertions carry validity windows, so a host more than a few minutes off fails authentication.
OIDC token or claim condition problem
Check three things: the IAM OIDC provider’s issuer URL and client id (audience) match the token; the trust policy’s claim conditions (...:sub, ...:aud) match the token’s actual claims — a renamed branch or repository breaks a pinned sub; and the token has not expired. For EKS, confirm the cluster’s OIDC provider is registered and the service account annotation names the right role.
Console sign-in failure
For an IAM user: password age or policy, a missing console password, or an unmet MFA requirement. For root: the account email, MFA device, and whether centralized root access has removed the root credential for that member account on purpose. For Identity Center users the console is reached through the access portal, not the IAM sign-in page — using the wrong URL is a common false alarm.
Identity Center provisioning or assignment drift
If the IdP side is healthy, check Identity Center: has the user been provisioned by SCIM, is the group membership present, does an account assignment exist, and has the permission set been re-provisioned since it was last edited? A stale provisioning state is the single most common Identity Center support ticket.
Look wider — an organization policy may be denying the assume itself
sts:AssumeRole is an API call like any other, so an SCP, an RCP on AWS STS, or a VPC endpoint policy can deny it. Confirm with the policy simulator and by checking whether the same call succeeds from a principal outside the affected OU.

The evidence lives in CloudTrail, and the fields that matter are always the same four: eventName, errorCode, errorMessage, and userIdentity. Read them in that order — the event name tells you which authentication path was attempted, the error code and message usually name the failure outright, and userIdentity tells you whether a principal was established at all.

Two details worth knowing. For a failed AssumeRoleWithSAML or AssumeRoleWithWebIdentity, there is no assumed-role identity yet, so userIdentity is thin — do not expect the role ARN there. And for authorization failures the error often carries an encoded authorization message; aws sts decode-authorization-message turns it into the policy context AWS actually evaluated, which is the fastest route from "AccessDenied" to the responsible statement.

Symptom → likely cause → where to look first
Symptom	Most likely cause	First place to look
Every federated user, all at once, since this morning	IdP signing certificate rotated or expired; metadata not re-uploaded	The IAM SAML provider (or Identity Center external IdP) metadata, and the IdP certificate expiry date
One user, others fine	Group membership, SCIM provisioning, or a missing account assignment	The IdP group, then Identity Center users/groups, then the assignment list
One new permission, nobody can use it	Permission set edited but not re-provisioned	Identity Center → the permission set’s provisioning status
A third-party integration stopped working	Missing or changed `sts:ExternalId`, or an `aws:SourceArn` pinned to a recreated resource	The role trust policy conditions, against the CloudTrail request parameters
A CI pipeline fails after a repository or branch rename	The OIDC trust policy pins the `sub` claim to the old name	The role trust policy conditions, against the token claims
Intermittent failures from one subnet or host	Clock skew, or a NAT/proxy changing the apparent source	Host time sync, then any `aws:SourceIp` conditions
An app using AD Connector fails when the VPN flaps	AD Connector proxies to on-premises AD — no link, no authentication	The Direct Connect or VPN path, and the connector’s subnets and security groups
Works in the console, fails in the CLI	An MFA condition: the console session carries MFA context, the access-key CLI request does not	The MFA condition and whether the CLI is using `GetSessionToken` with an MFA code

Key terms: AssumeRole, Role trust policy, Implicit deny, SCIM provisioning

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.