Us-east-2.compute.amazonaws.coms Password: Permission Denied

TL;DR:

iam:PassRole is an AWS permission that enables disquisitional privilege escalation; many supposedly depression-privilege identities tend to have it
It's hard to tell which IAM users and roles need the permission
We have mapped out a listing of AWS actions where it is likely that iam:PassRole is required and the names of parameters that pass roles
We explain how to monitor your CloudTrail to determine which identities actually need iam:PassRole, and for which roles and services, to help yous enforce "use it or lose it" least-privilege

Why should we care nigh PassRole?

By at present we all know how important IAM governance is. IAM acts both as the perimeter of a cloud deployment and as one of the main mitigators of impairment when a node in the cloud deployment becomes (inevitably) jeopardized. Unfortunately, fifty-fifty in some of the all-time-built cloud environments, betwixt all the frequently-rotated credentials, granular-permission policies and requirements for MFA protected access, one little permission with massive hidden privileges tends to get stuffed -- like cramming a closet full in an otherwise tidy room: iam:PassRole. If this scenario resonates with you in any way: A. I feel your pain, B. It's time for some cloud security KonMari.

The AWS Security community is painfully aware of this problem, and we can exist sure attackers are (gleefully) enlightened of it likewise.

Passrole*

— Corcoran Smith (@irlcorcoran) Dec 10, 2020

What is iam:PassRole?

The bones thought of iam:PassRole is simple: whenever a master (which can be a user or a office, a human, code or a service) uses a service that needs to perform other actions, the AWS architecture often has that service assume an AWS role to perform the actions. When that happens, the service performing the actions is "passed" a role past the calling principal and implicitly (without performing sts:AssumeRole) assumes that role to perform the actions. The privileges associated with the part are dissimilar from -- and tin exist greater than -- those of the master calling the activeness.

Perhaps the most well-known example of this is when launching an EC2 instance with a certain IAM Instance profile. The instance profile is resolved to an IAM part whose permissions determine what the instance can and can't do. Whenever behavior similar this happens, AWS checks, backside the scenes, if the calling primary has the permission iam:PassRole to pass the role to the service. If it does, and if the role is unrestricted by IAM policies, the attacker is able to grant elevated permissions and escalated privilege to the EC2 instance. This scenario and others are described in a post past Rhino Security Labs. Oftentimes, in such cases, the attacker's ability to escalate privilege is just bounded by the highest-privilege role they can pass. You know all those unused loftier-privileged roles and identities you may have lying around (and unfortunately, the majority of customer environments we encounter take a ton of these)? iam:PassRole is their opportunity to "smoothen," as they are now enablers of unfettered privilege escalation.

The fact that iam:PassRole is both a facilitator of critical privilege escalation and a permission for which information technology is remarkably difficult to monitor, command and create policies presents a trouble for security teams.

Monitoring PassRole to achieve least-privilege

At Ermetic, we ran into a problem trying to audit PassRole permissions. By nature, PassRole's privilege escalation vector makes the question of which roles an identity is immune to pass merely every bit important equally the question of if the identity should have permission to pass any role at all.

We needed an effective mode, with resource level permission granularity, to know where iam:PassRole is being "used," and exactly with which roles.

To determine which PassRole permissions are actually being used, we decided to leverage CloudTrail and other activity logs. Close examination of the logs helps us understand the overprivileged policies in place and strip them down to least-privilege without breaking usability.

Then, for case, with a policy similar this:

{     "Version": "2012-10-17",     "Argument": [         {             "Upshot": "Allow",             "Action": [                 "ec2:DescribeImages",                 "ec2:DescribeSubnets",                 "ec2:RequestSpotInstances",                 "ec2:TerminateInstances",                 "ec2:DescribeInstanceStatus",                 "ec2:CreateTags",                 "ec2:RunInstances"             ],             "Resource": "*"         },         {             "Effect": "Let",             "Action": "iam:PassRole",             "Resource": "*",             "Condition": {                 "StringEquals": {                     "iam:PassedToService": [                         "ec2.amazonaws.com"                     ]                 }             }         }     ] }

… we look at the CloudTrail logs and discover events that have logged deportment for which PassRole was required. If, for example, we find that the simply function is role/EC2EmptyRole, nosotros narrow the policy down to something like:

{     "Version": "2012-10-17",     "Statement": [         {             "Issue": "Allow",             "Action": [                 "ec2:DescribeImages",                 "ec2:DescribeSubnets",                 "ec2:RequestSpotInstances",                 "ec2:TerminateInstances",                 "ec2:DescribeInstanceStatus",                 "ec2:CreateTags",                 "ec2:RunInstances"             ],             "Resource": "*"         },         {             "Event": "Allow",             "Activity": "iam:PassRole",             "Resource":  "arn:aws:iam::123456789012:function/EC2EmptyRole",             "Condition": {                 "StringEquals": {                     "iam:PassedToService": [                         "ec2.amazonaws.com"                     ]                 }             }         }     ] }

And, voila! We accept a much safer policy that doesn't let user privilege escalation to the highest privilege case contour.

PassRole is a pain to audit

Let'south drill down into what makes PassRole so difficult to audit.

Not logged

The first problem is that because iam:PassRole is only a permission and non an API activity, "uses" of PassRole (or, rather, checks of the permission iam:PassRole) are by definition not logged to CloudTrail. In fact, even Access Advisor doesn't business relationship for iam:PassRole. That is, if y'all take an identity that really uses its PassRole permission all the time but doesn't do annihilation else on IAM, Access Advisor will betoken that you haven't used your IAM permissions. This could crusade you lot to remove PassRole and negatively impact usability. Try this yourselves: Using the AWS CLI, create a lambda that uses an existing role (using an identity that hasn't used IAM) and check out Access Advisor for that identity:

Insufficiently documented

Every bit mentioned in the contempo article past Dustin Whited of ScaleSec, deportment which are dependent on iam:PassRole are, ostensibly, documented in the AWS Actions, Resource, and Status Keys reference documents. Unfortunately, this documentation is highly insufficient. We sifted through the docs looking for deportment dependent on iam:PassRole and found reference to only 58 actions in 14 services. It was immediately apparent that even some of the most infamously problematic actions that depend on PassRole were absent-minded, for example: Lambda's CreateFunction/UpdateFunction and EC2's RunInstances. By the finish of our own PassRole investigation, nosotros found more than 300 deportment in more than 90 services dependent on iam:PassRole: roughly vi times the amount documented.

Hard to pin downwardly

Since iam:PassRole is not explicitly logged or comprehensively documented, we have to notice it between the lines and spaces, using the behavior, documentation and parameters of other API deportment to identify the tracks of a part being passed and infer that the iam:PassRole permission is probable to be required for that function.

Lack of standardization

Most IAM deportment are logged to CloudTrail. In the CloudTrail log, the parameters of the API action are (usually) logged under a key called "requestParameters". These asking parameters are our master source of data for inferring passed roles. Since iam:PassRole is not logged to CloudTrail, if we want to audit pass-role at resources-level granularity (and we practice!), nosotros take to deduce the part that iam:PassRole passes from each consequence's request parameters. This is no cake walk, as parameters for PassRole vary wildly. Some are relatively straightforward, some are more complex.

Here's a relatively simple example:

{   "awsRegion": "us-east-1",   "eventID": "57a1eefb-2c4d-4881-99e3-9a137e5db024",   "eventName": "CreateFunction20150331",   "eventSource": "lambda.amazonaws.com",   "eventTime": "2021-01-05T14:53:49Z",   "eventType": "AwsApiCall",   [... User identity information, source IP, etc ...]   "requestParameters": {     "functionName": "my-office",     "runtime": "nodejs12.10",     "role": "arn:aws:iam::123456789012:role/lambda-ex",     "handler": "index.handler",     "code": {},     "publish": false,     "environs": {}   },   [... Response elements and some more event data ...] }

Fake Positives

Of course, fifty-fifty where roles are mentioned in the request parameters, iam:PassRole is not necessarily involved. If we're not conscientious, this could lead to actions being mistakenly identified as requiring iam:PassRole. For instance: iam:GetRole, which requires a role every bit the target of the action and lists that role in its "requestParameters" field, does not require the PassRole permission.

Parameters can appear in the request without being logged by CloudTrail

CloudTrail events aren't exact representations of the actions they log; sometimes sure request parameters just aren't logged for certain actions. Have a look at this CloudTrail issue for elasticbeanstalk:AssociateEnvironmentOperationsRole (some details redacted):

{     "eventVersion": "one.05",     "userIdentity": {         "type": "IAMUser",         "principalId": <principalId>,         "arn": "arn:aws:iam::123456789012:user/<user>",         "accountId": "123456789012",         "accessKeyId": <accessKeyId>,         "userName": <user>     },     "eventTime": "2020-12-14T14:39:53Z",     "eventSource": "elasticbeanstalk.amazonaws.com",     "eventName": "AssociateEnvironmentOperationsRole",     "awsRegion": "united states of america-east-one",     "sourceIPAddress": "30.30.xxx.30",     "userAgent": "aws-cli/1.18.185 Python/3.nine.0+ Linux/4.4.0-19041-Microsoft botocore/one.19.25",     "requestParameters": zero,     "responseElements": null,     "requestID": <GUID>,     "eventID": <GUID>,     "eventType": "AwsApiCall",     "recipientAccountId": "123456789012" }

The action asking includes the parameters "EnvironmentName" and "OperationsRole" only Cloud Trail has not logged them. Information technology'southward therefore incommunicable to tell which role was passed from the CloudTrail log.

The "matryoshka" problem

Some other style roles can be passed is via other parameters that contain a reference to a role within them. The nigh well-known instance of this is iamInstanceProfile, the parameter that defines the IAM role that the EC2 case will use. When it runs, the case obtains temporary security credentials that correspond to the role the instance contour contains and passes these to the application. LaunchTemplates and LaunchConfigurations are also examples of this and can contain instance profiles that define the roles these instances volition employ.

In many of these cases, iam:PassRole is required to perform the action.

We nicknamed this the matryoshka trouble, after the famous Russian nesting dolls, every bit it can lend itself to multiple layers of nesting, e.g. a role within an iamInstanceProfile within a LaunchTemplate, further obfuscating understanding permissions telescopic.

Implicitly passed roles?

In some cases in which an action is performed with a service, a service-linked office is used (and created if needed) past default -- fifty-fifty when a parameter isn't explicitly stated. We must be aware when defaults are used in lieu of an explicitly stated parameter.

AWS API vs. CloudTrail vs. IAM

To elevation it all off, we also have to solve the classic trouble of name differences across CloudTrail, the AWS API and IAM permissions. Scott Piper and others take written about this before; y'all can help past upvoting the open Github bug: ane, 2.

Characterizing the trouble

Now that nosotros've understood the problem, this is what we need to do to solve it for our employ case:

Find function parameters
Sift through them to identify which require iam:PassRole, filtering out false positives
Find all the matryoshki (nested part parameters)
Notice implicit passes of a role

Once we exercise that, nosotros are able to expose where a part is in fact passed.

Automatic approaches to mapping PassRole requirements

Since AWS has over 8500 permissions (and counting), a fully transmission approach doesn't actually scale for this problem, so we had to notice ways to automate at least part of the piece of work.

Finding PassRole Candidates

Scanning the API

Boto3 is a python AWS SDK that contains, among other things, an updated bookkeeping of the AWS API in JSON files in the botocore project. We used a python script to scan these JSON files for every action in which "function" is mentioned and found 392 API actions.

(Note: We too handled nested parameters at this phase.)

It is clear that some of the API actions do not laissez passer a role. After sifting out read-only actions, we were down to 379 actions. But nosotros must recollect the matryoshki -- the "containers," equally information technology were. Nosotros started past calculation the three nearly well-known examples: InstanceProfile, LaunchTemplate and LaunchConfiguration. (To be on the condom side, we went a step farther and added everything with the word "launch" in information technology.) We were up to 408 actions. Nosotros also took a deep dive into other AWS Compute services -- which tend to be the riskiest in terms of privilege escalation -- and found additional role containing parameters.

Scanning CloudTrail

Working from the API side is inherently dependent on the skillful graces of AWS parameter naming which, unfortunately, occasionally disappoints. To cope with the problem, we scanned our customers' CloudTrail data and looked for cases in which either a part or the containers we identified appeared within the asking parameters. This approach helped observe parameters we would not observe from the API, such as those given by "key": <key>, "value": <value> entries in parameter dictionaries, as in this example from datapipeline:PutPipelineDefintion:

"pipelineObjects": [        {            "fields": [                {                    "key": "role",                    "stringValue": "DataPipelineDefaultRole"                }            ]        }    ]

As you lot can encounter, the passed office is stored in the BSON "array of documents" way: in a lexicon within a list inside a lexicon inside a list that is the value of the parameter key "pipelineObjects".

Transmission approaches to mapping PassRole requirements

Ultimately, automated approaches can only provide hints to uses of iam:PassRole. Some decisions cannot be made without really trying to perform the deportment. Nosotros manually checked some of the actions identified by the automatic API and CloudTrail scans. Nosotros focused on actions in AWS Compute services outset as these tend to be the easiest for attackers to exploit.

Examining Special Cases

We created the IAM user Melo to serve as a principal for manually checking if PassRole is required to perform an action. These are Melo's IAM policies:

AdministratorAccess

{     "Version": "2012-ten-17",     "Argument": [         {             "Result": "Allow",             "Action": "*",             "Resource": "*"         }     ] }

MeloCantPass

{     "Version": "2012-10-17",     "Statement": [         {             "Event": "Deny",             "Activeness": "iam:PassRole",             "Resource": "*"         }     ]                                                                                                                                                                                                                                                                                           }

[Nosotros affectionately named user/Melo after future basketball hall-of-famer and New York legend Carmelo Anthony, who -- much like user/Melo -- sometimes seemed able to do anything except laissez passer.]

Note that we opted to utilise an IAM user since we were using a completely standalone account so the entity bore no special risks. Using an IAM role is preferable, especially if your sandbox account is connected to your organization in whatsoever way. If you lot exercise cull to use an IAM user, nosotros strongly suggest you to: utilize SCP guardrails, brand the API keys inactive when not in employ and rotate keys frequently.

Using Melo's credentials, we perform an action. If the action requires PassRole, we get this effect:

user@Motorcar:~$ aws autoscaling create-launch-configuration  --launch-configuration-proper name rsc-test-launch-config --iam-example-profile arn:aws:iam::123456789012:instance-profile/<Ec2RoleName>  An error occurred (AccessDenied) when calling the CreateLaunchConfiguration functioning: User: arn:aws:iam::123456789012:user/Melo is not authorized to perform: iam:PassRole on resource: arn:aws:iam::123456789012:role/<Ec2RoleName> with an explicit deny

Here, we did the manual work of checking deportment identified by our hints for potential uses of iam:PassRole, focusing on AWS Compute services first to be most granular where information technology matters most.

Which types of actions require PassRole?

The full general rule is that PassRole is required during setup and creation of resources, every bit the AWS user manual states:

[...] You only take to pass the office to the service once during set-up, and not every time that the service assumes the role.

Still, this rule doesn't utilise consistently -- sometimes PassRole could be required for two stages of the same process. One interesting example is CreateLaunchConfiguration and CreateLaunchTemplate. Yous would call up these two actions are largely similar and should behave in the same way. However, CreateLaunchTemplate is something Melo tin can practice and does not require PassRole, whereas CreateLaunchConfiguration gives an AccessDenied mistake message!

Another instance can be institute in ECS. If you want to create and run a task, ecs:RegisterTaskDefinition and ecs:RunTask both require iam:PassRole so, in the chain of creating and running a task, you really may need to have the PassRole permission for ii actions.

Service-linked roles requiring iam:PassRole

The AWS user transmission has this to say:

When you create a service-linked function, you must also have permission to pass that role to the service. Some services automatically create a service-linked role in your account when you lot perform an action in that service. For example, Amazon EC2 Auto Scaling creates the AWSServiceRoleForAutoScaling service-linked office for you the start time that you create an Motorcar Scaling group. If you try to create an Auto Scaling grouping without the PassRole permission, you receive an mistake.

This is not a very fun paragraph to read. Implicit requirements for iam:PassRole in 61 different services? Sounds like a lot of trouble. However, nosotros sent Melo to check a few sample services (EC2 Car Scaling, for ane) and found no issue at all. It seems that the association of the AWS service role does not check PassRole. You tin try this! Create a role with Melo'southward permissions and endeavor to apply 1 of the 61 AWS services that support a service-linked function. If you discover an instance where an action fails, we'd love to hear from yous.

[cloudshell-user@ip-xxxxx ~]$ aws autoscaling create-car-scaling-group --auto-scaling-group-name <asg-name> --launch-template "LaunchTemplateName=<lt-name>,Version=1" --min-size 1 --max-size 3 --vpc-zone-identifier "<subnet-i>,<subnet-ii>,<subnet-3>" [cloudshell-user@ip-xxxxxx ~]$

[CreateAutoScalingGroup did not fail, fifty-fifty when PassRole was denied and AWSServiceRoleForAutoScaling did not exist.]

Whatever its cause, this discrepancy does brand auditing PassRole and configuring IAM policies easier.

Conclusion

To audit IAM permissions, we searched customer CloudTrail logs and the AWS API for all references to role. We filtered the identified actions for non-read-only actions, identified function-containing parameters, iterated the procedure with the role-containing parameters and then checked actions manually in high-take chances services for requiring PassRole. Note that for actions non checked manually, we erred on the side of over-permission when compiling this full general-employ listing. Nosotros suggest that you lot use this list not equally a be-all-end-all solution, rather, as a starting indicate for exploring PassRole least-privilege.

The PassRole permission requirement is ubiquitous. In our listing, it extends to more than than 300 actions in more than xc services and, in some cases, is obscured by parameters that indirectly incorporate the role being passed. Comprehensively auditing it tin be a handful but is absolutely worth information technology as not doing so means leaving relatively piece of cake avenues for privilege escalation wide open. If yous're constructing your deject deployment, endeavour to resist the pressure to facilitate usability at all costs: avoid assuasive PassRole on resource "*".

We're releasing our best-judge, true-as-of-today list of PassRole API actions. You tin use this listing to determine if an identity really needs iam:PassRole and what roles it needs it for, every bit follows:

Wait for CloudTrail events corresponding to an PassRole-requiring action performed by the identity in question
Find the parameter that describes the role that was passed
This identity might indeed need iam:PassRole for that function and/or service -- or others like information technology

After you've affirmed what permissions are really needed, remove whatever that aren't.

We offering this as a broad starting signal to addressing the trouble, with a focus on AWS Compute services. Everyone knows different services well -- and a community attempt can get a long way -- so delight be sure to add your updates and corrections to the GitHub gist.

Thoughts? Contact me!
Twitter: @NoamDahan

—

Noam Dahan is a Senior Security Researcher at Ermetic.

veatchwispond.blogspot.com

Source: https://ermetic.com/blog/aws/auditing-passrole-a-problematic-privilege-escalation-permission/