downloader

This module provides utilities to download AWS resource data from AWS API.

aws_resource_search.downloader.T_RESULT_DATA

Type hint for boto3 API result data. Each one represents a single AWS resource.

alias of Union[sayt.T_DOCUMENT, str]

class aws_resource_search.downloader.ResourceIterproxy(iterable: Iterable)

Advanced iterator object for AWS resource data in boto3 API response.

Ref: https://github.com/MacHu-GWU/iterproxy-project

class aws_resource_search.downloader.ResultPath(path: str)

Defines how to extract list of AWS resource data from boto3 API call response.

For example, the s3_client.list_buckets API call returns the following response:

{
    'Buckets': [
        {
            'Name': 'string',
            'CreationDate': datetime(2015, 1, 1)
        },
    ],
    'Owner': {
        'DisplayName': 'string',
        'ID': 'string'
    }
}

We aim to extract the list of S3 bucket data from the Buckets field of the response. Similarly, for EC2 Instance, the result path for ec2_client.describe_instances <https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2/client/describe_instances.html>`_ API call response is ``Reservations[].Instances[].

Parameters:

• path – the jmespath notation to the result path. It will return an empty list if the result path doesn’t exist in the response.

• _compiled – the compiled jmespath expression. This class will be created only once for each AWS resource type, so that we should cache it for better performance.

extract(response: dict) → Iterator[Union[sayt.T_DOCUMENT, str]]

Extract list of AWS resource data from boto3 API call response.

Parameters:

response – original boto3 API response

Returns:

for example, for s3_client.list_buckets, it will return:

[
    {
        'Name': 'string',
        'CreationDate': datetime(2015, 1, 1)
    },
    ...
]

aws_resource_search.downloader.list_resources(bsm: BotoSesManager, service: str, method: str, is_paginator: bool, boto_kwargs: Optional[dict], result_path: ResultPath) → ResourceIterproxy

Call boto3 API to list AWS resources.

Example:

>>> for iam_group_data in list_resources(
...     bsm=bsm,
...     service="iam",
...     method="list_groups",
...     is_paginator=True,
...     boto_kwargs=dict(
...         PaginationConfig=dict(
...             MaxItems=9999,
...             PageSize=1000,
...         )
...     ),
...     result_path=ResultPath(path="Groups"),
... ):
...     print(iam_group_data)

Parameters:

• bsm – the boto_session_manager.BotoSesManager object.

• service – the AWS service name for creating the boto3 client. for example, the AWS S3 service name is s3.

• method – the boto3 client API method to call for listing AWS resources. for example, we use list_buckets method for getting AWS S3 buckets, we use describe_instances method for getting AWS EC2 instances.

• is_paginator – boolean value to indicate whether the method is a paginator. for example, it is False for s3.list_buckets method, it is True for ec2.describe_instances method.

• boto_kwargs – the keyword arguments for the boto3 client API call. if it is a paginator, it often contains PaginationConfig key.

• result_path – the ResultPath object to extract list of AWS resource

aws_resource_search.downloader.extract_tags(data: dict) → Dict[str, str]

Extract tags key value pair from boto3 API call response data.

Parameters:

data – it is the dictionary representation of one AWS resource, it could be the original boto3 API response, it also could be a nested dictionary. For example: in s3.list_buckets response, the original response is the data, because it has a TagSet field. in iam.list_roles response, the dict in the Roles list is the data.