Maintainer Guide for Authoring#

This document is designed for myself to remember how this software designed.

前言#

一般有两种描述一个系统的内部设计的方式:

  1. 从底层模块出发, 一点点的构建起系统的各个模块, 并最后串联成一个系统. 这是一种在设计一个系统时用到的思维.

  2. 从最终的用户界面触发, 一点点往下深挖, 看实际底层按照顺序调用了哪些模块. 这是一种在理解一个系统时用到的思维.

这篇文档的主要目的是方便我自己在过了一段时间后回来维护这个项目时理解这个系统用的, 所以我会按照 #2 的思路来写这篇文档.

CLI 命令行初探#

当你在命令行输入 ars 的时候, 就会调用 aws_resource_search.cli.main.run() 这个函数. 由于你没有输入任何 subcommand 以及参数, 所以它会调用 aws_resource_search.cli.ArsCli.__call__() 这个方法. 这两个函数都是位于 aws_resource_search.cli 这个模块中的, 是负责 CLI 的接口的用的, 并不涉及具体业务逻辑. 而你看 __call__ 这个方法就知道, 里面会调用 aws_resource_search.ui_init.run_ui() 这个函数. 这个函数是位于 aws_resource_search.ui_init 下的, 属于 UI 的核心逻辑的入口函数. 而 CLI 函数只是对这个函数的初探.

UI Event Loop#

UI 的入口函数 aws_resource_search.ui_init.run_ui() 里面的内容和恩简单, 就是实例化一个 aws_resource_search.ui_def.UI 对象, 然后进入 event loop. 对于 UI 来说, 你每按下一个按键, 都会调用一个函数来处理你的输入, 然后重新渲染整个界面. 这个用于处理输入的函数就是 aws_resource_search.ui_def.handler()

UI Handler#

UI 的函数是用来处理

如何支持更多的 AWS 服务和资源#

本节主要介绍如果你发现这个项目不支持你想要的 AWS 服务或者资源, 你应该如何去添加它.

  1. 首先到 aws_resource_search/code/searcher_enum.json 仿照已经支持的 AWS Resource, 添加你要支持的 AWS Resource 的类型. 其中 description 是给人类看的一句话介绍, 一般是 AWS Document 官网首页的第一句话. 而 ngram 则是额外的用于搜索 ngram 搜索的关键字, 你可以把人类在想搜这个资源时能联想到的各种词汇的全称和缩写都放在这里.

aws_resource_search/code/searcher_enum.json
  1{
  2    "cloudformation-stack": {
  3        "desc": "A stack is a collection of AWS resources that you can manage as a single unit.",
  4        "ngram": "cfn cft"
  5    },
  6    "codebuild-job-run": {
  7        "desc": "Codebuild job run, it is not a batch job run.",
  8        "ngram": "jobrun"
  9    },
 10    "codebuild-project": {
 11        "desc": "A build project includes information about how to run a build.",
 12        "ngram": ""
 13    },
 14    "codecommit-repository": {
 15        "desc": "A repository is where you store code and files for your project.",
 16        "ngram": ""
 17    },
 18    "codepipeline-pipeline": {
 19        "desc": "Code pipeline is a workflow construct that describes how software changes go through a release process.",
 20        "ngram": ""
 21    },
 22    "dynamodb-table": {
 23        "desc": "A table is a collection of data.",
 24        "ngram": ""
 25    },
 26    "ec2-instance": {
 27        "desc": "An EC2 instance is simply a virtual server in AWS",
 28        "ngram": "aws elastic compute cloud"
 29    },
 30    "ec2-security-group": {
 31        "desc": "A security group acts as a firewall that controls the traffic allowed to and from the resources in your VPC.",
 32        "ngram": "sg securitygroup"
 33    },
 34    "ec2-subnet": {
 35        "desc": "A subnet is a range of IP addresses in your VPC.",
 36        "ngram": ""
 37    },
 38    "ec2-vpc": {
 39        "desc": "A virtual private cloud (VPC) is a virtual network dedicated to your AWS account.",
 40        "ngram": "virtual private cloud"
 41    },
 42    "ecr-repository": {
 43        "desc": "AWS managed container image registry service that is secure, scalable, and reliable.",
 44        "ngram": "ecr private repository container"
 45    },
 46    "ecr-repository-image": {
 47        "desc": "An container image in ECR repository.",
 48        "ngram": "ecr private repository container"
 49    },
 50    "ecs-cluster": {
 51        "desc": "An Amazon ECS cluster is a logical grouping of tasks or services.",
 52        "ngram": ""
 53    },
 54    "ecs-task-run": {
 55        "desc": "A task run is the instantiation of a task definition within a cluster.",
 56        "ngram": ""
 57    },
 58    "ecs_task_definition_family": {
 59        "desc": "A name of the task definition, without revision id.",
 60        "ngram": ""
 61    },
 62    "glue-crawler": {
 63        "desc": "You can use a crawler to populate the AWS Glue Data Catalog with tables.",
 64        "ngram": ""
 65    },
 66    "glue-database": {
 67        "desc": "Databases are used to organize metadata tables in the AWS Glue. ",
 68        "ngram": "db"
 69    },
 70    "glue-database-table": {
 71        "desc": "The metadata definition that represents your data.",
 72        "ngram": "db tb"
 73    },
 74    "glue-job": {
 75        "desc": "The business logic that is required to perform ETL work.",
 76        "ngram": ""
 77    },
 78    "glue-job-run": {
 79        "desc": "A job run is the execution of an ETL job.",
 80        "ngram": ""
 81    },
 82    "iam-group": {
 83        "desc": "An IAM user group is a collection of IAM users.",
 84        "ngram": ""
 85    },
 86    "iam-policy": {
 87        "desc": "IAM policies define permissions for an action regardless of the method that you use to perform the operation.",
 88        "ngram": ""
 89    },
 90    "iam-role": {
 91        "desc": "An IAM role is an IAM identity that you can create in your account that has specific permissions.",
 92        "ngram": ""
 93    },
 94    "iam-user": {
 95        "desc": "IAM user is an entity that you create in AWS.",
 96        "ngram": ""
 97    },
 98    "kms-key-alias": {
 99        "desc": "A human friendly name for KMS keys.",
100        "ngram": "key management service"
101    },
102    "lambda-function": {
103        "desc": "A function is a resource that you can invoke to run your code in Lambda.",
104        "ngram": "lbd"
105    },
106    "lambda-function-alias": {
107        "desc": "A Lambda alias is a pointer to a function version that you can update.",
108        "ngram": "lbd"
109    },
110    "lambda-layer": {
111        "desc": "A Lambda layer is a .zip file archive that can contain additional code or other content.",
112        "ngram": ""
113    },
114    "rds-db-cluster": {
115        "desc": "A DB cluster deployment is a semi-synchronous, high availability deployment mode of Amazon RDS with two readable standby DB instances.",
116        "ngram": "relational database service"
117    },
118    "rds-db-instance": {
119        "desc": "A DB instance is an isolated database environment running in the cloud.",
120        "ngram": "relational database service"
121    },
122    "s3-bucket": {
123        "desc": "A bucket is a container for objects.",
124        "ngram": "simple storage service"
125    },
126    "secretsmanager-secret": {
127        "desc": "A secret consists of secret information, the secret value, plus metadata about the secret.",
128        "ngram": "sm"
129    },
130    "sfn-state-machine": {
131        "desc": "A series of event-driven steps",
132        "ngram": "step functions stepfunctions"
133    },
134    "sfn-state-machine-execution": {
135        "desc": "A execution of a state machine.",
136        "ngram": "step functions stepfunctions"
137    },
138    "sns-topic": {
139        "desc": "An Amazon SNS topic is a logical access point that acts as a communication channel.",
140        "ngram": "simple notification service"
141    },
142    "sqs-queue": {
143        "desc": "A form of asynchronous service-to-service communication used in serverless and microservices architectures",
144        "ngram": "simple queue service"
145    },
146    "ssm-parameter": {
147        "desc": "Provides secure, hierarchical storage for configuration data management and secrets management. ",
148        "ngram": "system manager parameter store"
149    }
150}

  1. 然后到 aws_resource_search/res/ 下, 找一个跟你要支持的服务比较相近的服务作为模版, copy paste 创建一个新的模块. 模块的名字要跟 AWS Service 对应上. 然后参考其他的模块实现这个搜索器.

  2. 运行 scripts/code_work.py, 自动更新其他的 enum 模块, 数据, 和代码.

  3. 如果你这个 resource 是一个先要搜索 parent resource, 然后才能搜的 sub resource, 你还要到 aws_resource_search/ui/search_patterns.py 模块中更新 ArsSearchPatternsMixin.get_search_patterns 这个函数中定义的映射关系.

aws_resource_search/ars_search_patterns.py
  1# -*- coding: utf-8 -*-
  2
  3"""
  4This module defines the search patterns for those resource types that requires
  5special handling.
  6"""
  7
  8import typing as T
  9
 10from .compat import TypedDict, cached_property
 11from .searcher_enum import SearcherEnum
 12
 13if T.TYPE_CHECKING:
 14    from .ars_def import ARS
 15
 16
 17class T_SEARCH_PATTERN(TypedDict):
 18    partitioner_resource_type: str
 19    get_boto_kwargs: T.Callable
 20
 21
 22K_PARTITIONER_RESOURCE_TYPE = "partitioner_resource_type"
 23K_GET_BOTO_KWARGS = "get_boto_kwargs"
 24
 25
 26class ArsSearchPatternsMixin:
 27    """
 28    todo: docstring
 29    """
 30    def get_search_patterns(self: "ARS"):
 31        """
 32        This variable defines those resource types that requires a parent resource name
 33        for the boto3 API call. For example:
 34
 35        - in order to search glue table, you need to specify glue database
 36        - in order to search glue job run, you need to specify glue job
 37        """
 38        return {
 39            SearcherEnum.ecr_repository_image.value: {
 40                K_PARTITIONER_RESOURCE_TYPE: SearcherEnum.ecr_repository.value,
 41                K_GET_BOTO_KWARGS: lambda partitioner_query: {
 42                    "repositoryName": partitioner_query
 43                },
 44            },
 45            SearcherEnum.ecs_task_run.value: {
 46                K_PARTITIONER_RESOURCE_TYPE: SearcherEnum.ecs_cluster.value,
 47                K_GET_BOTO_KWARGS: lambda partitioner_query: {
 48                    "cluster": partitioner_query
 49                },
 50            },
 51            SearcherEnum.glue_database_table.value: {
 52                K_PARTITIONER_RESOURCE_TYPE: SearcherEnum.glue_database.value,
 53                K_GET_BOTO_KWARGS: lambda partitioner_query: {
 54                    "DatabaseName": partitioner_query
 55                },
 56            },
 57            SearcherEnum.glue_job_run.value: {
 58                K_PARTITIONER_RESOURCE_TYPE: SearcherEnum.glue_job.value,
 59                K_GET_BOTO_KWARGS: lambda partitioner_query: {
 60                    "JobName": partitioner_query
 61                },
 62            },
 63            SearcherEnum.sfn_state_machine_execution: {
 64                K_PARTITIONER_RESOURCE_TYPE: SearcherEnum.sfn_state_machine.value,
 65                K_GET_BOTO_KWARGS: lambda partitioner_query: {
 66                    "stateMachineArn": self.aws_console.step_function.get_state_machine_arn(
 67                        partitioner_query
 68                    )
 69                },
 70            },
 71            SearcherEnum.codebuild_job_run.value: {
 72                K_PARTITIONER_RESOURCE_TYPE: SearcherEnum.codebuild_project.value,
 73                K_GET_BOTO_KWARGS: lambda partitioner_query: {
 74                    "projectName": partitioner_query
 75                },
 76            },
 77            SearcherEnum.lambda_function_alias.value: {
 78                K_PARTITIONER_RESOURCE_TYPE: SearcherEnum.lambda_function.value,
 79                K_GET_BOTO_KWARGS: lambda partitioner_query: {
 80                    "FunctionName": partitioner_query
 81                },
 82            },
 83        }
 84
 85    @cached_property
 86    def search_patterns(self: "ARS"):
 87        return self.get_search_patterns()
 88
 89    def _clear_search_patterns_cache(self: "ARS"):
 90        """
 91        Clear the :meth:`ArsSearchPatternsMixin.search_patterns` cache.
 92        """
 93        del self.search_patterns
 94
 95    def has_partitioner(
 96        self: "ARS",
 97        resource_type: str,
 98    ) -> bool:
 99        """
100        Check if a resource type need a partitioner resource.
101        """
102        return resource_type in self.search_patterns
103
104    def get_partitioner_resource_type(
105        self: "ARS",
106        resource_type: str,
107    ) -> str:
108        """
109        Get the partitioner "resource type" of a resource type.
110        """
111        return self.search_patterns[resource_type][K_PARTITIONER_RESOURCE_TYPE]
112
113    def get_partitioner_boto_kwargs(
114        self: "ARS",
115        resource_type: str,
116        partitioner_query: str,
117    ) -> dict:
118        """
119        Get the boto3 kwargs for the partitioner resource.
120        """
121        return self.search_patterns[resource_type][K_GET_BOTO_KWARGS](partitioner_query)

  1. 为了验证你的实现是否正确, 你需要到 aws_resource_search/tests/fake_aws/ 目录下创建 mock AWS resource 的模块, 你可以在里面找和你的 Resource 类似的模块作为参考.

  2. 然后到 aws_resource_search/tests/fake_aws/main.py 模块中更新 mock_list 中你要 mock 的列表, 以及更新 def create_all(self) 中的逻辑, 把创建你刚实现的 Resource 的逻辑加进去.

  3. 然后你就可以用 pyops cov 命令, 或者手动运行 tests/test_ars_init.py 单元测试来自动 mock 所有支持的 AWS 资源并尝试进行搜索了.

What is Searcher#

我们这个 App 的核心功能就是搜索 AWS Resource. 而 AWS Resource 有很多种不同的类型, 例如 EC2 Instance, S3 Bucket, IAM Role. 搜索每种类型的资源的 API 都不一样. 而 Searcher 就是对搜索特定 AWS 资源的逻辑的一个封装. 我们有一个 Searcher Base Class, 然后让负责搜索特定 AWS 资源的 Search 继承这个 Base Class, 并且实现对应的一些方法.

Code Architecture#

Low level modules

底层模块主要是实现一些抽象的基类, 使得我们实现实体类 (就是不会再被继承的类) 的时候能更轻松.

Middle level modules:

中层模块主要是一些跟业务逻辑相关的实体类.

Per AWS Resource Type Searcher modules:

这一层主要是实现对应的 AWS Resource 的 Searcher 类. 以及把他们汇总到一个 ARS 单例对象中, 便于 import 和调用 search 的 API.

UI modules:

这一层主要是实现 UI.