网站部署到终端机怎么做,公司名称被大量网站收录,免费视频模板网站,wordpress英文字体样式文章目录 aws(学习笔记第四十四课) opensearch学习内容#xff1a;1. 整体架构1.1 代码链接1.2 整体架构 2. 什么是opensearch2.1 elastic search的aws实现2.2 elastic search的数据源2.3 开始学习elastic search 3. opensearch的cdk代码架构3.1 cdk代码的整体架构3.2 代码详细… 文章目录 aws(学习笔记第四十四课) opensearch学习内容1. 整体架构1.1 代码链接1.2 整体架构 2. 什么是opensearch2.1 elastic search的aws实现2.2 elastic search的数据源2.3 开始学习elastic search 3. opensearch的cdk代码架构3.1 cdk代码的整体架构3.2 代码详细3.2.1 创建对opensearch的访问限制3.2.2 创建对opensearch的访问密码3.2.3 创建对opensearch的capacity config3.2.4 创建对opensearch的ebs config3.2.5 设定opensearch的enable zone awareness3.2.6 设定encrypt setting和用户名密码3.2.7 开始真正创建opensearch3.2.8 将必要的output进行输出 4 执行cdk创建opensearch4.1 执行cdk的注意点4.2 执行cdk的错误回避对策(workaround)4.3 创建opensearch的花费 aws(学习笔记第四十四课) opensearch 在aws上部署opensearch
学习内容
部署opensearch并了解什么是opensearch 1. 整体架构
1.1 代码链接
代码连接(opensearch-simple-domain)
1.2 整体架构 这里
会构建一个aws的opensearch服务。自动生成管理员的密码并将管理员密码报错在aws secrets manager上。这里设定允许访问的source ip adress。
2. 什么是opensearch
2.1 elastic search的aws实现
Elasticsearch 是一个强大的分布式搜索和分析引擎基于 Apache Lucene 构建广泛应用于全文搜索、日志分析、实时数据处理等场景。以下是它的 主要功能
全文搜索实时搜索与分析分布式架构与高可用性日志与监控分析结构化 非结构化数据处理机器学习与 AI 增强安全与权限管理
2.2 elastic search的数据源
Elasticsearch 可以与多种类型的数据源集成支持从不同系统中导入数据并进行索引和搜索。以下是 Elasticsearch 主要支持的数据源类型及其集成方式
关系型数据库MySQL、PostgreSQL、Oracle 等NoSQL 数据库MongoDB、Cassandra 等日志系统Logstash、Filebeat、Fluentd消息队列Kafka、RabbitMQ文件系统CSV、JSON、日志文件云服务AWS S3、Google Cloud Storage大数据组件Hadoop、Spark、FlinkAPI 数据RESTful 服务、爬虫数据
2.3 开始学习elastic search
elastic search get start
3. opensearch的cdk代码架构
3.1 cdk代码的整体架构 3.2 代码详细
3.2.1 创建对opensearch的访问限制
这里首先执行opensearch的版本选择最新2.19。 OPENSEARCH_VERSION 2.19# Add the authorized IP addresses (using CIDR format) that should# be granted access to the OpenSearch Domain.# Create an environment variable before running cdk deploy. E.g.:OPENSEARCH_ALLOWED_IP[33.45.123.8/32]# allowed_ip_addresses os.environ.get(OPENSEARCH_ALLOWED_IP, x.x.x.x/32)allowed_ip_addresses OPENSEARCH_ALLOWED_IP# Creating OpenSearch access policy to restrict# access to a specific list of IPs. We are allowing all# types of HTTP commands.opensearch_access_policy cdk_iam.PolicyStatement(effectcdk_iam.Effect.ALLOW,principals[cdk_iam.AnyPrincipal()],actions[es:ESHttp*],resources[],conditions{IpAddress: {aws:SourceIp: allowed_ip_addresses}})3.2.2 创建对opensearch的访问密码 # Generating a secret and storing it with AWS Secrets Manager.# https://aws.amazon.com/secrets-manager/# To list secret using CLI and jq, run:# aws secretsmanager list-secrets | jq .SecretList[].Name# To retrieve a secret value using CLI and jq, run:# aws secretsmanager get-secret-value --secret-id secret-namesecret_opensearch_admin_password cdk_sm.Secret(self, OpenSearchDemoDomainAdminUser)3.2.3 创建对opensearch的capacity config
# Capacity config documentation:# https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.aws_opensearchservice/CapacityConfig.html#aws_cdk.aws_opensearchservice.CapacityConfig# Available instance types:# https://docs.aws.amazon.com/opensearch-service/latest/developerguide/supported-instance-types.htmlcapacity_config cdk_opensearch.CapacityConfig(master_nodes3,master_node_instance_typet3.small.search,data_nodes3,data_node_instance_typet3.medium.search 3.2.4 创建对opensearch的ebs config # Available EBS options# https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.aws_opensearchservice/EbsOptions.html#aws_cdk.aws_opensearchservice.EbsOptionsebs_config EbsOptions(volume_size10,volume_typecdk_ec2.EbsDeviceVolumeType.GP3)3.2.5 设定opensearch的enable zone awareness # Enabling zone awareness to allow data replication across AZs.# https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.aws_opensearchservice/ZoneAwarenessConfig.html#aws_cdk.aws_opensearchservice.ZoneAwarenessConfigzone_awareness_config ZoneAwarenessConfig(availability_zone_count3,enabledTrue)3.2.6 设定encrypt setting和用户名密码 # Required when FGAC is enabledencryption_config EncryptionAtRestOptions(enabledTrue)# Required when FGAC is enabledopensearch_admin_user admin-useradvanced_security_config AdvancedSecurityOptions(master_user_nameopensearch_admin_user,master_user_passwordsecret_opensearch_admin_password.secret_value) 3.2.7 开始真正创建opensearch # Required when FGAC is enabledencryption_config EncryptionAtRestOptions(enabledTrue)# Required when FGAC is enabledopensearch_admin_user admin-useradvanced_security_config AdvancedSecurityOptions(master_user_nameopensearch_admin_user,master_user_passwordsecret_opensearch_admin_password.secret_value) 3.2.8 将必要的output进行输出 cdk.CfnOutput(self,OpenSearchDomainEndpoint, valueaos_domain.domain_endpoint)cdk.CfnOutput(self,OpenSearchDashboardsURL, value(aos_domain.domain_endpoint /_dashboards))cdk.CfnOutput(self,OpenSearchPasswordSecretName, valuesecret_opensearch_admin_password.secret_name)cdk.CfnOutput(self,OpenSearchAdminUser, valueopensearch_admin_user)4 执行cdk创建opensearch
4.1 执行cdk的注意点
这里如果直接执行如下命令创建opensearch会报错。
cdk --require-approval never deploy错误信息如下
OpensearchSimpleDomainStack | 4/10 | 16:31:40 | CREATE_IN_PROGRESS | AWS::Lambda::Function | AWS679f53fac002430cb0da5b7982bd2287 (AWS679f53fac002430cb0da5b7982bd22872D164C4C) Resource creation Initiated
OpensearchSimpleDomainStack | 4/10 | 16:31:40 | CREATE_FAILED | AWS::Lambda::Function | AWS679f53fac002430cb0da5b7982bd2287 (AWS679f53fac002430cb0da5b7982bd22872D164C4C) Resource handler returned message: The runtime parameter of nodejs14.x is no longer supported for creating or updating AWS Lambda functions. We recommend you use a supported runtime while creating or updating functions. (Service: Lambda, Status Code: 400, Request ID: c49cc9d9-f4b1-42e4-8a98-af597a4aa3f4) (SDK Attempt Count: 1) (RequestToken: 07111475-e581-91e9-185c-ce93ab079201, HandlerErrorCode: InvalidRequest)
OpensearchSimpleDomainStack | 4/10 | 16:31:40 | CREATE_IN_PROGRESS | AWS::Lambda::Function | DummyLambdaRuntimeSetter (DummyLambdaRuntimeSetter4B38A37F) Resource creation Initiated
OpensearchSimpleDomainStack | 4/10 | 16:31:40 | CREATE_FAILED | AWS::Lambda::Function | DummyLambdaRuntimeSetter (DummyLambdaRuntimeSetter4B38A37F) Resource creation cancelled
OpensearchSimpleDomainStack | 4/10 | 16:31:40 | CREATE_FAILED | AWS::OpenSearchService::Domain | OpensearchDemoDomain (OpensearchDemoDomainBEE1301C) Resource creation cancelled
原因是默认opensearch使用lambda是采用nodejs14.x runtime所以会报错。
4.2 执行cdk的错误回避对策(workaround)
cdk synth template.yaml # 首先不直接部署cdk而是将cdk输出到template.yaml
sed -i s/nodejs14.x/nodejs18.x/g template.yaml # 进行替换升级到nodejs18
aws cloudformation deploy --template-file template.yaml --stack-name OpensearchSimpleDomainStack --capabilities CAPABILITY_IAM # 继续部署opensearch4.3 创建opensearch的花费
创建了五六次opensearch花费了$8,所以建议大家慎重测试opensearch
