怎么做代刷网站长,wordpress get_field,杭州小型网站建设服务,洛宁网站开发Day07-ES集群加密#xff0c;kibana的RBAC实战#xff0c;zookeeper集群搭建#xff0c;zookeeper基本管理及kafka单点部署实战 0、昨日内容回顾:1、基于nginx的反向代理控制访问kibana2、配置ES集群TSL认证:3、配置kibana连接ES集群4、配置filebeat连接ES集群5、配置logsta… Day07-ES集群加密kibana的RBAC实战zookeeper集群搭建zookeeper基本管理及kafka单点部署实战 0、昨日内容回顾:1、基于nginx的反向代理控制访问kibana2、配置ES集群TSL认证:3、配置kibana连接ES集群4、配置filebeat连接ES集群5、配置logstash连接ES集群6、自定义角色使用logstash组件写入数据到ES集群7、部署zookeeper单点8、zookeeper的命令行基本管理9、zookeeper集群部署10、编写zk的集群管理脚本11、使用zkWeb管理zookeeper集群12、快速搭建kafka单点环境 0、昨日内容回顾: filebeat多实例 logstash的多实例 logstash的分支语法 logstash的pipeline logstash的filter插件之mutate,useragent ELFK架构采集日志写入ES集群并使用kibana出图展示 map可视化库dashboard filebeat的模块使用
1、基于nginx的反向代理控制访问kibana
(1)部署nginx服务 略参考之前的笔记即可。
(2)编写nginx的配置文件
cat /etc/nginx/conf.d/kibana.conf EOF
server {listen 80;server_name kibana.oldboyedu.com;location / {proxy_pass http://10.0.0.103:5601$request_uri;auth_basic oldboyedu kibana web!;auth_basic_user_file conf/htpasswd;}
}
EOF(3)创建账号文件
mkdir -pv /etc/nginx/conf
yum -y install httpd-tools
htpasswd -c -b /etc/nginx/conf/htpasswd admin oldboyedu(4)启动nginx服务
nginx -t
systemctl reload nginx(5)访问nginx验证kibana访问
如下图所示。
2、配置ES集群TSL认证:
(1)elk101节点生成证书文件
cd /oldboyedu/softwares/es7/elasticsearch-7.17.5/
./bin/elasticsearch-certutil cert -out config/elastic-certificates.p12 -pass --days 3650(2)elk101节点为证书文件修改属主和属组
chown oldboyedu:oldboyedu config/elastic-certificates.p12 (3)elk101节点同步证书文件到其他节点
data_rsync.sh pwd/config/elastic-certificates.p12 (4)elk101节点修改ES集群的配置文件
vim /oldboyedu/softwares/es7/elasticsearch-7.17.5/config/elasticsearch.yml
...
cluster.name: oldboyedu-linux85-binary
path.data: /oldboyedu/data/es7
path.logs: /oldboyedu/logs/es7
network.host: 0.0.0.0
discovery.seed_hosts: [elk101.oldboyedu.com,elk102.oldboyedu.com,elk103.oldboyedu.com]
cluster.initial_master_nodes: [elk103.oldboyedu.com]
reindex.remote.whitelist: 10.0.0.*:19200
node.data: true
node.master: true# 在最后一行添加以下内容xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12(5)elk101节点同步ES配置文件到其他节点
data_rsync.sh pwd/config/elasticsearch.yml (6)所有节点重启ES集群
systemctl restart es7(7)生成随机密码
[rootelk101.oldboyedu.com elasticsearch-7.17.5]# ./bin/elasticsearch-setup-passwords auto
warning: usage of JAVA_HOME is deprecated, use ES_JAVA_HOME
Future versions of Elasticsearch will require Java 11; your Java version from [/oldboyedu/softwares/jdk1.8.0_291/jre] does not meet this requirement. Consider switching to a distribution of Elasticsearch with a bundled JDK. If you are already using a distribution with a bundled JDK, ensure the JAVA_HOME environment variable is not set.
Initiating the setup of passwords for reserved users elastic,apm_system,kibana,kibana_system,logstash_system,beats_system,remote_monitoring_user.
The passwords will be randomly generated and printed to the console.
Please confirm that you would like to continue [y/N]yChanged password for user apm_system
PASSWORD apm_system by9j4WkXTocxc7Gl7l8SChanged password for user kibana_system
PASSWORD kibana_system t0HSSsrBPACFTDxor4IxChanged password for user kibana
PASSWORD kibana t0HSSsrBPACFTDxor4IxChanged password for user logstash_system
PASSWORD logstash_system JUXrlCfaMa74seZJnhw4Changed password for user beats_system
PASSWORD beats_system 2V39PZkHNGIymaVaDFx0Changed password for user remote_monitoring_user
PASSWORD remote_monitoring_user UZplScGKm6zAmMCO9JmgChanged password for user elastic
PASSWORD elastic e31LGPoUxik7fnitQidO(8)postman访问 3、配置kibana连接ES集群 (1)修改kibana的配置文件
[rootelk103.oldboyedu.com elasticsearch-7.17.5]# yy /etc/kibana/kibana.yml
server.host: 0.0.0.0
elasticsearch.hosts: [http://10.0.0.101:9200,http://10.0.0.102:9200,http://10.0.0.103:9200]
elasticsearch.username: kibana_system
elasticsearch.password: VxFV4WjsHyxsA3CH2LQT
i18n.locale: zh-CN
[rootelk103.oldboyedu.com elasticsearch-7.17.5]# (2)重启kibana
[rootelk103.oldboyedu.com elasticsearch-7.17.5]# systemctl restart kibana(3)使用elastic用户登录并修改密码 4、配置filebeat连接ES集群
(1)修改配置文件
[rootelk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# cat config/24-log-to-es_tls.yaml
filebeat.inputs:
- type: logpaths:- /tmp/oldboyedu-linux85/test.log output.elasticsearch:hosts: [http://10.0.0.101:9200,http://10.0.0.102:9200,http://10.0.0.103:9200] username: elasticpassword: yinzhengjieindex: oldboyedu-jiaoshi07-testsetup.ilm.enabled: false
setup.template.name: oldboyedu-jiaoshi07
setup.template.pattern: oldboyedu-jiaoshi07-*
setup.template.overwrite: true
setup.template.settings:index.number_of_shards: 3index.number_of_replicas: 0(2)启动filebeat实例
[rootelk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# filebeat -e -c config/24-log-to-es_tls.yaml5、配置logstash连接ES集群
(1)编写配置文件
[rootelk101.oldboyedu.com ~]# cat config/16-file-to-es_tsl.conf
input { file {
# 指定本地文件的路径path /tmp/oldboyedu-linux85-file
# 指定读取文件的起始位置但前提是该文件之前未读取过或者未在.sincedb文件中记录。
start_position beginning}
} output { # stdout {} elasticsearch {hosts [http://localhost:9200]index oldboyedu-linux85-logstash-fileuser elasticpassword yinzhengjie}
}
[rootelk101.oldboyedu.com ~]#
[rootelk101.oldboyedu.com ~]# (2)启动logstash实例
[rootelk101.oldboyedu.com ~]# logstash -rf config/16-file-to-es_tsl.conf查看logstash采集文件的偏移量路径。
ls -la /oldboyedu/softwares/logstash-7.17.5/data/plugins/inputs/file/6、自定义角色使用logstash组件写入数据到ES集群
[rootelk101.oldboyedu.com ~]# cat config/16-file-to-es_tsl.conf
input { file {# 指定本地文件的路径path /tmp/oldboyedu-linux85-file# 指定读取文件的起始位置但前提是该文件之前未读取过或者未在.sincedb文件中记录。start_position beginning}
} output { # stdout {} elasticsearch {hosts [http://localhost:9200]#index oldboyedu-linux85-logstash-fileindex oldboyedu-linux85-logstash-demouser jiaoshi07-logstashpassword 123456}
}
[rootelk101.oldboyedu.com ~]# 7、部署zookeeper单点
(1)下载zookeeper软件
https://zookeeper.apache.org/releases.html[rootelk101.oldboyedu.com ~]# wget http://192.168.15.253/ElasticStack/day07-/softwares/apache-zookeeper-3.8.0-bin.tar.gz(2)解压软件包
[rootelk101.oldboyedu.com ~]# tar xf apache-zookeeper-3.8.0-bin.tar.gz -C /oldboyedu/softwares/(3)创建符号连接
[rootelk101.oldboyedu.com ~]# cd /oldboyedu/softwares/ ln -sv apache-zookeeper-3.8.0-bin zk(4)声明zk的环境变量
[rootelk101.oldboyedu.com softwares]# cat /etc/profile.d/kafka.sh EOF
#!/bin/bashexport ZK_HOME/oldboyedu/softwares/zk
export PATH$PATH:$ZK_HOME/bin
EOF
source /etc/profile.d/kafka.sh(5)创建zk的配置文件
[rootelk101.oldboyedu.com ~]# cp /oldboyedu/softwares/zk/conf/{zoo_sample.cfg,zoo.cfg}(6)启动zk节点
[rootelk101.oldboyedu.com ~]# zkServer.sh start
[rootelk101.oldboyedu.com ~]# zkServer.sh status # 查看zk服务的状态信息
[rootelk101.oldboyedu.com ~]# zkServer.sh stop
[rootelk101.oldboyedu.com ~]# zkServer.sh restart(7)连接ZK节点
[rootelk101.oldboyedu.com ~]# zkCli.sh 8、zookeeper的命令行基本管理
# 查看
ls / # 查看根(/)下有多少子zookeeper node简称znode。
get /oldboyedu-linux85/jiaoshi07 # 查看/oldboyedu-linux85/jiaoshi07的数据。# 创建
create /oldboyedu-linux85 # 在根路径下创建一个名为oldboyedu-linux85的znode。
create /oldboyedu-linux85/jiaoshi07 123 # 在/oldboyedu-linux85/znode下创建一个名为jiaoshi07的子znode并指定该znode数据为123。
create -s /oldboyedu-linux85/jiaoshi07/liwenxuan 88888 # 创建一个前缀为/oldboyedu-linux85/jiaoshi07/liwenxuan有序编号的znode数据为88888
create -s -e /oldboyedu-linux85/linux85/test # 创建一个名为/oldboyedu-linux85/linux85/test的临时znode。当前会话结束临时znode会自动删除。# 修改
set /oldboyedu-linux85/jiaoshi07 456 # 将/oldboyedu-linux85/jiaoshi07 的znode数据修改为456.# 删除
delete /oldboyedu-linux85/test02 # 删除名为/oldboyedu-linux85/test02的znode但该znode不能有子znode。即必须为空。
deleteall /oldboyedu-linux85/jiaoshi07 # 递归删除/oldboyedu-linux85/jiaoshi07下的所有znode。9、zookeeper集群部署
(1)创建zk的数据目录
[rootelk101.oldboyedu.com ~]# install -d /oldboyedu/data/zk(2)修改单点zk的配置文件
[rootelk101.oldboyedu.com ~]# vim /oldboyedu/softwares/zk/conf/zoo.cfg
...
# 定义最小单元的时间范围tick。
tickTime2000
# 启动时最长等待tick数量。
initLimit5
# 数据同步时最长等待的tick时间进行响应ACK
syncLimit2
# 指定数据目录
dataDir/oldboyedu/data/zk
# 监听端口
clientPort2181
# 开启四字命令允许所有的节点访问。
4lw.commands.whitelist*
# server.IDA:B:C[:D]
# ID:
# zk的唯一编号。
# A:
# zk的主机地址。
# B:
# leader的选举端口是谁leader角色就会监听该端口。
# C:
# 数据通信端口。
# D:
# 可选配置指定角色。
server.10110.0.0.101:2888:3888
server.10210.0.0.102:2888:3888
server.10310.0.0.103:2888:3888(3)同步数据即可
[rootelk101.oldboyedu.com ~]# data_rsync.sh /oldboyedu/softwares/zk/
[rootelk101.oldboyedu.com ~]# data_rsync.sh /oldboyedu/softwares/apache-zookeeper-3.8.0-bin/
[rootelk101.oldboyedu.com ~]# data_rsync.sh /oldboyedu/data/zk/
[rootelk101.oldboyedu.com ~]# data_rsync.sh /etc/profile.d/kafka.sh (4)创建myid文件
[rootelk101.oldboyedu.com ~]# for ((host_id101;host_id103;host_id)) do ssh 10.0.0.${host_id} echo ${host_id} /oldboyedu/data/zk/myid;done(5)所有节点启动zk服务
[rootelk101.oldboyedu.com ~]# zkServer.sh start[rootelk102.oldboyedu.com ~]# source /etc/profile.d/kafka.sh
[rootelk102.oldboyedu.com ~]# zkServer.sh start[rootelk103.oldboyedu.com ~]# source /etc/profile.d/kafka.sh
[rootelk103.oldboyedu.com ~]# zkServer.sh start(6)查看zk的角色状态
[rootelk101.oldboyedu.com ~]# zkServer.sh statusleader选举流程图解 10、编写zk的集群管理脚本
[rootelk101.oldboyedu.com ~]# cat /usr/local/sbin/zkManager.sh
#!/bin/bash#判断用户是否传参
if [ $# -ne 1 ];thenecho 无效参数用法为: $0 {start|stop|restart|status}exit
fi#获取用户输入的命令
cmd$1#定义函数功能
function zookeeperManger(){case $cmd instart)echo 启动服务 remoteExecution start;;stop)echo 停止服务remoteExecution stop;;restart)echo 重启服务remoteExecution restart;;status)echo 查看状态remoteExecution status;;*)echo 无效参数用法为: $0 {start|stop|restart|status};;esac
}#定义执行的命令
function remoteExecution(){for (( i101 ; i103 ; i )) ; dotput setaf 2echo 10.0.0.${i} zkServer.sh $1 tput setaf 9ssh 10.0.0.${i} source /etc/profile.d/kafka.sh; zkServer.sh $1 2/dev/nulldone
}#调用函数
zookeeperManger[rootelk101.oldboyedu.com ~]# chmod x /usr/local/sbin/zkManager.sh
[rootelk101.oldboyedu.com ~]# zkManager.sh start
[rootelk101.oldboyedu.com ~]# zkManager.sh staus验证集群
[rootelk103.oldboyedu.com ~]# zkCli.sh -server 10.0.0.101:2181,10.0.0.102:2181,10.0.0.103:218111、使用zkWeb管理zookeeper集群
(1)下载软件包
[rootelk103.oldboyedu.com ~]# wget http://192.168.15.253/ElasticStack/day07-/softwares/zkWeb-v1.2.1.jar(2)启动zkweb
java -jar zkWeb-v1.2.1.jar 12、快速搭建kafka单点环境
(1)下载kafka软件包
[rootelk101.oldboyedu.com ~]# wget http://192.168.15.253/ElasticStack/day07-/softwares/kafka_2.13-3.2.1.tgz(2)解压软件包
[rootelk101.oldboyedu.com ~]# tar xf kafka_2.13-3.2.1.tgz -C /oldboyedu/softwares/(3)创建符号连接
[rootelk101.oldboyedu.com ~]# cd /oldboyedu/softwares/ ln -svf kafka_2.13-3.2.1 kafka(4)配置环境变量
[rootelk101.oldboyedu.com softwares]# cat /etc/profile.d/kafka.sh
#!/bin/bashexport ZK_HOME/oldboyedu/softwares/zk
export PATH$PATH:$ZK_HOME/bin
export KAFKA_HOME/oldboyedu/softwares/kafka
export PATH$PATH:$KAFKA_HOME/bin[rootelk101.oldboyedu.com softwares]# source /etc/profile.d/kafka.sh (5)修改配置文件
[rootelk101.oldboyedu.com ~]# yy /oldboyedu/softwares/kafka/config/server.properties
...
broker.id101
zookeeper.connect10.0.0.101:2181,10.0.0.102:2181,10.0.0.103:2181/oldboyedu-linux85-kafka321
[rootelk101.oldboyedu.com ~]# (6)启动kafka单点
[rootelk101.oldboyedu.com softwares]# kafka-server-start.sh -daemon $KAFKA_HOME/config/server.properties (7)验证zookeeper的源数据信息
[rootelk101 softwares]# kafka-server-start.sh -daemon $KAFKA_HOME/config/server.properties
[rootelk101 softwares]# jps
4134 Jps
1134 Elasticsearch
1135 Elasticsearch
2815 QuorumPeerMain
[rootelk101 softwares]# cat /tmp/kafka-logs/meta.properties
#
#Fri Jun 07 11:14:46 CST 2024
cluster.idH2ceIpqTT1iUzb46e5jeKw
version0
broker.id101
[rootelk101 softwares]# jps
4156 Jps
1134 Elasticsearch
1135 Elasticsearch
2815 QuorumPeerMain
[rootelk101 softwares]# rm -rf /tmp/kafka-logs/
[rootelk101 softwares]# kafka-server-start.sh -daemon $KAFKA_HOME/config/server.properties在zkWeb查看即可。
今日作业 (1)完成课堂的所有练习并整理思维导图; 扩展作业: (1)使用ansible一键部署zookeeper集群;
