TABLE OF CONTENTS

  1. 系统资源
  2. 基础配置
    1. 关闭防火墙
  3. 生成证书信息
    1. 创建CA配置文件
    2. 创建CA证书签名请求
    3. 生成CA证书和私钥
    4. 创建etcd的TLS认证证书
    5. 生成并分发etcd证书和私钥
  4. ETCD安装和配置
    1. 下载并分发etcd
    2. 配置etcd
    3. etcd健康检查
  5. 参考资料

ETCD是一个分布式键值存储组件，可以用于服务注册与发现。这篇文章主要记录下ETCD多节点单实例集群的搭建。etcd是服务端，etcdctl是客户端命令行。

系统资源

IP 角色 主机名 组件 配置
192.168.2.183 Windows10，个人PC Tsukasa VMWare -
192.168.2.100 deploy tsukasa ansible, docker, harbor 4Cx2Gx20G
192.168.2.100 kubernetes k8s-master kubectl, kubelet, kubeadm, flnanel 2Cx2Gx20G
192.168.2.101 kubernetes k8s-worker01 kubectl 2Cx1Gx20G
192.168.2.102 kubernetes k8s-worker02 kubectl 2Cx1Gx20G
192.168.2.103 etcd etcd01 etcd 2Cx1Gx20G
192.168.2.104 etcd etcd02 etcd 1Cx1Gx10G
192.168.2.105 etcd etcd03(leader) etcd 1Cx1Gx10G
192.168.2.106 mysql mysql01 mysql 2Cx1Gx15G
192.168.2.107 mysql mysql02 mysql 2Cx1Gx15G

基础配置

默认已经完成SSH免密登陆、时间同步等基础配置，如果未完成需要参考同系列之前的文章配置。

关闭防火墙

deploy主机执行

1
2
ansible etcd -m shell -a "systemctl stop firewalld && systemctl disable firewalld"
ansible etcd -m shell -a "sed -i 's/^SELINUX=.*/SELINUX=disabled/g' /etc/selinux/config"

生成证书信息

deploy主机执行，后续通过ansible命令将生成好的证书信息发送到etcd节点。

1
2
3
4
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -O /usr/bin/cfssl
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -O /usr/bin/cfssljson
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -O /usr/bin/cfssl-certinfo 
chmod +x /usr/bin/cfssl*

创建CA配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
mkdir -p /root/ssl
cd /root/ssl
cat >ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "876000h"
    },
    "profiles": {
      "etcd": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "876000h"
      }
    }
  }
}
EOF

创建CA证书签名请求

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

cat >ca-csr.json <<EOF
{
  "CN": "etcd",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "beijing",
      "L": "beijing",
      "O": "jdt",
      "OU": "iot"
    }
  ]
}
EOF

生成CA证书和私钥

1
cfssl gencert -initca ca-csr.json | cfssljson -bare ca

创建etcd的TLS认证证书

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
cfssl gencert -initca ca-csr.json | cfssljson -bare ca

cat > etcd-csr.json <<EOF
{
  "CN": "etcd",
  "hosts": [
    "192.168.2.103", 
    "192.168.2.104", 
    "192.168.2.105",
    "192.168.2.100",
    "etcd01",
    "etcd02",
    "etcd03",
    "master"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "beijing",
      "L": "beijing",
      "O": "jdt",
      "OU": "iot"
    }
  ]
}
EOF

生成并分发etcd证书和私钥

1
2
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=etcd etcd-csr.json | cfssljson -bare etcd
ansible  etcd -m copy -a "src=/root/ssl/ dest=/export/Data/certs/"

ETCD安装和配置

上面已经把etcd的证书和私钥分发到了各个etcd节点，下面进行etcd的安装和配置。

下载并分发etcd

1
2
3
4
5
6
7
8
9
10
11
# 创建etcd数据目录
ansible etcd -m shell -a "mkdir -p /export/Data/etcd_data"

# 如果下载速度很慢，可以通过用windows主机下载再上传至deploy主机
wget https://github.com/etcd-io/etcd/releases/download/v3.5.1/etcd-v3.5.1-linux-amd64.tar.gz
tar xf etcd-v3.5.1-linux-amd64.tar.gz  && cd etcd-v3.5.1-linux-amd64
ansible etcd -m copy -a "src=etcd  dest=/usr/bin/"
ansible etcd -m copy -a "src=etcdutl  dest=/usr/bin/"
ansible etcd -m copy -a "src=etcdctl  dest=/usr/bin/"
ansible etcd -m shell -a "chmod +x /usr/bin/etcd*"

配置etcd

创建etcd配置文件：/export/upload/etcd_config.sh

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
#!/bin/bash

#PEER_NAME指定本节点的主机名称/域名，
#PRIVATE_IP指定本节点的IP(用于后面配置文件的生成)
#ETCD_CLUSTER群集列表，是所有节点信息（内容格式: 各节点名称=https://ip:端口  名称任意但要有标识性）
#ETCD_INITIAL_CLUSTER_TOKEN为该etcd集群Token,同一集群token一致
interface_name=`cat /proc/net/dev | sed -n '3,$p' | awk -F ':' {'print $1'} | grep  -E "^ " | grep -v lo | head -n1`
ipaddr=`ip a | grep $interface_name  | awk '{print $2}' | awk -F"/"  '{print $1}' | awk -F':' '{print $NF}'`
export PEER_NAME=`hostname`
export PRIVATE_IP=`echo $ipaddr | tr -d '\r'`
export ETCD_CLUSTER="etcd01=https://192.168.2.103:2380,etcd02=https://192.168.2.104:2380,etcd03=https://192.168.2.105:2380"
export ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-1"


cat > /etc/systemd/system/etcd.service  <<EOF
[Unit]
Description=etcd
Documentation=https://github.com/coreos/etcd
Conflicts=etcd.service

[Service]
Type=notify
Restart=always
RestartSec=5s
LimitNOFILE=65536
TimeoutStartSec=0

ExecStart=/usr/bin/etcd --name ${PEER_NAME} \
    --data-dir /export/Data/etcd_data\
    --listen-client-urls https://${PRIVATE_IP}:2379 \
    --advertise-client-urls https://${PRIVATE_IP}:2379 \
    --listen-peer-urls https://${PRIVATE_IP}:2380 \
    --initial-advertise-peer-urls https://${PRIVATE_IP}:2380 \
    --cert-file=/export/Data/certs/etcd.pem \
    --key-file=/export/Data/certs/etcd-key.pem \
    --client-cert-auth \
    --trusted-ca-file=/export/Data/certs/ca.pem \
    --peer-cert-file=/export/Data/certs/etcd.pem \
    --peer-key-file=/export/Data/certs/etcd-key.pem \
    --peer-client-cert-auth \
    --peer-trusted-ca-file=/export/Data/certs/ca.pem \
    --initial-cluster ${ETCD_CLUSTER} \
    --initial-cluster-token etcd-cluster-1 \
    --initial-cluster-state new

[Install]
WantedBy=multi-user.target

EOF

分发etcd配置文件到各个etcd节点。

1
2
3
4
5
6
7
8
# 分发etcd_config.sh配置脚本到etcd节点
ansible etcd -m copy -a "src=/export/upload/etcd_config.sh dest=/root/etcd_config.sh"
ansible etcd -m shell -a "chmod u+x,g+x /root/etcd_config.sh"
ansible etcd -m shell -a "sh /root/etcd_config.sh"
# 重启etcd服务
ansible etcd -m shell -a "systemctl daemon-reload"
ansible etcd -m service -a 'name=etcd  state=started'
ansible etcd -m shell -a "systemctl enable etcd"

etcd健康检查

创建etcd的健康检查脚本：/export/upload/check_etcd.sh，用于在各个etcd节点快速检查etcd集群健康状态。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
#!/bin/bash

HOST1=192.168.2.103
HOST2=192.168.2.104
HOST3=192.168.2.105
ENDPOINTS=$HOST1:2379,$HOST2:2379,$HOST3:2379
#因为开启了证书验证，因此执行命令需加上证书
KEY="--cacert=/export/Data/certs/ca.pem \
--cert=/export/Data/certs/etcd.pem \
--key=/export/Data/certs/etcd-key.pem"


#etcd集群健康信息
etcdctl --endpoints=$ENDPOINTS $KEY endpoint health

#etcd集群状态信息
etcdctl --endpoints=$ENDPOINTS $KEY --write-out=table endpoint status

#etcd集群成员信息
etcdctl --endpoints=$ENDPOINTS $KEY member list -w table

etcdctl --endpoints 192.168.2.103:2379,192.168.2.104:2379,192.168.2.105:2379 --cacert=/export/Data/certs/ca.pem --cert=/export/Data/certs/etcd.pem --key=/export/Data/certs/etcd-key.pem endpoint health

分发到各个etcd节点：

1
2
3
4
5
6
7
8
9
chmod u+x,g+x check_etcd.sh

ansible etcd -m copy -a "src=/export/upload/check_etcd.sh dest=/root/check_etcd.sh"
ansible etcd -m shell -a "chmod u+x,g+x /root/check_etcd.sh"


ETCDCTL_API=3 etcdctl --cacert=/export/Data/certs/ca.pem --cert=/export/Data/certs/etcd.pem --key=/export/Data/certs/etcd-key.pem --endpoints="https://192.168.2.103:2379,https://192.168.2.104:2379,https://192.168.2.105:2379" endpoint health --write-out=table

etcdctl --cacert=/export/Data/certs/ca.pem --cert=/export/Data/certs/etcd.pem --key=/export/Data/certs/etcd-key.pem --endpoints="https://192.168.2.103:2379,https://192.168.2.104:2379,https://192.168.2.105:2379" endpoint health --write-out=table

执行etcd健康检查脚本，即可查询到etcd集群状态。下图显示etcd集群健康检查正常。

参考资料

etcd github repo

声明：本站所有文章均为原创或翻译，遵循署名 - 非商业性使用 - 禁止演绎 4.0 国际许可协议，如需转载请确保您对该协议有足够了解，并附上作者名 (Tsukasa) 及原文地址