k8s部署之分布式KV存储Etcd

Etcd是什么

Etcd是一个分布式、使用Raft算法维护一致性的kv存储系统,与其类似产品有Zookeeper(老牌经典)、Consul等,Etcd相对ZK,更加轻量、易运维。具体三者之间的对比可参考 https://luyiisme.github.io/2017/04/22/spring-cloud-service-discovery-products/

使用场景

和zk、consul等类似,使用场景多用于:

  • 服务发现
  • 消息发布与订阅
  • 负载均衡
  • 分布式锁
  • 分布式队列

读写性能

压测数据参考官方:
https://coreos.com/etcd/docs/latest/op-guide/performance.html

本地集群部署

  • 操作系统:Debian8 x64
  • Etcd v3.2.7

A. 安装

wget https://github.com/coreos/etcd/releases/download/v3.2.7/etcd-v3.2.7-linux-arm64.tar.gz
tar xf etcd-v3.2.7-linux-arm64.tar.gz
cd etcd-v3.2.7-linux-amd64
cp etc* /usr/local/bin/

etcd: Etcd服务端文件
etcdctl: 供用户使用的命令客户端

B. 启动服务

root@a4c8d490:/home/geekwolf# etcd
2017-09-07 15:42:23.957656 I | etcdmain: etcd Version: 3.2.7
2017-09-07 15:42:23.957699 I | etcdmain: Git SHA: bb66589
2017-09-07 15:42:23.957718 I | etcdmain: Go Version: go1.8.3
2017-09-07 15:42:23.957723 I | etcdmain: Go OS/Arch: linux/amd64
2017-09-07 15:42:23.957729 I | etcdmain: setting maximum number of CPUs to 8, total number of available CPUs is 8
2017-09-07 15:42:23.957739 W | etcdmain: no data-dir provided, using default data-dir ./default.etcd
2017-09-07 15:42:23.957764 N | etcdmain: the server is already initialized as member before, starting as etcd member...
2017-09-07 15:42:23.957995 I | embed: listening for peers on http://localhost:2380
2017-09-07 15:42:23.958107 I | embed: listening for client requests on localhost:2379
2017-09-07 15:42:23.964607 I | etcdserver: name = default
2017-09-07 15:42:23.964633 I | etcdserver: data dir = default.etcd
2017-09-07 15:42:23.964652 I | etcdserver: member dir = default.etcd/member
2017-09-07 15:42:23.964657 I | etcdserver: heartbeat = 100ms
2017-09-07 15:42:23.964663 I | etcdserver: election = 1000ms
2017-09-07 15:42:23.964668 I | etcdserver: snapshot count = 100000
2017-09-07 15:42:23.964680 I | etcdserver: advertise client URLs = http://localhost:2379
2017-09-07 15:42:23.973007 I | etcdserver: restarting member 8e9e05c52164694d in cluster cdf818194e3a8c32 at commit index 14
2017-09-07 15:42:23.973041 I | raft: 8e9e05c52164694d became follower at term 2
2017-09-07 15:42:23.973065 I | raft: newRaft 8e9e05c52164694d [peers: [], term: 2, commit: 14, applied: 0, lastindex: 14, lastterm: 2]
2017-09-07 15:42:23.984367 W | auth: simple token is not cryptographically signed
2017-09-07 15:42:23.993237 I | etcdserver: starting server... [version: 3.2.7, cluster version: to_be_decided]
2017-09-07 15:42:23.993659 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32
2017-09-07 15:42:23.993754 N | etcdserver/membership: set the initial cluster version to 3.2
2017-09-07 15:42:23.993796 I | etcdserver/api: enabled capabilities for version 3.2
2017-09-07 15:42:24.473288 I | raft: 8e9e05c52164694d is starting a new election at term 2
2017-09-07 15:42:24.473451 I | raft: 8e9e05c52164694d became candidate at term 3
2017-09-07 15:42:24.473519 I | raft: 8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 3
2017-09-07 15:42:24.473568 I | raft: 8e9e05c52164694d became leader at term 3
2017-09-07 15:42:24.473605 I | raft: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 3
2017-09-07 15:42:24.478746 I | etcdserver: published {Name:default ClientURLs:[http://localhost:2379]} to cluster cdf818194e3a8c32
2017-09-07 15:42:24.478824 I | embed: ready to serve client requests
2017-09-07 15:42:24.479116 N | embed: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!

由上面的输出可知:

  • etcd服务之间通信端口是2380,暴露给客户端端口为2379
  • 默认将数据存放到当前路径default.etcd/目录下
  • 该节点的名称默认为default
  • 集群和节点都会生成唯一的uuid
  • 启动服务时,会根据raft算法,选举leader

C. 测试

查看api版本(默认api版本是v2)
root@a4c8d490:~/k8s/etcd-v3.2.7-linux-amd64# etcdctl  --version
etcdctl version: 3.2.7
API version: 2

使用API V3方法:
Etcd服务端和客户端添加变量 export ETCDCTL_API=3,重新启动etcd服务即可
以下操作在api v3版本:
写入key: etcdctl put foo bar
读取key: etcdctl get foo
多key范围读取: etcdctl get foo foo9(会将foo..foo8的key读取,不包括foo9)
读取过往版本key的值(Etcd键值对的修改都会增加全局修订版本号,--rev为版本号):
 etcdctl get --rev=4 foo foo9
删除key: etcdctl del foo
范围删除(foo->foo9):etcdctl del foo foo9
观察key变化:etcdctl watch foo
观察范围key变化: etcdctl watch foo foo9
从rev=2版本开始观察key变化: etcdctl watch --rev=2 foo
压缩版本5之前的修订版本(压缩后5之前的版本不可能访问): etcdctl compact 5
授予key有效期:
创建租约: 
$ etcdctl lease grant 10
lease 694d5e5b63a74f31 granted with TTL(10s)
附加key foo到租约694d5e5b63a74f31,该租约过期后,会删除附加的所有key
撤销租约(撤销后,附加改租约的所有key被删除): etcdctl lease revoke 32695410dcc0ca06
维持租约(执行后,会一直维持该租约): etcdctl lease keep-alive 32695410dcc0ca0

其他参数可参考 etcdctl --help
通过HTTP操作:
Etcd v2: https://coreos.com/etcd/docs/latest/v2/api.html
Etcd v3: https://coreos.com/etcd/docs/latest/dev-guide/api_grpc_gateway.html

多节点集群部署

静态模式部署
环境说明(三节点集群)
节点 地址 主机
etcd1 192.168.234.133 etcd1.simlinux.com
etcd2 192.168.234.133 etcd2.simlinux.com
etcd3 192.168.234.133 etcd3.simlinux.com
初始化环境

三个节点分别设置主机名:

hostnamectl --static  set-hostname etcd1.simlinux.com
hostnamectl --static  set-hostname etcd2.simlinux.com
hostnamectl --static  set-hostname etcd3.simlinux.com

三个节点hosts文件添加:

vim /etc/hosts
192.168.234.133 etcd1.simlinux.com 
192.168.234.133 etcd2.simlinux.com 
192.168.234.133 etcd3.simlinux.com 

生成etcd证书(用于etcd间、客户端与etcd通信)

由上篇k8s部署之使用CFSSL创建证书的CA来生成

cat  etcd.json 
{
    "CN": "etcd",
    "hosts": [
        "127.0.0.1",
        "192.168.234.133",
        "192.168.234.134",
        "192.168.234.135"
    ],
    "key": {
        "algo": "ecdsa",
        "size": 256
    },
    "names": [
        {
            "C": "CN",
            "L": "ShangHai",
            "ST": "ShangHai",
            "O": "K8s",
            "OU": "System"
        }
    ]
}

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer etcd.json | cfssljson -bare etcd

将CA和etcd证书拷贝到etcd所有节点:
cp ca.pem  etcd-key.pem  etcd.pem /etc/etcd/ssl/
安装etcd节点(所有节点)
wget https://github.com/coreos/etcd/releases/download/v3.2.7/etcd-v3.2.7-linux-amd64.tar.gz
tar xf etcd-v3.2.7-linux-amd64.tar.gz
cd etcd-v3.2.7-linux-amd64
chmod +x etcd*
cp etcd* /bin
etcd配置

服务管理(所有节点相同):
vim /usr/lib/systemd/system/etcd.service

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/data/k8s/etcd/
EnvironmentFile=/etc/etcd/etcd.conf
ExecStart=/bin/etcd 
  --name=${NAME} 
  --cert-file=/etc/etcd/ssl/etcd.pem 
  --key-file=/etc/etcd/ssl/etcd-key.pem 
  --peer-cert-file=/etc/etcd/ssl/etcd.pem 
  --peer-key-file=/etc/etcd/ssl/etcd-key.pem 
  --trusted-ca-file=/etc/etcd/ssl/ca.pem 
  --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem 
  --initial-advertise-peer-urls=${INITIAL_ADVERTISE_PEER_URLS} 
  --listen-peer-urls=${LISTEN_PEER_URLS} 
  --listen-client-urls=${LISTEN_CLIENT_URLS} 
  --advertise-client-urls=${ADVERTISE_CLIENT_URLS} 
  --initial-cluster-token=${INITIAL_CLUSTER_TOKEN} 
  --initial-cluster=${INITIAL_CLUSTER} 
  --initial-cluster-state=new 
  --data-dir=${DATA_DIR}
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

配置文件:
vim /etc/etcd/etcd.conf

#节点名称
NAME="etcd1"
#etcd数据存放目录
DATA_DIR="/data/k8s/etcd"
#etcd节点间通信监听地址
LISTEN_PEER_URLS="https://192.168.234.133:2380"
#对外提供服务的地址
LISTEN_CLIENT_URLS="https://192.168.234.133:2379,https://127.0.0.1:2379"
#通知其他etcd节点本实例地址
INITIAL_ADVERTISE_PEER_URLS="https://192.168.234.133:2380"
#初始化集群内节点地址
INITIAL_CLUSTER="etcd1=https://192.168.234.133:2380,etcd2=https://192.168.234.134:2380,etcd3=https://192.168.234.135:2380"
#初始化状态.new表示新建,已经存在的集群使用existing
INITIAL_CLUSTER_STATE="new"
#创建集群的token,每个集群唯一
INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster"
#告知其他集群本节点客户端监听地址
ADVERTISE_CLIENT_URLS="https://192.168.234.133:2379"
ETCDCTL_API=3
其中NAME/LISTEN_PEER_URLS/LISTEN_CLIENT_URLS/INITIAL_ADVERTISE_PEER_URLS/ADVERTISE_CLIENT_URLS替换成相应节点名称和地址
服务管理

systemctl start etcd.service
systemctl stop etcd.service
systemctl status etcd.service(查看服务状态及日志)

测试验证
[root@etcd1 ~]# export etcd1=192.168.234.133
[root@etcd1 ~]# export etcd2=192.168.234.134
[root@etcd1 ~]# export etcd3=192.168.234.135
[root@etcd1 ~]# export ENDPOINTS=$etcd1:2379,$etcd2:2379,$etcd3:2379


查看集群成员:
[root@etcd1 etcd]#  etcdctl --write-out=table --endpoints=$ENDPOINTS --cacert=/etc/etcd/ssl/ca.pem  --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem member list
+------------------+---------+-------+------------------------------+------------------------------+
|        ID        | STATUS  | NAME  |          PEER ADDRS          |         CLIENT ADDRS         |
+------------------+---------+-------+------------------------------+------------------------------+
| 1a4a83ef243ff1c9 | started | etcd2 | https://192.168.234.134:2380 | https://192.168.234.134:2379 |
| 68243ef8797bd1ce | started | etcd1 | https://192.168.234.133:2380 | https://192.168.234.133:2379 |
| fa30209a63d949b0 | started | etcd3 | https://192.168.234.135:2380 | https://192.168.234.135:2379 |
+------------------+---------+-------+------------------------------+------------------------------+

查看集群状态:

[root@etcd1 ~]# etcdctl --write-out=table --endpoints=$ENDPOINTS --cacert=/etc/etcd/ssl/ca.pem  --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem endpoint status
+----------------------+------------------+---------+---------+-----------+-----------+------------+
|       ENDPOINT       |        ID        | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+----------------------+------------------+---------+---------+-----------+-----------+------------+
| 192.168.234.133:2379 | 68243ef8797bd1ce |   3.2.7 |   25 kB |     false |        10 |          9 |
| 192.168.234.134:2379 | 1a4a83ef243ff1c9 |   3.2.7 |   25 kB |     false |        10 |          9 |
| 192.168.234.135:2379 | fa30209a63d949b0 |   3.2.7 |   25 kB |      true |        10 |          9 |
+----------------------+------------------+---------+---------+-----------+-----------+------------+

[root@etcd1 ~]# etcdctl --write-out=table --endpoints=$ENDPOINTS --cacert=/etc/etcd/ssl/ca.pem  --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem endpoint health
192.168.234.135:2379 is healthy: successfully committed proposal: took = 1.374345ms
192.168.234.134:2379 is healthy: successfully committed proposal: took = 2.217525ms
192.168.234.133:2379 is healthy: successfully committed proposal: took = 1.996245ms

保存快照:
[root@etcd1 ~]# etcdctl --write-out=table --endpoints=$ENDPOINTS --cacert=/etc/etcd/ssl/ca.pem  --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem snapshot save my.db
Snapshot saved at my.db

查看快照状态:
[root@etcd1 ~]# etcdctl --write-out=table --endpoints=$ENDPOINTS --cacert=/etc/etcd/ssl/ca.pem  --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem snapshot status my.db
+----------+----------+------------+------------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| 9a496339 |        3 |          8 |      25 kB |
+----------+----------+------------+------------+

恢复数据(要先删除原来数据目录,所有节点操作):
[root@etcd1 ~]# etcdctl  --endpoints=$ENDPOINTS --cacert=/etc/etcd/ssl/ca.pem  --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem snapshot restore my.db  --data-dir=/data/k8s/etcd/
2017-09-09 02:28:52.439616 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32

删除节点:
[root@etcd1 ~]# etcdctl --write-out=table --endpoints=$ENDPOINTS --cacert=/etc/etcd/ssl/ca.pem  --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem member remove 68243ef8797bd1ce

更新节点:
[root@etcd1 ~]# etcdctl --write-out=table --endpoints=$ENDPOINTS --cacert=/etc/etcd/ssl/ca.pem  --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem member update 68243ef8797bd1ce https://192.168.234.133:1111(INITIAL_ADVERTISE_PEER_URLS)


添加节点(删除etcd3,添加etcd4):
export etcd4=192.168.234.136
[root@etcd1 ~]#  etcdctl --endpoints=${etcd1}:2379,${etcd2}:2379 --cacert=/etc/etcd/ssl/ca.pem  --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem  member add etcd4 --peer-urls=http://192.168.234.136:2380

Etcd:从应用场景到实现原理的全方位解读 http://www.infoq.com/cn/articles/etcd-interpretation-application-scenario-implement-principle
Eetcd集群管理 https://coreos.com/etcd/docs/latest/demo.html

k8s部署之使用CFSSL创建证书

安装CFSSL

curl -s -L -o /bin/cfssl https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
curl -s -L -o /bin/cfssljson https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
curl -s -L -o /bin/cfssl-certinfo https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x /bin/cfssl*

容器相关证书类型

  • client certificate: 用于服务端认证客户端,例如etcdctl、etcd proxy、fleetctl、docker客户端
  • server certificate: 服务端使用,客户端以此验证服务端身份,例如docker服务端、kube-apiserver
  • peer certificate: 双向证书,用于etcd集群成员间通信

创建CA证书

生成默认CA配置
mkdir /opt/ssl
cd /opt/ssl
cfssl print-defaults config > ca-config.json
cfssl print-defaults csr > ca-csr.json

修改ca-config.json,分别配置针对三种不同证书类型的profile,其中有效期43800h为5年

{
    "signing": {
        "default": {
            "expiry": "43800h"
        },
        "profiles": {
            "server": {
                "expiry": "43800h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth"
                ]
            },
            "client": {
                "expiry": "43800h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "client auth"
                ]
            },
            "peer": {
                "expiry": "43800h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            }
        }
    }
}

修改ca-csr.config

{
    "CN": "Self Signed Ca",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "SH",
            "O": "Netease",
            "ST": "SH",            
            "OU": "OT"
        }    ]
}

生成CA证书和私钥

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
生成ca.pem、ca.csr、ca-key.pem(CA私钥,需妥善保管)

签发Server Certificate
cfssl print-defaults csr > server.json
vim server.json
{
    "CN": "Server",
    "hosts": [
        "192.168.1.1"
       ],
    "key": {
        "algo": "ecdsa",
        "size": 256
    },
    "names": [
        {
            "C": "CN",
            "L": "SH",
            "ST": "SH"
        }
    ]
}
生成服务端证书和私钥
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server server.json | cfssljson -bare server

签发Client Certificate
cfssl print-defaults csr > client.json
vim client.json
{
    "CN": "Client",
    "hosts": [],
    "key": {
        "algo": "ecdsa",
        "size": 256
    },
    "names": [
        {
            "C": "CN",
            "L": "SH",
            "ST": "SH"
        }
    ]
}
生成客户端证书和私钥
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client client.json | cfssljson -bare client

签发peer certificate
cfssl print-defaults csr > member1.json
vim member1.json
{
    "CN": "member1",
    "hosts": [
        "192.168.1.1"
    ],
    "key": {
        "algo": "ecdsa",
        "size": 256
    },
    "names": [
        {
            "C": "CN",
            "L": "SH",
            "ST": "SH"
        }
    ]
}
为节点member1生成证书和私钥:
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer member1.json | cfssljson -bare member1
针对etcd服务,每个etcd节点上按照上述方法生成相应的证书和私钥
最后校验证书

校验生成的证书是否和配置相符

openssl x509 -in ca.pem -text -noout
openssl x509 -in server.pem -text -noout
openssl x509 -in client.pem -text -noout

k8s集群所需证书

参考

https://coreos.com/os/docs/latest/generate-self-signed-certificates.html