k8s部署redis cluster集群的实现

目录

• redis 介绍
• 为什么要用redis
• 什么是redis cluster集群
• k8s以statefulset方式部署redis cluster集群：

redis 介绍

• redis代表remote dictionary server是一种开源的内存中数据存储，通常用作数据库，缓存或消息代理。它可以存储和操作高级数据类型，例如列表，地图，集合和排序集合。
• 由于redis接受多种格式的密钥，因此可以在服务器上执行操作，从而减少了客户端的工作量。
• 它仅将磁盘用于持久性，而将数据完全保存在内存中。
• redis是一种流行的数据存储解决方案，并被github，pinterest，snapchat，twitter，stackoverflow，flickr等技术巨头所使用。

为什么要用redis

• 它的速度非常快。它是用ansi c编写的，并且可以在posix系统上运行，例如linux，mac os x和solaris。
• redis通常被排名为最流行的键/值数据库和最流行的与容器一起使用的nosql数据库。
• 其缓存解决方案减少了对云数据库后端的调用次数。
• 应用程序可以通过其客户端api库对其进行访问。
• 所有流行的编程语言都支持redis。
• 它是开源且稳定的。

什么是redis cluster集群

• redis cluster是一组redis实例，旨在通过对数据库进行分区来扩展数据库，从而使其更具弹性。
• 群集中的每个成员（无论是主副本还是辅助副本）都管理哈希槽的子集。如果主机无法访问，则其从机将升级为主机。在由三个主节点组成的最小redis群集中，每个主节点都有一个从节点（以实现最小的故障转移），每个主节点都分配有一个介于0到16,383之间的哈希槽范围。节点a包含从0到5000的哈希槽，节点b从5001到10000，节点c从10001到16383。
• 群集内部的通信是通过内部总线进行的，使用协议传播有关群集的信息或发现新节点。

k8s以statefulset方式部署redis cluster集群：

1. 部署nfs

2. 创建pv

3. 部署redis

4. 初始化redis集群

主机说明：

系统	ip	角色	cpu	内存	hostname
centos 7.8	192.168.30.128	master	>=2	>=2g	master1
centos 7.8	192.168.30.129	master	>=2	>=2g	master2
centos 7.8	192.168.30.130	node	>=2	>=2g	node1
centos 7.8	192.168.30.131	node	>=2	>=2g	node2
centos 7.8	192.168.30.132	node	>=2	>=2g	node3

kubectl get node

name      status   roles    age   version
master1   ready    master   33d   v1.14.0
master2   ready    master   33d   v1.14.0
node1     ready    <none>   33d   v1.14.0
node2     ready    <none>   33d   v1.14.0
node3     ready    <none>   33d   v1.14.0

部署nfs

在master2节点上做nfs共享，

yum -y install nfs-utils rpcbind
mkdir -p /data/redis/{cluster0,cluster1,cluster2,cluster3,cluster4,cluster5}
vim /etc/exports

/data/redis/cluster0 192.168.30.0/24(rw,sync,no_root_squash)
/data/redis/cluster1 192.168.30.0/24(rw,sync,no_root_squash)
/data/redis/cluster2 192.168.30.0/24(rw,sync,no_root_squash)
/data/redis/cluster3 192.168.30.0/24(rw,sync,no_root_squash)
/data/redis/cluster4 192.168.30.0/24(rw,sync,no_root_squash)
/data/redis/cluster5 192.168.30.0/24(rw,sync,no_root_squash)

chmod -r 755 /data/redis
exportfs -arv
systemctl enable rpcbind && systemctl start rpcbind
systemctl enable nfs && systemctl start nfs

nfs部署完毕。对于需要使用nfs的node节点，都要安装nfs：

yum -y install nfs-utils

创建pv

创建pv：

mkdir -p /home/k8s/redis  && cd /home/k8s/redis
vim pv.yaml

apiversion: v1
kind: persistentvolume
metadata:
  name: nfs-pv0
spec:
  capacity:
    storage: 1gi
  accessmodes:
    - readwritemany
  nfs:
    server: 192.168.30.129
    path: /data/redis/cluster0

---
apiversion: v1
kind: persistentvolume
metadata:
  name: nfs-pv1
spec:
  capacity:
    storage: 1gi
  accessmodes:
    - readwritemany
  nfs:
    server: 192.168.30.129
    path: /data/redis/cluster1

---
apiversion: v1
kind: persistentvolume
metadata:
  name: nfs-pv2
spec:
  capacity:
    storage: 1gi
  accessmodes:
    - readwritemany
  nfs:
    server: 192.168.30.129
    path: /data/redis/cluster2

---
apiversion: v1
kind: persistentvolume
metadata:
  name: nfs-pv3
spec:
  capacity:
    storage: 1gi
  accessmodes:
    - readwritemany
  nfs:
    server: 192.168.30.129
    path: /data/redis/cluster3

---
apiversion: v1
kind: persistentvolume
metadata:
  name: nfs-pv4
spec:
  capacity:
    storage: 1gi
  accessmodes:
    - readwritemany
  nfs:
    server: 192.168.30.129
    path: /data/redis/cluster4

---
apiversion: v1
kind: persistentvolume
metadata:
  name: nfs-pv5
spec:
  capacity:
    storage: 1gi
  accessmodes:
    - readwritemany
  nfs:
    server: 192.168.30.129
    path: /data/redis/cluster5

kubectl apply -f pv.yaml

kubectl get pv

name      capacity   access modes   reclaim policy   status      claim   storageclass   reason   age
nfs-pv0   1gi        rwx            retain           available                                   3s
nfs-pv1   1gi        rwx            retain           available                                   3s
nfs-pv2   1gi        rwx            retain           available                                   3s
nfs-pv3   1gi        rwx            retain           available                                   3s
nfs-pv4   1gi        rwx            retain           available                                   3s
nfs-pv5   1gi        rwx            retain           available                                   3s

pv创建完毕。

部署redis

vim namespace.yaml

apiversion: v1
kind: namespace
metadata:
  name: public-service

vim redis.conf

appendonly yes
cluster-enabled yes
cluster-config-file /var/lib/redis/nodes.conf
cluster-node-timeout 5000
dir /var/lib/redis
port 6379

vim redis.yaml

apiversion: v1
kind: service
metadata:
  name: redis
  namespace: public-service
  labels:
    app: redis
spec:
  selector:
    app: redis
    appcluster: redis-cluster
  ports:
  - name: redis
    port: 6379
  clusterip: none
  
---
apiversion: v1
kind: service
metadata:
  name: redis-access
  namespace: public-service
  labels:
    app: redis
spec:
  selector:
    app: redis
    appcluster: redis-cluster
  ports:
  - name: redis-access
    protocol: tcp
    port: 6379
    targetport: 6379

---
apiversion: apps/v1
kind: statefulset
metadata:
  name: redis
  namespace: public-service
spec:
  servicename: redis
  replicas: 6
  selector:
    matchlabels:
      app: redis
      appcluster: redis-cluster
  template:
    metadata:
      labels:
        app: redis
        appcluster: redis-cluster
    spec:
      terminationgraceperiodseconds: 20
      affinity:
        podantiaffinity:
          preferredduringschedulingignoredduringexecution:
          - weight: 100
            podaffinityterm:
              labelselector:
                matchexpressions:
                - key: app
                  operator: in
                  values:
                  - redis
              topologykey: kubernetes.io/hostname
      containers:
      - name: redis
        image: redis:latest
        command:
          - "redis-server"
        args:
          - "/etc/redis/redis.conf"
          - "--protected-mode"
          - "no"
        resources:
          requests:
            cpu: "500m"
            memory: "500mi"
        ports:
        - containerport: 6379
          name: redis
          protocol: tcp
        - containerport: 16379
          name: cluster
          protocol: tcp
        volumemounts:
        - name: conf
          mountpath: /etc/redis
        - name: data
          mountpath: /var/lib/redis
      volumes:
      - name: conf
        configmap:
          name: redis-conf
          items:
          - key: redis.conf
            path: redis.conf
  volumeclaimtemplates:
  - metadata:
      name: data
      namespace: public-service
    spec:
      accessmodes: [ "readwritemany" ]
      resources:
        requests:
          storage: 1gi

kubectl apply -f namespace.yaml

kubectl create configmap redis-conf --from-file=redis.conf -n public-service

kubectl apply -f redis.yaml

kubectl get svc -n public-service

name           type        cluster-ip    external-ip   port(s)    age
redis          clusterip   none          <none>        6379/tcp   20s
redis-access   clusterip   10.96.2.100   <none>        6379/tcp   20s

kubectl get pod -n public-service

name      ready   status    restarts   age
redis-0   1/1     running   0          2m43s
redis-1   1/1     running   0          2m18s
redis-2   1/1     running   0          108s
redis-3   1/1     running   0          80s
redis-4   1/1     running   0          48s
redis-5   1/1     running   0          30s

kubectl get pvc -n public-service

name           status   volume    capacity   access modes   storageclass   age
data-redis-0   bound    nfs-pv4   1gi        rwx                           3m4s
data-redis-1   bound    nfs-pv0   1gi        rwx                           2m39s
data-redis-2   bound    nfs-pv1   1gi        rwx                           2m9s
data-redis-3   bound    nfs-pv2   1gi        rwx                           101s
data-redis-4   bound    nfs-pv3   1gi        rwx                           69s
data-redis-5   bound    nfs-pv5   1gi        rwx                           51s

redis-access这个service方便集群内访问redis集群，redis部署完毕。

初始化redis集群

使用redis-tribe工具进行集群的初始化。

下载redis-tribe：

kubectl run -it ubuntu --image=ubuntu --restart=never -n public-service bash

root@ubuntu:/# cat > /etc/apt/sources.list << eof
deb http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ xenial main restricted
deb http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ xenial-updates main restricted
deb http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ xenial universe
deb http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ xenial-updates universe
deb http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ xenial multiverse
deb http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ xenial-updates multiverse
deb http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ xenial-backports main restricted universe multiverse
deb http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ xenial-security main restricted
deb http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ xenial-security universe
deb http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ xenial-security multiverse
eof

root@ubuntu:/# apt-get update

root@ubuntu:/# apt-get install -y libncursesw5 libreadline6 libtinfo5 --allow-remove-essential

root@ubuntu:/# apt-get install -y libpython2.7-stdlib python2.7 python-pip redis-tools dnsutils

root@ubuntu:/# pip install --upgrade pip

root@ubuntu:/# pip install redis-trib==0.5.1

初始化集群：

root@ubuntu:/# redis-trib.py create \
  `dig +short redis-0.redis.public-service.svc.cluster.local`:6379 \
  `dig +short redis-1.redis.public-service.svc.cluster.local`:6379 \
  `dig +short redis-2.redis.public-service.svc.cluster.local`:6379
  
root@ubuntu:/# redis-trib.py replicate \
  --master-addr `dig +short redis-0.redis.public-service.svc.cluster.local`:6379 \
  --slave-addr `dig +short redis-3.redis.public-service.svc.cluster.local`:6379

root@ubuntu:/# redis-trib.py replicate \
  --master-addr `dig +short redis-1.redis.public-service.svc.cluster.local`:6379 \
  --slave-addr `dig +short redis-4.redis.public-service.svc.cluster.local`:6379

root@ubuntu:/# redis-trib.py replicate \
  --master-addr `dig +short redis-2.redis.public-service.svc.cluster.local`:6379 \
  --slave-addr `dig +short redis-5.redis.public-service.svc.cluster.local`:6379
  
root@ubuntu:/# exit

查看集群：

kubectl exec -it -n public-service redis-0 bash

root@redis-0:/data# redis-cli -c

127.0.0.1:6379> cluster nodes               #列出节点信息

aac2b3d320da67eedf3512ed0e38a1cdce5bc8fe 172.10.2.55:6379@16379 slave 7c4d60cf32685484ea6c5cb4493a937dfbf6b8a5 0 1592276224727 3 connected
2efad514b2f3c7fe4530dd6dc63c0df8ffdb793d 172.10.2.54:6379@16379 master - 0 1592276224224 1 connected 0-5461
524f03526a4b683d7d4de19296431810bfdc22cf 172.10.3.60:6379@16379 slave df5bc3c2e2851d63cdb9f762efde6e1b0d38efed 0 1592276223117 5 connected
7c4d60cf32685484ea6c5cb4493a937dfbf6b8a5 172.10.4.77:6379@16379 myself,master - 0 1592276224000 2 connected 5462-10922
df5bc3c2e2851d63cdb9f762efde6e1b0d38efed 172.10.3.59:6379@16379 master - 0 1592276223217 0 connected 10923-16383
c1dbaaef4a583e372c43eed52c22cd9ad7184d18 172.10.4.78:6379@16379 slave 2efad514b2f3c7fe4530dd6dc63c0df8ffdb793d 0 1592276223719 4 connected

127.0.0.1:6379> cluster info                #集群状态

cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:5
cluster_my_epoch:2
cluster_stats_messages_ping_sent:514
cluster_stats_messages_pong_sent:491
cluster_stats_messages_meet_sent:2
cluster_stats_messages_sent:1007
cluster_stats_messages_ping_received:491
cluster_stats_messages_pong_received:516
cluster_stats_messages_received:1007

redis集群初始化完成，已经形成3主3从的cluster集群。

写入数据：

127.0.0.1:6379> set key1 aaa
ok

127.0.0.1:6379> set key2 bbb
-> redirected to slot [4998] located at 172.10.2.54:6379
ok

172.10.2.54:6379> set key3 ccc
ok

kubectl exec -it -n public-service redis-4 bash

root@redis-4:/data# redis-cli -c

127.0.0.1:6379> get key1
-> redirected to slot [9189] located at 172.10.4.77:6379
"aaa"

172.10.4.77:6379> get key2
-> redirected to slot [4998] located at 172.10.2.54:6379
"bbb"

172.10.2.54:6379> get key3
"ccc"

可以看出redis cluster集群是去中心化的，每个节点都是平等的，连接哪个节点都可以获取和设置数据。

主从切换：

选择redis-2模拟宕掉，测试主从切换，

kubectl get pod -n public-service -o wide

name      ready   status      restarts   age   ip            node    nominated node   readiness gates
redis-0   1/1     running     0          62m   172.10.4.77   node1   <none>           <none>
redis-1   1/1     running     0          62m   172.10.2.54   node2   <none>           <none>
redis-2   1/1     running     0          61m   172.10.3.59   node3   <none>           <none>
redis-3   1/1     running     0          61m   172.10.2.55   node2   <none>           <none>
redis-4   1/1     running     0          61m   172.10.4.78   node1   <none>           <none>
redis-5   1/1     running     0          61m   172.10.3.60   node3   <none>           <none>
ubuntu    0/1     completed   0          57m   172.10.2.56   node2   <none>           <none>

kubectl exec -it -n public-service redis-2 bash

root@redis-2:/data# redis-cli -c

127.0.0.1:6379> role

1) "master"
2) (integer) 2898
3) 1) 1) "172.10.3.60"
      2) "6379"
      3) "2898"

可以看到，redis-2是master，它的slave是172.10.3.60，即redis-5。

kubectl delete pod -n public-service redis-2                #模拟节点宕掉

kubectl get pod -n public-service redis-2 -o wide

name      ready   status    restarts   age   ip            node    nominated node   readiness gates
redis-2   1/1     running   0          38s   172.10.3.61   node3   <none>           <none>

kubectl exec -it -n public-service redis-2 bash

root@redis-2:/data# redis-cli -c

127.0.0.1:6379> role

1) "slave"
2) "172.10.3.60"
3) (integer) 6379
4) "connected"
5) (integer) 3430

kubectl exec -it -n public-service redis-5 bash

root@redis-5:/data# redis-cli -c

127.0.0.1:6379> role

1) "master"
2) (integer) 3584
3) 1) 1) "172.10.3.61"
      2) "6379"
      3) "3570"
      
127.0.0.1:6379> cluster nodes

aac2b3d320da67eedf3512ed0e38a1cdce5bc8fe 172.10.2.55:6379@16379 slave 7c4d60cf32685484ea6c5cb4493a937dfbf6b8a5 0 1592278859530 2 connected
2efad514b2f3c7fe4530dd6dc63c0df8ffdb793d 172.10.2.54:6379@16379 master - 0 1592278859000 1 connected 0-5461
c1dbaaef4a583e372c43eed52c22cd9ad7184d18 172.10.4.78:6379@16379 slave 2efad514b2f3c7fe4530dd6dc63c0df8ffdb793d 0 1592278859000 1 connected
524f03526a4b683d7d4de19296431810bfdc22cf 172.10.3.60:6379@16379 myself,master - 0 1592278857000 6 connected 10923-16383
7c4d60cf32685484ea6c5cb4493a937dfbf6b8a5 172.10.4.77:6379@16379 master - 0 1592278858021 2 connected 5462-10922
df5bc3c2e2851d63cdb9f762efde6e1b0d38efed 172.10.3.61:6379@16379 slave 524f03526a4b683d7d4de19296431810bfdc22cf 0 1592278859000 6 connected

可以看到，redis-2在重启之后变为slave，而它之前的slave——redis-5变为master，而且是新redis-2的master。

集群的主从切换没有问题。k8s部署redis cluster集群完成。

到此这篇关于k8s部署redis cluster集群的实现的文章就介绍到这了,更多相关k8s部署redis cluster内容请搜索www.887551.com以前的文章或继续浏览下面的相关文章希望大家以后多多支持www.887551.com！