Redis Cluster集群模式容器化部署

安装Docker和docker-compose
准备docker-compose文件
准备Redis配置文件
Linux内核参数优化
启停Redis实例
Redis集群搭建

环境准备：

IP	版本	角色	端口
172.x.x.11	RHEL 7.9	master	6379
172.x.x.12	RHEL 7.9	master	6379
172.x.x.13	RHEL 7.9	master	6379
172.x.x.11	RHEL 7.9	replica	6380
172.x.x.12	RHEL 7.9	replica	6380
172.x.x.13	RHEL 7.9	replica	6380

dockercompose_12">安装Docker和docker-compose

安装docker-ce：

# 移除旧版本
sudo yum remove docker \
                  docker-client \
                  docker-client-latest \
                  docker-common \
                  docker-latest \
                  docker-latest-logrotate \
                  docker-logrotate \
                  docker-engine
		  
# 配置仓库
sudo yum install -y yum-utils device-mapper-persistent-data lvm2
#官方源
#sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
#阿里云源
sudo yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
sudo yum makecache fast

# 检查可安装的版本
yum list docker-ce --showduplicates | sort -r

# 安装指定版本
#sudo yum install docker-ce-<VERSION_STRING> docker-ce-cli-<VERSION_STRING> containerd.io docker-buildx-plugin docker-compose-plugin
yum install -y docker-ce-25.0.1 docker-ce-cli-25.0.1 containerd.io docker-buildx-plugin docker-compose-plugin

# 启动
sudo systemctl enable docker
sudo systemctl start docker

配置阿里云镜像源:

cat > /etc/docker/daemon.json << EOF
{
 "registry-mirrors" : ["https://b9pmyelo.mirror.aliyuncs.com"]
}
EOF

sudo systemctl restart docker

🐬Docker社区版部署参见：

https://docs.docker.com/engine/install/centos/
https://gottdeskrieges.blog.csdn.net/article/details/113242029

安装docker-compose：

curl -SL https://github.com/docker/compose/releases/download/v2.24.6/docker-compose-linux-x86_64 -o /usr/local/bin/docker-compose

chmod 755 /usr/local/bin/docker-compose

📖参考：https://docs.docker.com/compose/install/standalone/

dockercompose_72">准备docker-compose文件

Redis版本：7.2.4
Redis安装路径：/opt/docker-compose/redis

准备单台服务的compose文件，每台服务器上部署两个Redis实例（服务端口分别为6379和6380）和一个redis exporter。容器网络采用仅主机模式。

version: '3'

services:
  redis-server-6379:
    image: redis:7.2.4
    network_mode: host
    container_name: redis-server-6379
    restart: unless-stopped
    #ports:
    #  - 6379:6379
    environment:
      TZ: "Asia/Shanghai"
    command: ["/6379/conf/redis.conf"]
    volumes:
      - ./6379/conf/:/6379/conf/
      - ./6379/data/:/6379/data/
    
  redis-server-6380:
    image: redis:7.2.4
    network_mode: host
    container_name: redis-server-6380
    restart: unless-stopped
    #ports:
    #  - 6380:6380
    environment:
      TZ: "Asia/Shanghai"
    command: ["/6380/conf/redis.conf"]
    volumes:
      - ./6380/conf/:/6380/conf/
      - ./6380/data/:/6380/data/ 
   
  redis-exporter:
    image: oliver006/redis_exporter
    network_mode: host
    restart: unless-stopped
    environment:
      TZ: Asia/Shanghai
      REDIS_ADDR: redis://localhost:6379,redis://localhost:6380
      REDIS_USER: redis_monitor
      REDIS_PASSWORD: Monpass_xxxx
    # ports:
    #   - 9121:9121

准备Redis配置文件

本机创建目录结构，并配置好redis.conf和users.acl文件。

mkdir -vp /opt/docker-compose/redis/6379/conf/
mkdir -vp /opt/docker-compose/redis/6379/data/

mkdir -vp /opt/docker-compose/redis/6380/conf/
mkdir -vp /opt/docker-compose/redis/6380/data/

主实例配置文件/opt/docker-compose/redis/6379/conf/redis.conf：

bind 0.0.0.0
port 6379
timeout 3600
tcp-keepalive 300
loglevel notice
databases 16

save 600 100
save 120 10000
save 7200 1

dir /6379/data/
masteruser clusterrepl  
masterauth Replpass_xxxx

aclfile /6379/conf/users.acl
replica-read-only yes

cluster-announce-ip ""172.x.x.11""
cluster-announce-port 6379

maxclients 10000
maxmemory 10gb
maxmemory-policy noeviction

appendonly yes
appendfsync everysec
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb

cluster-enabled yes
cluster-config-file nodes-6379.conf
cluster-node-timeout 15000

从实例配置文件/opt/docker-compose/redis/6380/conf/redis.conf：

bind 0.0.0.0
port 6380
timeout 3600
tcp-keepalive 300
loglevel notice
databases 16

save 600 100
save 120 10000
save 7200 1

dir /6380/data/
masteruser clusterrepl  
masterauth Replpass_xxxx

aclfile /6380/conf/users.acl
replica-read-only yes

cluster-announce-ip "172.x.x.11"
cluster-announce-port 6380

maxclients 10000
maxmemory 10gb
maxmemory-policy noeviction

appendonly yes
appendfsync everysec
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb

cluster-enabled yes
cluster-config-file nodes-6380.conf
cluster-node-timeout 15000

在/6379/conf/和/6380/conf/下配置ACL文件，定义好管理员用户、redis-exporter监控用户、主从同步用户、应用用户的相关权限。示例如下：

user default off nopass &* -@all
user root on >Rootpass_xxxx allkeys allchannels +@all
user redis_monitor on >Monpass_xxxx +client +ping +info +config|get +cluster|info +slowlog +latency +memory +select +get +scan +xinfo +type +pfcount +strlen +llen +scard +zcard +hlen +xlen +eval allkeys
user clusterrepl on >Replpass_xxxx +psync +replconf +ping
user appuser on >Apppass_xxxx allkeys allchannels +@all -@admin -@dangerous

Linux内核参数优化

修改Linux内核参数：

sed -i '/vm.overcommit_memory/d' /etc/sysctl.conf
echo "vm.overcommit_memory = 1" >> /etc/sysctl.conf
echo "net.core.somaxconn=2048" >> /etc/sysctl.conf
sysctl -p

禁用透明大页：

cat <<EOF >> /etc/rc.d/rc.local
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
  echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
EOF

chmod +x /etc/rc.d/rc.local

编辑/etc/security/limits.conf和/etc/security/limits.d/90-nproc.conf，添加以下配置：

*       soft    core    unlimited
*       hard    core    unlimited
*       soft    nofile  65536
*       hard    nofile  65536
*       soft    nproc   65536
*       hard    nproc   65536

最后重启服务器生效。

启停Redis实例

修改文件权限：

cd /opt/docker-compose/
chown -R polkitd:root redis/

在docker-compose.yml文件所在路径下执行以下命令来启停单个节点上的两个REDIS实例。

#启动容器
cd /opt/docker-compose/redis && docker-compose up -d

#停止容器
cd /opt/docker-compose/redis && docker-compose stop

Redis集群搭建

将三台服务器上的6个实例搭建成一个三主三从的集群模式：

docker exec -it redis-server-6379 redis-cli --user root -a <PASSWORD> \
--cluster create 172.x.x.11:6379 172.x.x.11:6380 172.x.x.12:6379 172.x.x.12:6380 172.x.x.13:6379 172.x.x.13:6380 \
--cluster-replicas 1

集群创建过程中输出信息如下：

>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 172.x.x.11:6380 to 172.x.x.11:6379
Adding replica 172.x.x.12:6380 to 172.x.x.12:6379
Adding replica 172.x.x.13:6380 to 172.x.x.13:6379
>>> Trying to optimize slaves allocation for anti-affinity
[WARNING] Some slaves are in the same host as their master
M: a12d5d8daf4f7ef18a490f8db4766d66736c51e5 172.x.x.11:6379
   slots:[0-5460] (5461 slots) master
M: 7e6923ec3063d0833b134c3e30012d3a6605884b 172.x.x.12:6379
   slots:[5461-10922] (5462 slots) master
M: aa89fb16d032f62d58ac13d0b29450580f5f6337 172.x.x.13:6379
   slots:[10923-16383] (5461 slots) master
S: 26bfdc8f75bf45870e77b392e077b52380833bd9 172.x.x.11:6380
   replicates a12d5d8daf4f7ef18a490f8db4766d66736c51e5
S: 025b9b7e1163a39c4fcb76c6c8cb53b4c5a6604a 172.x.x.12:6380
   replicates 7e6923ec3063d0833b134c3e30012d3a6605884b
S: 042b02321735a6995f474703adb8d73601fc1773 172.x.x.13:6380
   replicates aa89fb16d032f62d58ac13d0b29450580f5f6337
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join

>>> Performing Cluster Check (using node 172.x.x.11:6379)
M: a12d5d8daf4f7ef18a490f8db4766d66736c51e5 172.x.x.11:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
M: 7e6923ec3063d0833b134c3e30012d3a6605884b 172.x.x.12:6379
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 042b02321735a6995f474703adb8d73601fc1773 172.x.x.13:6380
   slots: (0 slots) slave
   replicates aa89fb16d032f62d58ac13d0b29450580f5f6337
S: 025b9b7e1163a39c4fcb76c6c8cb53b4c5a6604a 172.x.x.12:6380
   slots: (0 slots) slave
   replicates 7e6923ec3063d0833b134c3e30012d3a6605884b
S: 26bfdc8f75bf45870e77b392e077b52380833bd9 172.x.x.11:6380
   slots: (0 slots) slave
   replicates a12d5d8daf4f7ef18a490f8db4766d66736c51e5
M: aa89fb16d032f62d58ac13d0b29450580f5f6337 172.x.x.13:6379
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

检查集群节点：

docker exec -it redis-server-6379 redis-cli -c -h 172.x.x.11 -p 6379 cluster nodes

其中 -c 表示以集群模式执行，不加的话只会从当前连接的实例查询。

输出信息如下：

a12d5d8daf4f7ef18a490f8db4766d66736c51e5 172.x.x.11:6379@16379 myself,master - 0 1710740884000 1 connected 0-5460
7e6923ec3063d0833b134c3e30012d3a6605884b 172.x.x.12:6379@16379 master - 0 1710740885000 2 connected 5461-10922
042b02321735a6995f474703adb8d73601fc1773 172.x.x.13:6380@16380 slave aa89fb16d032f62d58ac13d0b29450580f5f6337 0 1710740887000 3 connected
025b9b7e1163a39c4fcb76c6c8cb53b4c5a6604a 172.x.x.12:6380@16380 slave 7e6923ec3063d0833b134c3e30012d3a6605884b 0 1710740885837 2 connected
26bfdc8f75bf45870e77b392e077b52380833bd9 172.x.x.11:6380@16380 slave a12d5d8daf4f7ef18a490f8db4766d66736c51e5 0 1710740886839 1 connected
aa89fb16d032f62d58ac13d0b29450580f5f6337 172.x.x.13:6379@16379 master - 0 1710740887841 3 connected 10923-16383

也可以直接检查节点信息文件。

cat /opt/docker-compose/redis/6379/data/nodes-6379.conf