MongoDB k8s下部署 Replica Set

星辰

2020-11-26

2267

人

人评论

人举报

[TOC]

# MongoDB k8s下部署 Replica Set
副本集相关知识的介绍请查看专题下的文章《[MongDB 复制与分片集群](https://www.itnoteshare.com/course/11/175/previewcourse.htm "MongDB 复制与分片集群")》

**说明：**
如果k8s集群用的存储卷是分布式存储卷的话，用mongo-sidecar的方式去搭建更为简单，本次版主使用部署3个`StatefulSet`原因是版主的k8s集群采用的是`nfs`存储，并不能算是真正的分布式存储，为了确保通过副本集实现有效的数据备份，需要开启多个`StatefulSet`然后分存储卷需要分别挂载到不同的nfs服务器。

## K8s 下搭建 MongoDB过程记录
**环境介绍：**
此次搭建环境是k8s，实际操作是在kubesphere上操作的。
**mongo版本：**
`mongo:4.4`
**说明**

* 搭建的集群没有设置密码，设置密码参数配置请自行添加，在命令里面开启密码验证，并在环境变量中加入密码即可。
* 本次搭建没有建存储卷，这块需要自行建立

**节点数：**
设置了3个副本

### 创建3个mongo服务
**第一步：**
创建3个`StatefulSet`，脚本如下，创建其他几个时请自行修改其中的标识`mongo-rep-work`为`mongo-rep-work2`和`mongo-rep-work3`，这里只贴一份yaml文件，其他内容是一样的

StatefulSet.yaml
```
kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: mongo-rep-work2
  namespace: jiangyin
  labels:
    app: mongo-rep-work2
  annotations:
    kubesphere.io/alias-name: mongo-rep-work2
    kubesphere.io/description: |-
      mongo-rep-work2
      mongo:4.4
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mongo-rep-work2
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: mongo-rep-work2
      annotations:
        kubesphere.io/containerSecrets: ''
        kubesphere.io/restartedAt: '2020-11-26T11:33:28.075Z'
        logging.kubesphere.io/logsidecar-config: '{}'
    spec:
      containers:
        - name: container-zp3y70
          image: 'mongo:4.4'
          command:
            - mongod
            - '--replSet'
            - rs0
            - '--bind_ip'
            - 0.0.0.0
          ports:
            - name: tcp-27017
              containerPort: 27017
              protocol: TCP
          resources:
            limits:
              cpu: '4'
              memory: 4000Mi
            requests:
              cpu: 200m
              memory: 200Mi
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      serviceAccountName: default
      serviceAccount: default
      securityContext: {}
      affinity: {}
      schedulerName: default-scheduler
  serviceName: mongo-rep-work2-9ha52x
  podManagementPolicy: OrderedReady
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 0
  revisionHistoryLimit: 10
```

**第二步：**
创建对应的`Service`，下面仅贴一个的yaml文件其他几个自行修改值`mongo-rs-svc`为`mongo-rs2-svc`和`mongo-rs3-svc`
Service.yaml
```
kind: Service
apiVersion: v1
metadata:
  name: mongo-rs-svc
  namespace: jiangyin
  labels:
    app: mongo-rs-svc
  annotations:
    kubesphere.io/alias-name: mongo-rs-svc
    kubesphere.io/description: mongo-rs-svc
spec:
  ports:
    - name: http-27017
      protocol: TCP
      port: 27017
      targetPort: 27017
      nodePort: 30384
  selector:
    app: mongo-rep-work
  clusterIP: 10.233.22.140
  type: NodePort
  sessionAffinity: ClientIP
  externalTrafficPolicy: Cluster
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 10800
```

### 集群初始化
进入上面任何一个服务的容器，比如`mongo-rep-work-0`执行初始化脚本，建立副本集之间的关系
```
# 连接mongo，由于没创建密码所以不需要密码
mongo --port 27017
# 初始化集群关系
https://docs.mongodb.com/manual/reference/method/rs.initiate/
rs.initiate(
   {
      _id: "rs0",
      version: 1,
      members: [
         { _id: 0, host : "mongo-rs-svc:27017" },
         { _id: 1, host : "mongo-rs2-svc:27017" },
         { _id: 2, host : "mongo-rs3-svc:27017" }
      ]
   }
)
# 以上脚本由于采用的是服务名创建关联，所以只能部署在集群中的应用只能通过服务名连接，外部连接不到，如果需要外部能连接，可以把服务名和端口换成外部ip和端口，如下
rs.initiate(
   {
      _id: "rs0",
      version: 1,
      members: [
         { _id: 0, host : "192.168.3.161:30384" },
         { _id: 1, host : "192.168.3.161:31963" },
         { _id: 2, host : "192.168.3.161:31313" }
      ]
   }
)
# 执行完后验证关系
rs.status()
```

### 验证数据是否有同步
```
# 进入主节点，这个需要连接上去看脚本提示是不是primary，默认只有主节点可以写入
mongo --port 27017
# 创建数据库
use noteshare_mongo
# 插入测试数据
db.noteshare_mongo.insert({"name":"noteshare"})

show dbs
show tables
db.noteshare_mongo.find()
# 主节点能查到数据
# 进入从节点查询下上面写入的数据是否可以查到，如果可以查到说明已经成功复制了数据
```
### 处理问题记录
* MongoDb的“not master and slaveok=false”错误及解决方法
此问题问子节点无法查询数据，通过以下命令开启即可查询
`rs.slaveOk();`

### mongodb副本集修改配置问题
如果已经初始化了集群，该如何修改配置呢，请查看一下相关指令说明
```
# 查看配置rs.config()；需要找到primary主机，在该主节点服务器上才有权限修改配置
rs.config()
# 移除原配置文件中的已经变更地址的主机
rs.remove("ip:port")
# 添加新的地址主机
rs.add("ip:port")
# 设置priority优先级和修改配置

>var config = rs.config()
>config.members[2].priority=2
# 重新更新配置
>rs.reconfig(config)

# 添加仲裁节点
rs.addArb("192.168.0.3:27019");
# 删除仲裁节点
rs.remove("192.168.0.3:27019");

## 初始化时添加仲裁节点
rs.initiate(
   {
      _id: "rs0",
      version: 1,
      members: [
         { _id: 0, host : "192.168.3.161:30384" },
         { _id: 1, host : "192.168.3.161:31963" },
         { _id: 2, host : "192.168.3.161:31313",arbiterOnly:true}
      ]
   }
)
```

### 开启mongo认证
```
mongo 副本集的认证需要使用--keyfile的形式来做
生成keyFile字符串
随便找台服务器生成
openssl rand -base64 756
将字符串配置为k8s的秘钥文件
# 挂载秘钥
挂载到临时目录，读写
/mongoconf/keyFile
启动命令：
/bin/sh,-c
参数
cp /mongoconf/keyFile /home/keyFile;chmod 600 /home/keyFile;mongod --replSet=rs0 --bind_ip=0.0.0.0 --auth --keyFile=/home/keyFile

# 启动3个mongo
# 建立集群连接 用rs.initiate
# 登录主节点
use admin
# 创建账号密码
db.createUser({user:'admin',pwd:'NoteShare2020',roles:['userAdminAnyDatabase']})
db.auth("admin","NoteShare2020")
# 更改为最高权限，之前创建的权限不够，无法恢复数据库
db.updateUser('admin',{pwd:'NoteShare2020',roles:[{role:'root',db:'admin'}]})
# 注意用navicat连接时，如果之前连接过未输入密码的，需要删除重新建立连接，感觉有缓存啥的存在误导

# 恢复数据到集群
在primary上进行，在没有赋予root权限时会报错，加上权限就好了。
mongorestore -h 192.168.3.161:30384 -uadmin -pNoteShare2020 --authenticationDatabase admin -d qyzc_jy --dir  /home/backup/2020_11_18/qyzc_jy
# 对恢复的库设置账号密码和附加权限，在主节点上进行
rs0:PRIMARY> use admin
switched to db admin
rs0:PRIMARY> db.auth("admin","NoteShare2020")
1
rs0:PRIMARY> use qyzc_jy
switched to db qyzc_jy
rs0:PRIMARY> db.createUser({user:'noteshare',pwd:'NoteShare2020',roles:['readWrite']})
Successfully added user: { "user" : "noteshare", "roles" : [ "readWrite" ] }
```

### 单节点数据迁移到集群
```
# 注意不能把之前的单节点直接加入集群，那样会把单节点的数据同步成集群先用的数据，会造成数据丢失
# 暂停服务器防止数据的继续写入  ===版主这里的应用暂时没有要求那么高，允许短时不提供服务，这里应该探索不断服务的方式来迁移，后续再研究下。
# 把单节点的数据备份下来
# 在主节点上通过恢复命令把数据恢复的集群中
mongorestore -d qyzc_jy --dir /home/backup/2020_11_18/qyzc_jy
# 应用更新连接到集群服务检查数据情况
mongo.properties

# mongo settings
dataBaseName=qyzc_jy
#hostName=192.168.3.161
#port=31013
#userName=test
#password=test2020
maxWaitTime=300000
socketTimeout=100000
maxConnectionLifeTime=86400000
connectTimeout=43200000

##副本集模式下
##是否副本集 默认 false
isReplica=true
##主机地址之间以英文逗号分隔
hostName=192.168.3.161,192.168.3.161,192.168.3.161,192.168.3.161
##主机端口之间以英文逗号分隔,个数与主机地址匹配，不够的默认为最后一个
port=30384,31963,31313,32702

#链接池数量
#connectionsPerHost=100
#连接超时时间默认10,000
#connectTimeout=10000
#最大等待时间默认120,000
#maxWaitTime=120000
#I/O socket读写时间,默认0代表不限制
#socketTimeout=0
```

### 通过挂载mongo.conf配置文件部署
发现挂载没起作用，可能挂载方式有问题还需要了解下原理。不知道是不是挂载是在启动mongo之后的动作，导致挂载进去的文件没起作用。
另外可以研究下`mikefarah/yq:2.4.1`容器的处理过程，看到较多是用这个来处理挂载配置文件的。

### 操作打印留痕
```
# mongo --port 27017
MongoDB shell version v4.4.2
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("063f2976-cad8-477e-ba16-91c1738d9668") }
MongoDB server version: 4.4.2
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
        https://docs.mongodb.com/
Questions? Try the MongoDB Developer Community Forums
        https://community.mongodb.com
---
The server generated these startup warnings when booting:
        2020-11-26T11:24:09.169+00:00: Access control is not enabled for the database. Read and write access to data and configuration is unrestricted
        2020-11-26T11:24:09.169+00:00: You are running this process as the root user, which is not recommended
        2020-11-26T11:24:09.170+00:00: /sys/kernel/mm/transparent_hugepage/enabled is 'always'. We suggest setting it to 'never'
        2020-11-26T11:24:09.170+00:00: /sys/kernel/mm/transparent_hugepage/defrag is 'always'. We suggest setting it to 'never'
---
---
        Enable MongoDB's free cloud-based monitoring service, which will then receive and display
        metrics about your deployment (disk utilization, CPU, operation statistics, etc).

The monitoring data will be available on a MongoDB website with a unique URL accessible to you
        and anyone you share the URL with. MongoDB may use this information to make product
        improvements and to suggest MongoDB products and deployment options to you.

To enable free monitoring, run the following command: db.enableFreeMonitoring()
        To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
---
> rs.initiate(
...    {
...       _id: "rs0",
...       version: 1,
...       members: [
...          { _id: 0, host : "mongo-rs-svc:27017" },
...          { _id: 1, host : "mongo-rs2-svc:27017" },
...          { _id: 2, host : "mongo-rs3-svc:27017" }
...       ]
...    }
... )
{
        "ok" : 1,
        "$clusterTime" : {
                "clusterTime" : Timestamp(1606389935, 1),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        },
        "operationTime" : Timestamp(1606389935, 1)
}
rs0:SECONDARY>
rs0:SECONDARY> rs.status()
{
        "set" : "rs0",
        "date" : ISODate("2020-11-26T11:26:19.508Z"),
        "myState" : 1,
        "term" : NumberLong(1),
        "syncSourceHost" : "",
        "syncSourceId" : -1,
        "heartbeatIntervalMillis" : NumberLong(2000),
        "majorityVoteCount" : 2,
        "writeMajorityCount" : 2,
        "votingMembersCount" : 3,
        "writableVotingMembersCount" : 3,
        "optimes" : {
                "lastCommittedOpTime" : {
                        "ts" : Timestamp(1606389976, 1),
                        "t" : NumberLong(1)
                },
                "lastCommittedWallTime" : ISODate("2020-11-26T11:26:16.269Z"),
                "readConcernMajorityOpTime" : {
                        "ts" : Timestamp(1606389976, 1),
                        "t" : NumberLong(1)
                },
                "readConcernMajorityWallTime" : ISODate("2020-11-26T11:26:16.269Z"),
                "appliedOpTime" : {
                        "ts" : Timestamp(1606389976, 1),
                        "t" : NumberLong(1)
                },
                "durableOpTime" : {
                        "ts" : Timestamp(1606389976, 1),
                        "t" : NumberLong(1)
                },
                "lastAppliedWallTime" : ISODate("2020-11-26T11:26:16.269Z"),
                "lastDurableWallTime" : ISODate("2020-11-26T11:26:16.269Z")
        },
        "lastStableRecoveryTimestamp" : Timestamp(1606389946, 3),
        "electionCandidateMetrics" : {
                "lastElectionReason" : "electionTimeout",
                "lastElectionDate" : ISODate("2020-11-26T11:25:46.253Z"),
                "electionTerm" : NumberLong(1),
                "lastCommittedOpTimeAtElection" : {
                        "ts" : Timestamp(0, 0),
                        "t" : NumberLong(-1)
                },
                "lastSeenOpTimeAtElection" : {
                        "ts" : Timestamp(1606389935, 1),
                        "t" : NumberLong(-1)
                },
                "numVotesNeeded" : 2,
                "priorityAtElection" : 1,
                "electionTimeoutMillis" : NumberLong(10000),
                "numCatchUpOps" : NumberLong(0),
                "newTermStartDate" : ISODate("2020-11-26T11:25:46.263Z"),
                "wMajorityWriteAvailabilityDate" : ISODate("2020-11-26T11:25:47.020Z")
        },
        "members" : [
                {
                        "_id" : 0,
                        "name" : "mongo-rs-svc:27017",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 131,
                        "optime" : {
                                "ts" : Timestamp(1606389976, 1),
                                "t" : NumberLong(1)
                        },
                        "optimeDate" : ISODate("2020-11-26T11:26:16Z"),
                        "syncSourceHost" : "",
                        "syncSourceId" : -1,
                        "infoMessage" : "",
                        "electionTime" : Timestamp(1606389946, 1),
                        "electionDate" : ISODate("2020-11-26T11:25:46Z"),
                        "configVersion" : 1,
                        "configTerm" : 1,
                        "self" : true,
                        "lastHeartbeatMessage" : ""
                },
                {
                        "_id" : 1,
                        "name" : "mongo-rs2-svc:27017",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 44,
                        "optime" : {
                                "ts" : Timestamp(1606389976, 1),
                                "t" : NumberLong(1)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(1606389976, 1),
                                "t" : NumberLong(1)
                        },
                        "optimeDate" : ISODate("2020-11-26T11:26:16Z"),
                        "optimeDurableDate" : ISODate("2020-11-26T11:26:16Z"),
                        "lastHeartbeat" : ISODate("2020-11-26T11:26:18.259Z"),
                        "lastHeartbeatRecv" : ISODate("2020-11-26T11:26:19.266Z"),
                        "pingMs" : NumberLong(0),
                        "lastHeartbeatMessage" : "",
                        "syncSourceHost" : "mongo-rs-svc:27017",
                        "syncSourceId" : 0,
                        "infoMessage" : "",
                        "configVersion" : 1,
                        "configTerm" : 1
                },
                {
                        "_id" : 2,
                        "name" : "mongo-rs3-svc:27017",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 44,
                        "optime" : {
                                "ts" : Timestamp(1606389976, 1),
                                "t" : NumberLong(1)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(1606389976, 1),
                                "t" : NumberLong(1)
                        },
                        "optimeDate" : ISODate("2020-11-26T11:26:16Z"),
                        "optimeDurableDate" : ISODate("2020-11-26T11:26:16Z"),
                        "lastHeartbeat" : ISODate("2020-11-26T11:26:18.258Z"),
                        "lastHeartbeatRecv" : ISODate("2020-11-26T11:26:19.266Z"),
                        "pingMs" : NumberLong(0),
                        "lastHeartbeatMessage" : "",
                        "syncSourceHost" : "mongo-rs-svc:27017",
                        "syncSourceId" : 0,
                        "infoMessage" : "",
                        "configVersion" : 1,
                        "configTerm" : 1
                }
        ],
        "ok" : 1,
        "$clusterTime" : {
                "clusterTime" : Timestamp(1606389976, 1),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        },
        "operationTime" : Timestamp(1606389976, 1)
}
rs0:PRIMARY>
```

所有评论列表