搭建mongodb分布式集群(3台主机分片集群)

搭建mongodb分布式集群(分片集群+keyfile安全认证以及用户权限)

2020-01-02 12:56:37

介绍：

　　　　分片（sharding）是指将数据库拆分，将其分散在不同的机器上的过程。将数据分散到不同的机器上，不需要功能强大的服务器就可以存储更多的数据和处理更大的负载。基本思想就是将集合切成小块，这些块分散到若干片里，每个片只负责总数据的一部分，最后通过一个均衡器来对各个分片进行均衡（数据迁移）。通过一个名为mongos的路由进程进行操作，mongos知道数据和片的对应关系（通过配置服务器）。大部分使用场景都是解决磁盘空间的问题，对于写入有可能会变差（+++里面的说明+++），查询则尽量避免跨分片查询。使用分片的时机：

       Shard   Server: mongod 实例，用于存储实际的数据块，实际生产环境中一个shard server角色可由几台机器组个一个relica set承担，防止主机单点故障;
       Config  Server: mongod 实例，存储了整个 Cluster Metadata，其中包括 chunk 信息。
       Route  Server: mongos  实例，前端路由，客户端由此接入，且让整个集群看上去像单一数据库，前端应用可以透明使用

主机规划： 

IP地址	实例(端口)	实例(端口)	实例(端口)	实例(端口)	实例(端口)
192.168.2.3	mongos(27107)	configsvr(20000)	shard1(27018)	shard2(27019)	shard3(27020)
192.168.2.4	mongos(27107)	configsvr(20000)	shard1(27018)	shard2(27019)	shard3(27020)
192.168.2.5	mongos(27107)	configsvr(20000)	shard1(27018)	shard2(27019)	shard3(27020)

 

 

 

 

 

目录创建：

在每台服务器创建对应的目录

1 mkdir -p /data/mongodb/{share1,share2,share3}/db
2 mkdir -p /data/mongodb/mongos/db
3 mkdir -p /data/mongodb/configsvr/db
4 mkdir -p /data/mongodb/{conf,logs}

创建配置文件：

1 touch /data/mongodb/conf/configsvr.conf 
2 touch /data/mongodb/conf/mongos.conf 
3 touch /data/mongodb/conf/shard1.conf 
4 touch /data/mongodb/conf/shard2.conf 
5 touch /data/mongodb/conf/shard3.conf

配置文件详情：

    官网连接 ：https://docs.mongodb.com/manual/reference/configuration-options/

configsvr.conf

1 systemLog:
 2     destination: file
 3     logAppend: true
 4     path: /data/mongodb/logs/configsvr.log
 5 storage: 6  dbPath: /data/mongodb/configsvr/db 7  journal: 8  enabled: true 9 processManagement: 10  fork: true 11  pidFilePath: /data/mongodb/configsvr/configsvr.pid 12 net: 13 port: 20000 14 bindIp: 0.0.0.0 15 replication: 16  replSetName: config 17 sharding: 18 clusterRole: configsvr

shard1.conf

1 systemLog:
 2     destination: file
 3     logAppend: true
 4     path: /data/mongodb/logs/shard1.log
 5 storage: 6  dbPath: /data/mongodb/shard1/db 7  journal: 8  enabled: true 9 processManagement: 10  fork: true 11  pidFilePath: /data/mongodb/shard1/shard1.pid 12 net: 13 port: 27018 14 bindIp: 0.0.0.0 15 replication: 16  replSetName: shard1 17 sharding: 18 clusterRole: shardsvr

shard2.conf

1 systemLog:
 2     destination: file
 3     logAppend: true
 4     path: /data/mongodb/logs/shard2.log
 5 storage: 6  dbPath: /data/mongodb/shard2/db 7  journal: 8  enabled: true 9 processManagement: 10  fork: true 11  pidFilePath: /data/mongodb/shard2/shard2.pid 12 net: 13 port: 27019 14 bindIp: 0.0.0.0 15 replication: 16  replSetName: shard2 17 sharding: 18 clusterRole: shardsvr

shard3.conf

1 systemLog:
 2     destination: file
 3     logAppend: true
 4     path: /data/mongodb/logs/shard3.log
 5 storage: 6  dbPath: /data/mongodb/shard3/db 7  journal: 8  enabled: true 9 processManagement: 10  fork: true 11  pidFilePath: /data/mongodb/shard3/shard3.pid 12 net: 13 port: 27020 14 bindIp: 0.0.0.0 15 replication: 16  replSetName: shard3 17 sharding: 18 clusterRole: shardsvr

mongos.conf

1 systemLog:
 2     destination: file
 3     logAppend: true
 4     path: /data/mongodb/logs/mongos.log
 5 processManagement: 6  fork: true 7  pidFilePath: /data/mongodb/mongos/mongos.pid 8 net: 9 port: 27017 10 bindIp: 0.0.0.0 11 sharding: 12 configDB: config/192.168.2.3:20000,192.168.2.4:20000,192.168.2.5:20000

启动命令：

1 /data/mongodb/bin/mongod -f /data/mongodb/config/configser.conf
2 /data/mongodb/bin/mongod -f /data/mongodb/config/shard1.conf
3 /data/mongodb/bin/mongod -f /data/mongodb/config/shard2.conf
4 /data/mongodb/bin/mongod -f /data/mongodb/config/shard3.conf
5 /data/mongodb/bin/mongod -f /data/mongodb/config/mongos.conf

 停止命令:

killall mongod  #停止config 和shard 
killall mongos  #停止 mongos

连接 shard1 

/data/mongodb/bin/mongo 192.168.2.3:27018
执行

1 rs.initiate({
 2 "_id":"shard1",
 3 "members":[
 4 {
 5 "_id":0, 6 "host":"192.168.2.3:27018" 7 }, 8 { 9 "_id":1, 10 "host":"192.168.2.4:27018" 11 }, 12 { 13 "_id":2, 14 "host":"192.168.2.5:27018" 15 } 16 ] 17 })

连接 shard2

/data/mongodb/bin/mongo 192.168.2.3:27019
执行

1 rs.initiate({
 2 "_id":"shard2",
 3 "members":[
 4 {
 5 "_id":0, 6 "host":"192.168.2.3:27019" 7 }, 8 { 9 "_id":1, 10 "host":"192.168.2.4:27019" 11 }, 12 { 13 "_id":2, 14 "host":"192.168.2.5:27019" 15 } 16 ] 17 })

连接 shard3
/data/mongodb/bin/mongo 192.168.2.3:27020
执行

1 rs.initiate({
 2 "_id":"shard3",
 3 "members":[
 4 {
 5 "_id":0, 6 "host":"192.168.2.3:27020" 7 }, 8 { 9 "_id":1, 10 "host":"192.168.2.4:27020" 11 }, 12 { 13 "_id":2, 14 "host":"192.168.2.5:27020" 15 } 16 ] 17 })

连接 config
/data/mongodb/bin/mongo 192.168.2.3:20000

1 rs.initiate({
 2 "_id":"config",
 3 "members":[
 4 {
 5 "_id":0, 6 "host":"192.168.2.3:20000" 7 }, 8 { 9 "_id":1, 10 "host":"192.168.2.4:20000" 11 }, 12 { 13 "_id":2, 14 "host":"192.168.2.5:20000" 15 } 16 ] 17 })

连接 mongos 添加路由 

/data/mongodb/bin/mongo 192.168.2.3:27017

1 sh.addShard("shard1/192.168.2.3:27018,192.168.2.4:27018,192.168.2.5:27018")
2 sh.addShard("shard2/192.168.2.3:27019,192.168.2.4:27019,192.168.2.5:27019")
3 sh.addShard("shard3/192.168.2.3:27020,192.168.2.4:27020,192.168.2.5:27020")

查看状态

mongos> sh.status()
--- Sharding Status --- 
  sharding version: {
      "_id" : 1,
      "minCompatibleVersion" : 5,
      "currentVersion" : 6,
      "clusterId" : ObjectId("5e0995613452582938deff4f")
  }
  shards:
        {  "_id" : "shard1",  "host" : "shard1/192.168.2.3:27018,192.168.2.4:27018,192.168.2.5:27018",  "state" : 1 }
        {  "_id" : "shard2",  "host" : "shard2/192.168.2.3:27019,192.168.2.4:27019,192.168.2.5:27019",  "state" : 1 }
        {  "_id" : "shard3",  "host" : "shard3/192.168.2.3:27020,192.168.2.4:27020,192.168.2.5:27020",  "state" : 1 }
  active mongoses:
        "3.6.3" : 1
  autosplit:
        Currently enabled: yes
  balancer:
        Currently enabled:  yes
        Currently running:  no
        Failed balancer rounds in last 5 attempts:  5
        Last reported error:  Could not find host matching read preference { mode: "primary" } for set shard1
        Time of Reported error:  Mon Dec 30 2019 14:30:41 GMT+0800 (CST)
        Migration Results for the last 24 hours: 
                No recent migrations
  databases:
        {  "_id" : "config",  "primary" : "config",  "partitioned" : true }
                config.system.sessions
                        shard key: { "_id" : 1 }
                        unique: false
                        balancing: true
                        chunks:
                                shard1    1
                        { "_id" : { "$minKey" : 1 } } -->> { "_id" : { "$maxKey" : 1 } } on : shard1 Timestamp(1, 0) 
        {  "_id" : "dfcx_test",  "primary" : "shard2",  "partitioned" : false }
        {  "_id" : "test",  "primary" : "shard1",  "partitioned" : false }

 

初始化用户

接入其中一个mongos实例，添加管理员用户

1 use admin
2  db.createUser({user:'admin',pwd:'admin',roles：['clusterAdmin','dbAdminAnyDatabase','userAdminAnyDatabase','readWriteAnyDatabase']})
3 授权库账号
4 use df_test
5 db.createUser({user:'df_test',pwd:'admin',roles:['readWrite']})

role角色：
    数据库用户角色：read、readWrite;
    数据库管理角色：dbAdmin、dbOwner、userAdmin；
    集群管理角色：clusterAdmin、clusterManager、clusterMonitor、hostManager；
    备份恢复角色：backup、restore；
    所有数据库角色：readAnyDatabase、readWriteAnyDatabase、userAdminAnyDatabase、dbAdminAnyDatabase
    超级用户角色：root
    内部角色：__system
角色说明：
    read：允许用户读取指定数据库
    readWrite：允许用户读写指定数据库
    dbAdmin：允许用户在指定数据库中执行管理函数，如索引创建、删除，查看统计或访问system.profile
    userAdmin：允许用户向system.users集合写入，可以找指定数据库里创建、删除和管理用户
    clusterAdmin：只在admin数据库中可用，赋予用户所有分片和复制集相关函数的管理权限。
    readAnyDatabase：只在admin数据库中可用，赋予用户所有数据库的读权限
    readWriteAnyDatabase：只在admin数据库中可用，赋予用户所有数据库的读写权限
    userAdminAnyDatabase：只在admin数据库中可用，赋予用户所有数据库的userAdmin权限
    dbAdminAnyDatabase：只在admin数据库中可用，赋予用户所有数据库的dbAdmin权限。
    root：只在admin数据库中可用。超级账号，超级权限
    dbOwner: readWrite + dbAdmin + dbAdmin

 

数据操作

在案例中，创建appuser用户、为数据库实例df_test启动分片。

1 use df_test
2 db.createUser({user:'appuser',pwd:'AppUser@01',roles:[{role:'dbOwner',db:'df_test'}]}) 
3 sh.enableSharding("df_test")     #开启分片

 

创建集合userid，为其执行分片初始化。

1 use df_test
2 db.createCollection("users")
3 db.users.ensureIndex({userid:1}) 创建索引,
4 sh.shardCollection("dfcx_test.users",{userid:1}) 同时为集合指定片键
5 sh.shardCollection("dfcx_test.users.users", {users:"hashed"}, false, { numInitialChunks: 4} ) （添加参数,可以执行4 和5任意一个）

 

插入测试数据

1 mongos> for(var i=1;i<1000000;i++) db.users.insert({userid:i,username:"HSJ"+i,city:"beijing"})
2 mongos> for(var i=1;i<1000000;i++) db.users.insert({userid:i,username:"HSJ"+i,city:"tianjing"})

 

查看状态

sh.status()
--- Sharding Status --- 
  sharding version: {
      "_id" : 1,
      "minCompatibleVersion" : 5,
      "currentVersion" : 6,
      "clusterId" : ObjectId("5e0995613452582938deff4f")
  }
  shards:
        {  "_id" : "shard1",  "host" : "shard1/192.168.2.3:27018,192.168.2.4:27018,192.168.2.5:27018",  "state" : 1 }
        {  "_id" : "shard2",  "host" : "shard2/192.168.2.3:27019,192.168.2.4:27019,192.168.2.5:27019",  "state" : 1 }
        {  "_id" : "shard3",  "host" : "shard3/192.168.2.3:27020,192.168.2.4:27020,192.168.2.5:27020",  "state" : 1 }
  active mongoses:
        "3.6.3" : 1
  autosplit:
        Currently enabled: yes
  balancer:
        Currently enabled:  yes
        Currently running:  yes
        Collections with active migrations: 
                dfcx_test.users started at Thu Jan 02 2020 14:35:34 GMT+0800 (CST)
        Failed balancer rounds in last 5 attempts:  4
        Last reported error:  Could not find host matching read preference { mode: "primary" } for set shard1
        Time of Reported error:  Mon Dec 30 2019 14:32:41 GMT+0800 (CST)
        Migration Results for the last 24 hours: 
                1 : Success
  databases:
        {  "_id" : "config",  "primary" : "config",  "partitioned" : true }
                config.system.sessions
                        shard key: { "_id" : 1 }
                        unique: false
                        balancing: true
                        chunks:
                                shard1    1
                        { "_id" : { "$minKey" : 1 } } -->> { "_id" : { "$maxKey" : 1 } } on : shard1 Timestamp(1, 0) 
        {  "_id" : "dfcx_test",  "primary" : "shard2",  "partitioned" : true }
                dfcx_test.users
                        shard key: { "userid" : 1 }
                        unique: false
                        balancing: true
                        chunks:
                                shard1    1
                                shard2    3
                        { "userid" : { "$minKey" : 1 } } -->> { "userid" : 2 } on : shard1 Timestamp(2, 0) 
                        { "userid" : 2 } -->> { "userid" : 500002 } on : shard2 Timestamp(2, 1) 
                        { "userid" : 500002 } -->> { "userid" : 750003 } on : shard2 Timestamp(1, 3) 
                        { "userid" : 750003 } -->> { "userid" : { "$maxKey" : 1 } } on : shard2 Timestamp(1, 4) 
        {  "_id" : "test",  "primary" : "shard1",  "partitioned" : false }

 

mongodb 生产开启 keyfile安全认证以及用户权限

创建副本集认证key文件

1、创建key文件: 注意，三个节点必须要用同一份keyfile，在一台机器生成，拷贝到另外两台，并且修改成 600 的文件属性

1 openssl rand -base64 90 -out ./keyfile
2 cp keyfile /data/mongodb/conf、
3 chmod 600 /data/mongodb/keyfile

 

2.在每个配置文件里添加配置

config 和shard 添加：

security:
  keyFile: /data/mongodb/conf/keyfile
  authorization: enabled

 

mongos 添加

1 security:
2   keyFile: /data/mongodb/conf/keyfile

 

3.重启集群

 

 

 

文章参考连接：

     https://www.jianshu.com/p/f021f1f3c60b

     https://www.cnblogs.com/littleatp/p/8563273.html

     https://www.cnblogs.com/woxingwoxue/p/9875878.html

来源：oschina

链接：https://my.oschina.net/u/4308120/blog/3320978