You heard that MongoDB is very arrogant? MDB deploys sharded clusters and manages shards

Article Directory

• 1. The concept of copy
• 1. Two types of replica sets 2. Three roles: 4. Disadvantages
• Second, the concept of fragmentation
• Three, the components included in the sharded cluster
• Fourth, deploy MongoDB sharded cluster
• 1. Environment preparation 2. The topology diagram is as follows 3. Environment construction 4. config configuration server 5. Shard server 6. When a node has insufficient memory, allocate memory from other nodes 7. Start routing server 8. Set shards 9. Join shard node
• Five, fragment management
• 1. Add data, perform shard storage 2. Add tags 3. Connect to the config configuration server: 4. Add/delete shard servers

1. The concept of copy

Difference between master-slave replication and replica set
The biggest difference between master-slave cluster and copy set is that the replica set does not have a fixed "master node"; the entire cluster will select a master node when it hangs down, and then in the remaining slave nodes Select other nodes as the "master node". The replica set always has an active point (primary primary and one or more backup nodes (secondary).

1. Two types of replica sets

There are two types of replica sets and three roles.
Two types:
primary node (Primary) type: the main connection point for data operations, readable and writable.
Secondary (secondary, slave) node (Secondaries) type: data redundancy backup node, Can read or vote.

2. Three roles:

• Primary: Mainly receive all write operations. It is the master node.
• Replicate member (Replicate): from the master node through the replication operation to maintain the same data set. That is, the backup data can be written, but it can be read but needs to be configured. It is the default slave node type.
• Arbiter (Arbiter): Does not keep a copy of any data, only has the role of voting. Of course, the arbitration server can also be maintained as part of the replica set, that is, replica members can also be arbiters. It is also a type of slave node.

4. Disadvantages

It is equivalent to the cold after the replication master goes down. In order to achieve high availability, we have to introduce the concept of sharding.

Second, the concept of fragmentation

Sharding is a method of distributing data across multiple machines. MongoDB uses sharding to support deployments with very large data sets and high throughput operations.
In other words: sharding refers to the process of splitting data and dispersing it on different machines. Partitioning is sometimes used to express this concept. By distributing data to different machines, you can store more data and handle more loads without the need for powerful large computers.
Database systems with large data sets or high-throughput applications can challenge the capacity of a single server. For example, a high query rate will exhaust the server's CPU capacity. The working set size is larger than the RAM of the system will emphasize the I/0 capacity of the disk drive.

There are two ways to solve system growth: vertical expansion and horizontal expansion.

• Vertical expansion means increasing the capacity of a single server, such as using a more powerful CPU, adding more RAM or increasing the amount of storage space. The limitations of available technology may limit a single machine to be powerful enough for a given workload. In addition, cloud-based providers have hard
  caps based on available hardware configurations . As a result, the vertical scaling has a practical maximum.
• Horizontal expansion means dividing the system data set and loading multiple servers, adding other servers to increase capacity as needed. Although the overall speed or capacity of a single machine may not be high, each machine handles a subset of the entire workload and may provide higher efficiency than a single high-speed large-capacity server. Expanding deployment capacity only requires adding additional servers as needed, which may be lower than the overall cost of high-end hardware for a single machine. The trade-off is the increased complexity of infrastructure and deployment and maintenance.
• MongoDB supports horizontal expansion through sharding.

Three, the components included in the sharded cluster

The MongoDB
shard cluster contains the following components: ● Shard (storage) shard: Each shard contains a subset of shard data. Each shard can be deployed as a replica set.
●mongos (routing): mongos acts as a query router, providing an interface between client applications and sharded clusters.
●config servers ("scheduling" configuration): configure the metadata and configuration settings of the server storage cluster. Starting from MongoDB 3.4, the configuration server must be deployed as a replica set (CSRS).
The following figure describes the interaction of components in a sharded cluster:

 

Insert picture description here

Fourth, deploy MongoDB sharded cluster

1. Environmental preparation

Vmware virtual machine
centos7.6 192.168.110.132
mongodb3.2

2. The topology diagram is as follows

 

Insert picture description here

Create a data directory and log file for multiple instances;
modify the values ​​of ulimit -n and ulimit -u to 25000;

3. Environment construction

tar zxvf mongodb-linux-x86_64-3.2.1.tgz -C /opt/
mv mongodb-linux-x86_64-3.2.1/ /usr/local/mongodb
mkdir -p /data/mongodb/mongodb{1,2,3,4}
mkdir /data/mongodb/logstouch /data/mongodb/logs/mongodb{1,2,3,4}.log
chmod -R 777 /data/mongodb/logs/*.log
ulimit -n 25000        //临时修改 重启后失效   指定同一时间最多可打开的文件数
ulimit -u 25000        //临时修改 重启后失效   用户最多可启动的进程数目。
创建软连接：
ln -s /usr/local/mongodb/bin/mongo /usr/bin/mongo
ln -s /usr/local/mongodb/bin/mongod /usr/bin/mongod

4. config configuration server

cd /usr/local/mongodb/bin/
port=37017       端口
dbpath=/data/mongodb/mongodb1     数据存储位置logpath=/data/mongodb/logs/mongodb1.log   日志存储位置
logappend=true      mongos或mongod会将新条目附加到现有日志文件的末尾。
fork=true                   启用在后台运行mongos或mongod进程的守护进程模式                         
maxConns=5000       最大连接数
storageEngine=mmapv1     引擎configsvr=true     指定配置服务器
启动：mongod -f /usr/local/mongodb/bin/mongodb1.conf   '启动不起来  -f使用不了 改个环境变量'
mongo --port 27017   //查看端口是否正常开放

5. Shard server

cp -p mongodb1.conf mongodb2.conf
vim mongodb2.conf   port=47017
    dbpath=/data/mongodb/mongodb2
    logpath=/data/mongodb/logs/mongodb2.log
    logappend=true
    fork=true
    maxConns=5000
    storageEngine=mmapv1
    shardsvr=true
cp -p mongodb1.conf mongodb3.conf
vim mongodb3.conf
    port=47018
    dbpath=/data/mongodb/mongodb3
    logpath=/data/mongodb/logs/mongodb3.log
    logappend=true
    fork=true
    maxConns=5000
    storageEngine=mmapv1    shardsvr=true
    启动：mongod -f /usr/local/mongodb/bin/mongodb1.conf   '启动服务'

6. When a node has insufficient memory, allocate memory from other nodes

sysctl -w vm.zone_reclaim_mode=0
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

7. Start the routing server

–Port specifies the connection entrance of the other party 27017 –fork
background operation
–logpath specifies the storage path of the log file –
who is assigned by configdb

./mongos --port 27017 --fork --logpath=/usr/local/mongodb/bin/route.log --configdb 192.168.110.132:37017 --chunkSize 1

8. Set up fragmentation

 mongos> sh.addShard("192.168.110.132:47018");    添加分片服务器
{ "shardAdded" : "shard0000", "ok" : 1 }
mongos> sh.addShard("192.168.110.132:47017");     添加分片服务器
{ "shardAdded" : "shard0001", "ok" : 1 }
mongos> sh.status()      再次查看分片的服务器--- Sharding Status ---   sharding version: {    "_id" : 1,
    "minCompatibleVersion" : 5,
    "currentVersion" : 6,
    "clusterId" : ObjectId("5f5b73644e296224377d5e65")
}  shards:    {  "_id" : "shard0000",  "host" : "192.168.110.132:47018" }
    {  "_id" : "shard0001",  "host" : "192.168.110.132:47017" }
  active mongoses:    "3.2.1" : 1
  balancer:    Currently enabled:  yes    Currently running:  no
    Failed balancer rounds in last 5 attempts:  0
    Migration Results for the last 24 hours: 
        No recent migrations  databases:

9. Join the shard node

> use school
> for(var i=1;i<=10000;i++) db.users.insert({"id":i,"name":"tom"+i})
mongos> show dbsmongos> use schoolmongos> show collections      //查看集合
mongos> db.users.find().limit(5)   查看前五行
mongos> sh.status()          //查看数据库分片信息
'下面要开启集合的分片；'
'先对users集合创建索引；'
'然后对集合的索引进行分片。'
mongos> use shangmongos> db.users.createIndex({"id":1})       //对users表创建索引
mongos> sh.shardCollection("school.users",{"id":1})    //表分片给shard节点
mongos> sh.status()--- Sharding Status ---   sharding version: {    "_id" : 1,
    "minCompatibleVersion" : 5,
    "currentVersion" : 6,
    "clusterId" : ObjectId("5f5b73644e296224377d5e65")
}  shards:    {  "_id" : "shard0000",  "host" : "192.168.110.132:47018" }
    {  "_id" : "shard0001",  "host" : "192.168.110.132:47017" }
  active mongoses:    "3.2.1" : 1
  balancer:    Currently enabled:  yes
    Currently running:  no
    Failed balancer rounds in last 5 attempts:  0
    Migration Results for the last 24 hours: 
        1 : Success
  databases:    {  "_id" : "school",  "primary" : "shard0000",  "partitioned" : true }
        school.users            shard key: { "id" : 1 }
            unique: false
            balancing: true
            chunks:                shard0000   2
                shard0001   1
            { "id" : { "$minKey" : 1 } } -->> { "id" : 4682 } on : shard0001 Timestamp(2, 0) 
            { "id" : 4682 } -->> { "id" : 9364 } on : shard0000 Timestamp(2, 1) 
            { "id" : 9364 } -->> { "id" : { "$maxKey" : 1 } } on : shard0000 Timestamp(1, 2) 

Five, fragment management

1. Add data and perform fragmented storage

mongos> for(var i=1;i<=60000;i++) db.users2.insert({"id":i,"name":"tom"+i})
WriteResult({ "nInserted" : 1 })
mongos> db.users2.count()  60000
mongos> db.users2.createIndex({"id":1})   '添加索引'
{    "raw" : {
        "192.168.110.132:47018" : {
            "createdCollectionAutomatically" : false,
            "numIndexesBefore" : 1,
            "numIndexesAfter" : 2,
            "ok" : 1
        }    },    "ok" : 1
}mongos> mongos> sh.shardCollection("school.users2",{"id":1})   '添加分片存储'
{ "collectionsharded" : "school.users2", "ok" : 1 }
mongos> db.users2.stats(){    "sharded" : true,
    "paddingFactorNote" : "paddingFactor is unused and unmaintained in 3.0\. It remains hard coded to 1.0 for compatibility only.",
    "userFlags" : 1,
    "capped" : false,
    "ns" : "school.users2",
    "count" : 60000,
    "numExtents" : 12,
    "size" : 6720000,
    "storageSize" : 22364160,
    "totalIndexSize" : 3581088,
    "indexSizes" : {
        "_id_" : 1970416,
        "id_1" : 1610672
    },    "avgObjSize" : 112,
    "nindexes" : 2,
    "nchunks" : 13,
    "shards" : {
        "shard0000" : {
            "ns" : "school.users2",
            "count" : 31909,
            "size" : 3573808,
            "avgObjSize" : 112,
            "numExtents" : 6,
            "storageSize" : 11182080,
            "lastExtentSize" : 8388608,
            "paddingFactor" : 1,
            "paddingFactorNote" : "paddingFactor is unused and unmaintained in 3.0\. It remains hard coded to 1.0 for compatibility only.",
            "userFlags" : 1,
            "capped" : false,
            "nindexes" : 2,
            "totalIndexSize" : 1872304,
            "indexSizes" : {
                "_id_" : 1054704,
                "id_1" : 817600
            },            "ok" : 1
        },        "shard0001" : {
            "ns" : "school.users2",
            "count" : 28091,
            "size" : 3146192,
            "avgObjSize" : 112,
            "numExtents" : 6,
            "storageSize" : 11182080,
            "lastExtentSize" : 8388608,
            "paddingFactor" : 1,
            "paddingFactorNote" : "paddingFactor is unused and unmaintained in 3.0\. It remains hard coded to 1.0 for compatibility only.",
            "userFlags" : 1,
            "capped" : false,
            "nindexes" : 2,
            "totalIndexSize" : 1708784,
            "indexSizes" : {
                "_id_" : 915712,
                "id_1" : 793072
            },            "ok" : 1
        }    },    "ok" : 1
}mongos> 

2. Add tags

mongos> sh.addShardTag("shard0000","sales00")
mongos> sh.addShardTag("shard0001","sales01")

3. Connect to the config configuration server:

mongo --port 37017
configsvr> use config              //打开配置数据库
configsvr> show collections        //查看集合
configsvr> db.chunks.findOne()      //记录所有块的信息
configsvr> db.collections.find()    //分片集合信息
configsvr> db.databases.find()      //分片中所有数据库信息

4. Add/remove shard server

[root@localhost bin]# cp -p mongodb3.conf mongodb4.conf
[root@localhost bin]# vim mongodb4.conf 
port=47019
dbpath=/data/mongodb/mongodb4
logpath=/data/mongodb/logs/mongodb4.log
logappend=truefork=truemaxConns=5000
storageEngine=mmapv1shardsvr=true
[root@localhost bin]# ls /data/mongodb/
logs  mongodb1  mongodb2  mongodb3  mongodb4[root@localhost bin]# ls /data/mongodb/logs/
mongodb1.log  mongodb2.log  mongodb3.log  mongodb4.logmongod -f mongodb4.conf  '启动'

Add shard node:

mongos> sh.addShard("192.168.110.132:47019") //添加分片服务器
mongos> sh.status()    '查看分片信息'
--- Sharding Status --- 
  sharding version: {
    "_id" : 1,
    "minCompatibleVersion" : 5,
    "currentVersion" : 6,
    "clusterId" : ObjectId("5f5b73644e296224377d5e65")
}
  shards:
    {  "_id" : "shard0000",  "host" : "192.168.110.132:47018",  "tags" : [ "sales00" ] }
    {  "_id" : "shard0001",  "host" : "192.168.110.132:47017",  "tags" : [ "sales01" ] }
    {  "_id" : "shard0002",  "host" : "192.168.110.132:47019" }

Delete the shard node:

mongos> use admin
mongos> db.runCommand({"removeshard":"192.168.110.132:47019"})    //删除节点