MongoDB backup and restore mongodump, mongorestore MongoDB backup and restore mongodump, mongorestore

MongoDB backup and restore mongodump, mongorestore

 

MongoDB Backup and Restore

A, MongoDB backup

MongoDB several ways to back up:

• mongodump
• System Snapshot (not described here in detail, the specific content of the official website: the Back Up with Filesystem Snapshots )
• cp or rsync

1、mongodump

mongodump official MongoDB is a backup tool, it can read data from the MongoDB database, file and generate BSON, mongodump suitable for backing up and restoring a small amount of data MongoDB database, not suitable for large data backup.

By default, the local database content mongodump not get inside.

mongodump only backup documents in the database, do not back up the index, so the reduction we need to rebuild the index.

Mongodump impact on performance during the backup process mongod services, we recommend to operate in a business-peak hours. If our data backup, greater than the system memory, when we backed error prone.

Mongodump in the implementation of, mongod service can still provide services, data can be modified, if we add parameters --oplog, then backs up to it, it will be recorded this time is oplog operation time if we want to restore the there are also logging, we can use mongorestore --oplogReplay recovery

1.1, mongodump commonly used commands and parameters

The official document: mongodump

Mongodump default output directory named dump, dump if the output directory path contains, will direct coverage. The default is no backup compression.

parameter:

--host <hostname><:port>, -h <hostname><:port>  # 指定备份的主机ip和端口号，默认值localhost:27017
--port # 指定端口号 默认27017

--username <username>, -u <username> # 指定用户名
--password <password>, -p <password> # 指定密码
--authenticationDatabase <dbname> # 指定认证的数据库
--authenticationMechanism <name> # 指定认证的算法 ，默认值 SCRAM-SHA-1
--db <database>, -d <database> # 指定备份的数据库，未指定的话，备份所有的数据库，但不包含local库
--collection <collection>, -c <collection> # 指定备份的集合，未指定则备份指定库中的所有集合。
--query <json>, -q <json>  # 指定 json 作为查询条件。来备份我们过滤后的数据。
--queryFile <path>  # 指定 json 文档路径，以该文档的内容作为查询条件，来备份我们过滤后的数据。
--quit # 通过抑制 MongoDB的复制，连接等活动，来实现备份。
--gzip  # 开启压缩，3.2版本后可以使用，输出为文件的话会带有后缀.gz
--out <path>, -o <path>  # 输出的目录路径

--repir # 修复数据时使用 下面有详细介绍
--oplog # mongodump 会将 mongodump 执行期间的 oplog 日志 输出到文件 oplog.bson，这就意味着从备份开始到备份结束的数据操作我们都可以记录下来。
--archive <file> # 输出到单个存档文件或者是直接输出。


--dumpDbUsersAndRoles # 只有在 使用 --db 时才适用，备份数据库的包含的用户和角色。
--excludeCollection string # 排除指定的集合，如果要排除多个，使用多个--excludeCollection 
--numParallelCollections int, -j int # 并行导出的集合数，默认为4


--ssl  # 指定 TLS/SSL 协议
--sslCAFile filename # 指定认证文件名
--sslPEMKeyFile <filename>
--sslPEMKeyPassword <value>
--sslCRLFile <filename>
--sslAllowInvalidCertificates
--sslAllowInvalidHostnames
--sslFIPSMode

Example:

1. Exclude the specified collection

   mongodump  --db test --excludeCollection=users --excludeCollection=salaries

2. Designated ip, port, username, authentication, compression, output directory

   mongodump --host mongodb1.example.net --port 37017 --username user --password "pass" --gzip --out /opt/backup/mongodump-2019-04-17

1.2, Note

When we delete a data set of this collection of 10,000 pieces of data, which is then Oplog have 10000 delete records.

Such an operation would lead to the previous Oplog inside the operating records are quickly covered.

So when we backed up, you need to pay attention to the backup interval time and recording time Oplog covered (ie Oplog records how long the log). To the latter outweighs the former.

Example:

Our backup plan is to back up your data daily, Oplog record hold time greater than one day.

Oplog opening is required to open replica set to open, so the above backup strategy is for the replica set.

2, cp or rsync

We can directly copy the data files, but we have to stop the operation of MongoDB before copying files, otherwise we copy the file is invalid.

3, after a single node shuts down unexpectedly, how to recover data

Note that the following is for a single node.

After an unexpected shutdown node, we re node, log errors have found the following:

2018-10-24T18:05:18.248-0400 W STORAGE [initandlisten] Detected unclean shutdown - mongod.lock is not empty. ... 2018-10-24T17:24:53.122-0400 E STORAGE [initandlisten] Failed to get the cursor for uri: table:collection-2-6854866147293273505 2018-10-24T17:24:53.122-0400 E STORAGE [initandlisten] This may be due to missing data files. ... ... ***aborting after fassert() failure

And there is a file in the data storage mongod.lock path.

We can use  mongod --repair to recover data.

In MongoDB 4.0.3 on WiredTiger engine, use this command to do the following

• Rebuild all indexes
• Discarding corrupted data
• Create a stub file metadata incomplete.

For MMAPv1 engine

• Rebuild all indexes
• Discarding corrupted data

Data recovery process:

1. First back up your existing data

   We can use the cp command to all files in an entire directory of all existing data backup copy.

2. Use mongod --repair

   # 针对 所有数据库
   mongod --repair 
   # 针对 单个数据库
   mongod --dbpath /opt/mongodb/data/djx  --repair

Under normal circumstances, you should not manually delete the mongod.lockfile. Instead, use the above procedure to restore the database. In severe cases, you can delete files, use may corrupt the file to start the database and try to recover the data from the database, but there is a risk.

4, view the backup data

We are all part of the binary data, we see the direct view is not, then we can bsondump (install MongoDB comes) through tools for viewing.

[root@djx uu_dev]# bsondump  order_detail.bson
2019-04-17T16:14:51.430+0800    549 objects found
{"_id":{"$oid":"5cae98d0f23d932dscdfc2ff"}}
{"_id":{"$oid":"5cae98d0f23d932dscdfca00"}}
{"_id":{"$oid":"5cae98d1f23d932dscdfca31"}}
{"_id":{"$oid":"5cae98d1f23d932dscdfca32"}}

Two, MongoDB reduction

1, mongorestore Features

• mongorestore You can create a new database or add data to an existing database, but  mongorestore only perform insert operations not performed  updatethe operation. This means that if you restore the document to an existing database, the existing documents in a database of _idvalues and documents you want to restore the _id values are the same, is not covered by the original value of the database.
• Rebuild the index, mongorestore rebuild indexes.
• mongorestore Not recover  system.profile the data

2, mongorestore common parameters

--help # 查看帮助
--quiet # 通过抑制 MongoDB的复制，连接等活动，来实现数据恢复。

--host <hostname><:port>, -h <hostname><:port>  # 指定恢复的主机ip和端口号，默认值localhost:27017
--port # 指定端口号 默认27017

--username <username>, -u <username> # 指定用户名
--password <password>, -p <password> # 指定密码
--authenticationDatabase <dbname> # 指定认证的数据库
--authenticationMechanism <name> # 指定认证的算法 ，默认值 SCRAM-SHA-1
--objcheck # 开启验证，验证还原操作，确保没有无效的文档插入数据库。会有较小的性能影响
--oplogReplay # 恢复备份数据并将 mongodump 执行期间的操作(记录在导出的日志)恢复。
--oplogLimit  # 指定恢复
--oplogFile # 指定 Oplog 路径
--keepIndexVersion # 阻止mongorestore在还原过程中将索引升级到最新版本。
--restoreDbUsersAndRoles # 还原指定的数据库用户和角色。
--maintainInsertionOrder # 默认值为False,如果为 True,mongorestore 将按照输入源的文档顺序插入，否则是 随机执行插入。
--numParallelCollections int, -j int # 指定并行恢复的集合数。
--numInsertionWorkersPerCollection int # 默认值为 1，指定每个集合恢复的并发数，大数据量导入增加该值可提高 恢复速度。
--gzip # 从压缩文档中 恢复。
--archive # 从归档文件中恢复。
--dir # 指定还原数据储存目录。

3, an example of reduction

mongorestore --collection people --db accounts dump/
mongorestore --host mongodb1.example.net --port 37017 --username user --password "pass" /opt/backup/mongodump-2011-10-24
mongorestore --gzip --archive=test.20150715.gz --db test
mongorestore --archive=test.20150715.archive --db test

MongoDB Backup and Restore

A, MongoDB backup

MongoDB several ways to back up:

• mongodump
• System Snapshot (not described here in detail, the specific content of the official website: the Back Up with Filesystem Snapshots )
• cp or rsync

1、mongodump

mongodump official MongoDB is a backup tool, it can read data from the MongoDB database, file and generate BSON, mongodump suitable for backing up and restoring a small amount of data MongoDB database, not suitable for large data backup.

By default, the local database content mongodump not get inside.

mongodump only backup documents in the database, do not back up the index, so the reduction we need to rebuild the index.

Mongodump impact on performance during the backup process mongod services, we recommend to operate in a business-peak hours. If our data backup, greater than the system memory, when we backed error prone.

Mongodump in the implementation of, mongod service can still provide services, data can be modified, if we add parameters --oplog, then backs up to it, it will be recorded this time is oplog operation time if we want to restore the there are also logging, we can use mongorestore --oplogReplay recovery

1.1, mongodump commonly used commands and parameters

The official document: mongodump

Mongodump default output directory named dump, dump if the output directory path contains, will direct coverage. The default is no backup compression.

parameter:

--host <hostname><:port>, -h <hostname><:port>  # 指定备份的主机ip和端口号，默认值localhost:27017
--port # 指定端口号 默认27017

--username <username>, -u <username> # 指定用户名
--password <password>, -p <password> # 指定密码
--authenticationDatabase <dbname> # 指定认证的数据库
--authenticationMechanism <name> # 指定认证的算法 ，默认值 SCRAM-SHA-1
--db <database>, -d <database> # 指定备份的数据库，未指定的话，备份所有的数据库，但不包含local库
--collection <collection>, -c <collection> # 指定备份的集合，未指定则备份指定库中的所有集合。
--query <json>, -q <json>  # 指定 json 作为查询条件。来备份我们过滤后的数据。
--queryFile <path>  # 指定 json 文档路径，以该文档的内容作为查询条件，来备份我们过滤后的数据。
--quit # 通过抑制 MongoDB的复制，连接等活动，来实现备份。
--gzip  # 开启压缩，3.2版本后可以使用，输出为文件的话会带有后缀.gz
--out <path>, -o <path>  # 输出的目录路径

--repir # 修复数据时使用 下面有详细介绍
--oplog # mongodump 会将 mongodump 执行期间的 oplog 日志 输出到文件 oplog.bson，这就意味着从备份开始到备份结束的数据操作我们都可以记录下来。
--archive <file> # 输出到单个存档文件或者是直接输出。


--dumpDbUsersAndRoles # 只有在 使用 --db 时才适用，备份数据库的包含的用户和角色。
--excludeCollection string # 排除指定的集合，如果要排除多个，使用多个--excludeCollection 
--numParallelCollections int, -j int # 并行导出的集合数，默认为4


--ssl  # 指定 TLS/SSL 协议
--sslCAFile filename # 指定认证文件名
--sslPEMKeyFile <filename>
--sslPEMKeyPassword <value>
--sslCRLFile <filename>
--sslAllowInvalidCertificates
--sslAllowInvalidHostnames
--sslFIPSMode

Example:

1. Exclude the specified collection

   mongodump  --db test --excludeCollection=users --excludeCollection=salaries

2. Designated ip, port, username, authentication, compression, output directory

   mongodump --host mongodb1.example.net --port 37017 --username user --password "pass" --gzip --out /opt/backup/mongodump-2019-04-17

1.2, Note

When we delete a data set of this collection of 10,000 pieces of data, which is then Oplog have 10000 delete records.

Such an operation would lead to the previous Oplog inside the operating records are quickly covered.

So when we backed up, you need to pay attention to the backup interval time and recording time Oplog covered (ie Oplog records how long the log). To the latter outweighs the former.

Example:

Our backup plan is to back up your data daily, Oplog record hold time greater than one day.

Oplog opening is required to open replica set to open, so the above backup strategy is for the replica set.

2, cp or rsync

We can directly copy the data files, but we have to stop the operation of MongoDB before copying files, otherwise we copy the file is invalid.

3, after a single node shuts down unexpectedly, how to recover data

Note that the following is for a single node.

After an unexpected shutdown node, we re node, log errors have found the following:

2018-10-24T18:05:18.248-0400 W STORAGE [initandlisten] Detected unclean shutdown - mongod.lock is not empty. ... 2018-10-24T17:24:53.122-0400 E STORAGE [initandlisten] Failed to get the cursor for uri: table:collection-2-6854866147293273505 2018-10-24T17:24:53.122-0400 E STORAGE [initandlisten] This may be due to missing data files. ... ... ***aborting after fassert() failure

And there is a file in the data storage mongod.lock path.

We can use  mongod --repair to recover data.

In MongoDB 4.0.3 on WiredTiger engine, use this command to do the following

• Rebuild all indexes
• Discarding corrupted data
• Create a stub file metadata incomplete.

For MMAPv1 engine

• Rebuild all indexes
• Discarding corrupted data

Data recovery process:

1. First back up your existing data

   We can use the cp command to all files in an entire directory of all existing data backup copy.

2. Use mongod --repair

   # 针对 所有数据库
   mongod --repair 
   # 针对 单个数据库
   mongod --dbpath /opt/mongodb/data/djx  --repair

Under normal circumstances, you should not manually delete the mongod.lockfile. Instead, use the above procedure to restore the database. In severe cases, you can delete files, use may corrupt the file to start the database and try to recover the data from the database, but there is a risk.

4, view the backup data

We are all part of the binary data, we see the direct view is not, then we can bsondump (install MongoDB comes) through tools for viewing.

[root@djx uu_dev]# bsondump  order_detail.bson
2019-04-17T16:14:51.430+0800    549 objects found
{"_id":{"$oid":"5cae98d0f23d932dscdfc2ff"}}
{"_id":{"$oid":"5cae98d0f23d932dscdfca00"}}
{"_id":{"$oid":"5cae98d1f23d932dscdfca31"}}
{"_id":{"$oid":"5cae98d1f23d932dscdfca32"}}

Two, MongoDB reduction

1, mongorestore Features

• mongorestore You can create a new database or add data to an existing database, but  mongorestore only perform insert operations not performed  updatethe operation. This means that if you restore the document to an existing database, the existing documents in a database of _idvalues and documents you want to restore the _id values are the same, is not covered by the original value of the database.
• Rebuild the index, mongorestore rebuild indexes.
• mongorestore Not recover  system.profile the data

2, mongorestore common parameters

--help # 查看帮助
--quiet # 通过抑制 MongoDB的复制，连接等活动，来实现数据恢复。

--host <hostname><:port>, -h <hostname><:port>  # 指定恢复的主机ip和端口号，默认值localhost:27017
--port # 指定端口号 默认27017

--username <username>, -u <username> # 指定用户名
--password <password>, -p <password> # 指定密码
--authenticationDatabase <dbname> # 指定认证的数据库
--authenticationMechanism <name> # 指定认证的算法 ，默认值 SCRAM-SHA-1
--objcheck # 开启验证，验证还原操作，确保没有无效的文档插入数据库。会有较小的性能影响
--oplogReplay # 恢复备份数据并将 mongodump 执行期间的操作(记录在导出的日志)恢复。
--oplogLimit  # 指定恢复
--oplogFile # 指定 Oplog 路径
--keepIndexVersion # 阻止mongorestore在还原过程中将索引升级到最新版本。
--restoreDbUsersAndRoles # 还原指定的数据库用户和角色。
--maintainInsertionOrder # 默认值为False,如果为 True,mongorestore 将按照输入源的文档顺序插入，否则是 随机执行插入。
--numParallelCollections int, -j int # 指定并行恢复的集合数。
--numInsertionWorkersPerCollection int # 默认值为 1，指定每个集合恢复的并发数，大数据量导入增加该值可提高 恢复速度。
--gzip # 从压缩文档中 恢复。
--archive # 从归档文件中恢复。
--dir # 指定还原数据储存目录。

3, an example of reduction

mongorestore --collection people --db accounts dump/
mongorestore --host mongodb1.example.net --port 37017 --username user --password "pass" /opt/backup/mongodump-2011-10-24
mongorestore --gzip --archive=test.20150715.gz --db test
mongorestore --archive=test.20150715.archive --db test