MongoDB Advanced

aggregate aggregate

• Aggregate is mainly used to calculate data, similar to sum() and avg() in sql
• grammar

db.集合名称.aggregate([{管道:{表达式}}])

pipeline

• Pipes are generally used in Unix and Linux to use the output of the current command as the input of the next command

ps ajx | grep mongo

• In mongodb, the pipeline has the same function. After the document is processed, the next processing is carried out through the pipeline.
• Commonly used pipes
  • $group: Group the documents in the collection, which can be used for statistical results
  • $match: filter data and only output documents that meet the conditions
  • $project: Modify the structure of the input document, such as renaming, adding, deleting fields, creating calculation results
  • $sort: Sort the input documents and output them
  • $limit: limit the number of documents returned by the aggregation pipeline
  • $skip: Skip the specified number of documents and return the remaining documents
  • $unwind: split the fields of the array type

expression

• Process input documents and output
• grammar

表达式:'$列名'

• Common expressions
  • $sum: Calculate the sum, $sum:1 and count mean count
  • $avg: Calculate the average
  • $min: get the minimum value
  • $max: get the maximum value
  • $push: Insert values ​​into an array in the resulting document
  • $first: Get the first document data according to the sorting of resource documents
  • $last: Get the last document data according to the sorting of resource documents

$group

• Groups documents in a collection, which can be used for statistical results
• _id indicates the basis for grouping, and the format of a field is '$field'
• Example 1: Count the total number of boys and girls

db.stu.aggregate([
    {$group:
        {
            _id:'$gender',
            counter:{$sum:1}
        }
    }
])

Group by null

• Group all documents in a collection into a group
• Example 2: Find the total number of students and their average age

db.stu.aggregate([
    {$group:
        {
            _id:null,
            counter:{$sum:1},
            avgAge:{$avg:'$age'}
        }
    }
])

pivot data

• Example 3: Statistics of student gender and student name

db.stu.aggregate([
    {$group:
        {
            _id:'$gender',
            name:{$push:'$name'}
        }
    }
])

• Use $$ROOT to add the content of the document to the array of the result set, the code is as follows

db.stu.aggregate([
    {$group:
        {
            _id:'$gender',
            name:{$push:'$$ROOT'}
        }
    }
])

$match

• Used to filter data and only output documents that meet the conditions
• Use MongoDB's standard query operations
• Example 1: Query students whose age is greater than 20

db.stu.aggregate([
    {$match:{age:{$gt:20}}}
])

• Example 2: Query the number of boys and girls older than 20

db.stu.aggregate([
    {$match:{age:{$gt:20}}},
    {$group:{_id:'$gender',counter:{$sum:1}}}
])

$project

• Modify the structure of the input document, such as renaming, adding, deleting fields, creating calculation results
• Example 1: Query the student's name and age

db.stu.aggregate([
    {$project:{_id:0,name:1,age:1}}
])

• Example 2: Query the number of boys and girls, and output the number of people

db.stu.aggregate([
    {$group:{_id:'$gender',counter:{$sum:1}}},
    {$project:{_id:0,counter:1}}
])

$sort

• Sort the input documents and output them
• Example 1: Query student information, in ascending order by age

b.stu.aggregate([{$sort:{age:1}}])

• Example 2: Query the number of boys and girls, in descending order

db.stu.aggregate([
    {$group:{_id:'$gender',counter:{$sum:1}}},
    {$sort:{counter:-1}}
])

$limit

• Limit the number of documents returned by the aggregation pipeline
• Example 1: Query 2 pieces of student information

db.stu.aggregate([{$limit:2}])

$skip

• Skip the specified number of documents and return the remaining documents
• Example 2: Query student information starting from item 3

db.stu.aggregate([{$skip:2}])

• Example 3: Count the number of boys and girls, in ascending order of number, take the second data

db.stu.aggregate([
    {$group:{_id:'$gender',counter:{$sum:1}}},
    {$sort:{counter:1}},
    {$skip:1},
    {$limit:1}
])

• Note the order: write skip first, then limit

$unwind

• Split an array type field in the document into multiple entries, each containing a value in the array

Grammar 1

• Split a field value

db.集合名称.aggregate([{$unwind:'$字段名称'}])

• structure data

db.t2.insert({_id:1,item:'t-shirt',size:['S','M','L']})

• Inquire

db.t2.aggregate([{$unwind:'$size'}])

Grammar 2

• Split a field value
• Handling empty arrays, not arrays, no fields, null cases

db.inventory.aggregate([{
    $unwind:{
        path:'$字段名称',
        preserveNullAndEmptyArrays:<boolean>#防止数据丢失
    }
}])

• structure data

db.t3.insert([
{ "_id" : 1, "item" : "a", "size": [ "S", "M", "L"] },
{ "_id" : 2, "item" : "b", "size" : [ ] },
{ "_id" : 3, "item" : "c", "size": "M" },
{ "_id" : 4, "item" : "d" },
{ "_id" : 5, "item" : "e", "size" : null }
])

• query using syntax 1

db.t3.aggregate([{$unwind:'$size'}])

• Looking at the query results, it is found that documents with empty arrays, no fields, and null are discarded.
• Question: How can we not throw it away?
• Answer: query using syntax 2

db.t3.aggregate([{$unwind:{path:'$sizes',preserveNullAndEmptyArrays:true}}])

super administrator

• In order to access mongodb more securely, the visitor is required to provide a username and password, so a user needs to be created in mongodb
• The role-user-database security management method is adopted
• Common system roles are as follows:
  • root: only available in the admin database, super account, super privilege
  • Read: Allows the user to read the specified database
  • readWrite: Allows users to read and write to the specified database
• Create a super administrative user

use admin
db.createUser({
    user:'admin',
    pwd:'123',
    roles:[{role:'root',db:'admin'}]
})

Enable security authentication

• Modify the configuration file

sudo vi /etc/mongod.conf

• enable authentication
• Note: A space must be added between keys and values, otherwise parsing will report an error

security:
  authorization: enabled

• restart the service

sudo service mongod stop
sudo service mongod start

• terminal connection

 mongo -u 'admin' -p '123' --authenticationDatabase 'admin'

Ordinary user management

• Log in with the super administrator, and then enter the user management operation
• View users of the current database

use test1
show users

• Create a normal user

db.createUser({
    user:'t1',
    pwd:'123',
    roles:[{role:'readWrite',db:'test1'}]
})

• terminal connection

mongo -u t1 -p 123 --authenticationDatabase test1

• Switch the database, execute the command to view the effect

• Modify user: you can modify pwd, roles attributes

db.updateUser('t1',{pwd:'456'})

replication (replica set)

what is replication

• Replication provides redundant backup of data and stores data copies on multiple servers, improving data availability and ensuring data security
• Replication also allows data recovery from hardware failures and service interruptions

why copy

• data backup
• data disaster recovery
• read-write separation
• High (24*7) data availability
• No downtime maintenance
• Replica sets are transparent to the application

How replication works

• Replication requires at least two nodes A, B...
• A is the master node, responsible for processing client requests
• The rest are slave nodes, responsible for replicating the data on the master node
• The common collocation methods of nodes are: one master and one slave, one master and multiple slaves
• The master node records all operations on it, and the slave node periodically polls the master node to obtain these operations, and then performs these operations on its own data copy, thereby ensuring that the data of the slave node is consistent with the master node
• Data interaction between master node and slave node ensures data consistency

Features of replication

• A cluster of N nodes
• Any node can be the master node
• All writes are on the primary node
• automatic failover
• Automatic recovery

Set up replication nodes

• The next operation needs to open multiple terminal windows, and may connect multiple ubuntu hosts, which will be a bit messy. It is recommended to implement it in xshell
• step1: Create database directories t1, t2
• Demonstrate in the Desktop directory, other directories are also available, just pay attention to the permissions

mkdir t1
mkdir t2

• step2: Start mongod using the following format, note that the name of the replSet is the same

mongod --bind_ip 192.168.196.128 --port 27017 --dbpath ~/Desktop/t1 --replSet rs0
mongod --bind_ip 192.168.196.128 --port 27018 --dbpath ~/Desktop/t2 --replSet rs0

• step3: Connect to the main server, set 192.168.196.128:27017 as the main server here

mongo --host 192.168.196.128 --port 27017

• step4: initialization

rs.initiate()

• After the initialization is complete, the prompt is as follows:

• step5: View the current status

rs.status()

• The current status is as follows:

• step6: add replica set

rs.add('192.168.196.128:27018')

• step7: After the replica set is added successfully, the current status is as follows:

• step8: connect to the second mongo service

mongo --host 192.168.196.128 --port 27018

• After the connection is successful, the prompt is as follows:

• step9: insert data into the main server

use test1
for(i=0;i<10;i++){db.t1.insert({_id:i})}
db.t1.find()

• step10: Insert query from the server
• Description: If you are reading from the server, you need to set rs.slaveOk()

rs.slaveOk()
db.t1.find()

other instructions

• delete slave node

rs.remove('192.168.196.128:27018')

• After shutting down the master server and restarting it, you will find that the original slave server has become a slave server, and the newly started server (original slave server) has become a slave server

backup

• grammar

mongodump -h dbhost -d dbname -o dbdirectory

• -h: server address, you can also specify the port number
• -d: The name of the database to be backed up
• -o: The backup data storage location, this directory stores the backed up data
• Example 1

sudo mkdir test1bak
sudo mongodump -h 192.168.196.128:27017 -d test1 -o ~/Desktop/test1bak

recover

• grammar

mongorestore -h dbhost -d dbname --dir dbdirectory

• -h: server address
• -d: The database instance that needs to be restored
• --dir: the location of the backup data
• Example 2

mongorestore -h 192.168.196.128:27017 -d test2 --dir ~/Desktop/test1bak/test1

interact with python

• Click to view the official document
• install python package

进入虚拟环境
sudo pip install pymongo
或源码安装
python setup.py

• Import package pymongo

import pymongo

• connect, create client

client=pymongo.MongoClient("localhost", 27017)

• Get database test1

db=client.test1

• get collection stu

stu = db.stu

• Add documents

s1={name:'gj',age:18}
s1_id = stu.insert_one(s1).inserted_id

• find a document

s2=stu.find_one()

• Find multiple documents 1

for cur in stu.find():
    print cur

• Find multiple documents 2

cur=stu.find()
cur.next()
cur.next()
cur.next()

• Get the number of documents

print stu.count()