Category Archives: Mongo DB

Remove members from Replica Set

I had to deal with it, since the storage used by our three-members Replica Set needed to be redesigned and to achieve this I needed to remove practicaly 2 members. The replica set had to stay online only with one Primary node. The guys from 10Gen explain in this link, a step-by-step method to remove ONE member from three. When you are left with 2 members in a Replica Set, it is not the best option, but also not the worst. You can attach a witness which can decide between the two of them which one will become Primary, when one node fails. So the idea is that, if a node fails (shuts down, powers off, etc.. ) the other node which is online will have the SECONDARY status when you issue rs.status() and the writes will be prohibited. You will only be able to query the mongod databases.

And this is why I started this post. The mongodb creators suggest 2 methods to remove one member from a Replica Set. I will describe below step-by-step a workaround method to remove 2 Secondary members from a three members Replica Set. Finally you will have one member Replica Set.

  • Issue db.isMaster() on any node to know which is primary and which are secondary members and to get their host names. You will see something like this:
  •  

    RS:PRIMARY> db.isMaster()
    {
            "setName" : "RS",
            "ismaster" : true,
            "secondary" : false,
            "hosts" : [
                    "mongodb-01:27017",
                    "mongodb-03:27017",
                    "mongodb-02:27017"
            ],
            "primary" : "mongodb-01:27017" 
    ...}
    
  • Login to first Secondary member you want to remove. I will do it first for mongodb-02:27017.
  • Connect to the mongod instance of mongodb-02 and issue db.shutdownServer()
  • Check that it’s down. Issue a Linux bash command: ps -Aef | grep mongod, and you should not see any mongod processes started.
  • Go to Primary node and issue rs.conf() and you will see that the server is still included in Replica Set configuration.
  •  

     "_id" : "RSProd",
            "version" : 4,
            "members" : [
                    {
                            "_id" : 0,
                            "host" : "mongodb-01:27017",                        
                    },
                    {
                            "_id" : 1,
                            "host" : "mongodb-02:27017"
                    },
                    {
                            "_id" : 2,
                            "host" : "mongodb-03:27017"
                    }
            ]
    
  • To remove it issue rs.remove(“mongodb-02:27017”)
    After that, when you issue again rs.conf() you will get:
  •  

     "_id" : "RSProd",
            "version" : 4,
            "members" : [
                    {
                            "_id" : 0,
                            "host" : "mongodb-01:27017",                        
                    },
                    {
                            "_id" : 1,
                            "host" : "mongodb-03:27017"
                    }
            ]
    

    In order to remove also the mongodb-03:27017 node from the remaining 2 server Replica Set, the above method will not work since, if one server from two is down, the mongodb setting is configured to not allow any writes to the remaining online server. So you need to do it with the all servers online.

  • Issue on Primary:
  • cfg = rs.conf()
    cfg.members.splice(1,1)  // first 1 in the brackets represents the id of the node from rs.conf(). 
    rs.reconfig(cfg) 
    rs.conf()
     "_id" : "RSProd",
            "version" : 5,
            "members" : [
                    {
                            "_id" : 0,
                            "host" : "mongodb-01:27017",                        
                    }             
            ]
    
  • Now you can go and stop the mongod service from mongodb-03:27017. Issue the bash command:
  • sudo service mongod stop

Automated MongoDB backup

I recently had to setup an automated job to backup some mongo databases. Here it is how you do it.

1. Create users for each db that needs to be backed up with full rights in that database. Pay attention to NOT create the user in admin database and allow full rights for all other user db’s. Even if you will be able to connect with that user and query the collections for other databases, you will not be able to use mongodump.

2. Write in shell file with .sh extension following script:

#!/bin/bash
# Dump Mongo Dbs database every night using mongodump
# Author: by yrushka.com

BAK="/opt/backup"

## Binary path ##
#MONGO="/usr/bin/mongo"
#MONGODUMP="/usr/bin/mongodump"

echo $BAK

/usr/bin/mongodump  --port 27017 --username user1 -password user1pass --db db1 --out $BAK/`date +"%Y_%m_%d__"`db1
/usr/bin/mongodump  --port 27017 --username user2 -password user2pass --db db2 --out $BAK/`date +"%Y_%m_%d__"`db2
/usr/bin/mongodump  --port 27017 --username user3 -password user3pass --db db3 --out $BAK/`date +"%Y_%m_%d__"`db3

3. Create a task in CRON that will execute the above script to a specific schedule.

And that’s it.