Tag Archives: remove all secondary members from replica set

Remove members from Replica Set

I had to deal with it, since the storage used by our three-members Replica Set needed to be redesigned and to achieve this I needed to remove practicaly 2 members. The replica set had to stay online only with one Primary node. The guys from 10Gen explain in this link, a step-by-step method to remove ONE member from three. When you are left with 2 members in a Replica Set, it is not the best option, but also not the worst. You can attach a witness which can decide between the two of them which one will become Primary, when one node fails. So the idea is that, if a node fails (shuts down, powers off, etc.. ) the other node which is online will have the SECONDARY status when you issue rs.status() and the writes will be prohibited. You will only be able to query the mongod databases.

And this is why I started this post. The mongodb creators suggest 2 methods to remove one member from a Replica Set. I will describe below step-by-step a workaround method to remove 2 Secondary members from a three members Replica Set. Finally you will have one member Replica Set.

  • Issue db.isMaster() on any node to know which is primary and which are secondary members and to get their host names. You will see something like this:
  •  

    RS:PRIMARY> db.isMaster()
    {
            "setName" : "RS",
            "ismaster" : true,
            "secondary" : false,
            "hosts" : [
                    "mongodb-01:27017",
                    "mongodb-03:27017",
                    "mongodb-02:27017"
            ],
            "primary" : "mongodb-01:27017" 
    ...}
    
  • Login to first Secondary member you want to remove. I will do it first for mongodb-02:27017.
  • Connect to the mongod instance of mongodb-02 and issue db.shutdownServer()
  • Check that it’s down. Issue a Linux bash command: ps -Aef | grep mongod, and you should not see any mongod processes started.
  • Go to Primary node and issue rs.conf() and you will see that the server is still included in Replica Set configuration.
  •  

     "_id" : "RSProd",
            "version" : 4,
            "members" : [
                    {
                            "_id" : 0,
                            "host" : "mongodb-01:27017",                        
                    },
                    {
                            "_id" : 1,
                            "host" : "mongodb-02:27017"
                    },
                    {
                            "_id" : 2,
                            "host" : "mongodb-03:27017"
                    }
            ]
    
  • To remove it issue rs.remove(“mongodb-02:27017”)
    After that, when you issue again rs.conf() you will get:
  •  

     "_id" : "RSProd",
            "version" : 4,
            "members" : [
                    {
                            "_id" : 0,
                            "host" : "mongodb-01:27017",                        
                    },
                    {
                            "_id" : 1,
                            "host" : "mongodb-03:27017"
                    }
            ]
    

    In order to remove also the mongodb-03:27017 node from the remaining 2 server Replica Set, the above method will not work since, if one server from two is down, the mongodb setting is configured to not allow any writes to the remaining online server. So you need to do it with the all servers online.

  • Issue on Primary:
  • cfg = rs.conf()
    cfg.members.splice(1,1)  // first 1 in the brackets represents the id of the node from rs.conf(). 
    rs.reconfig(cfg) 
    rs.conf()
     "_id" : "RSProd",
            "version" : 5,
            "members" : [
                    {
                            "_id" : 0,
                            "host" : "mongodb-01:27017",                        
                    }             
            ]
    
  • Now you can go and stop the mongod service from mongodb-03:27017. Issue the bash command:
  • sudo service mongod stop