M202 July 2014 Fifth and sixth week

Sharded cluster maintenance

Cluster maintenance is unavoidable, it usually means correcting an unhealthy state of the cluster or an upgrade. Adding shards also counts as maintenance, but this topic is covered in M102 and understood to be common knowledge for M202 participants. In case it was forgotten, a quick view through the sharding helper commands shows the addShard function which needs an URI and uses it to call the addShard database command. Calling the database command directly will give you more control, but the helper command gives a sensible default.

Upgrading a sharded cluster

The upgrade process of any environment usually involves reading through the documentation and trying it out before starting the upgrade in production environment. Upgrades to a new minor version, for instance 2.4 to 2.6, should definitely start with reading through the release notes. There is still a lot of significant change for MongoDB to simply ignore this step.

Once you read the release notes the order of upgrades in the cluster is straightforward. First you disable the balancer before you upgrade the mongos processes. Then you upgrade the config servers and finally the shards. Because it’s recommended to use replica sets in production you can follow the steps of upgrading replica sets. To speed up the process you could upgrade multiple secondary nodes across replica sets at the same time. At the end you should enable the balancer again.

Removing a shard

Removing a shard from a cluster is harder than adding one. This is because of the steps involved in doing so. The process starts with moving all data of it. You need to ensure the balancer is enabled, because it will be used to migrate chunks to other shards. Once all sharded data is moved off the shard you need to move the rest. Sometimes you have databases and collections that aren’t sharded, by telling the cluster another shard is going to be the primary shard for the data these collections will also be moved off the shard. Once all is finished you can shut down the shard.

Log files

MongoDB logs a lot of information to it’s log files. Ranging from connections and configuration to slow queries (when configured to do so). But dealing with that data is cumbersome. The videos hint that week 7 will have some graphical tools to process data. If you understand the basic Unix shell commands and some scripting languages you already got enough to set up your own analysis tools.

One of week six’ homework assignments was about going through log files to identify a problem. Because some parts of the log contain JSON, I added jq to my library of tools to get a better overview of what’s going on. Of course not everything is JSON, so I had to grep and cut a little until I got the desired input string for jq. All the cutting makes you lose context, thankfully my attention span is long enough to remember what I was looking for.


During the fifth and sixth week I discovered that version 2.6 of MongoDB has more improvements than I previously thought. Updates from MongoDB reach my inbox, but clearly some aspects didn’t get through until I picked up this course. Now I think there’s definitely a reason to sign up again for a development course and the updated M102 DBA course. The tedious task of setting up a replica set or even a sharded cluster just for testing is greatly simplified with the ReplicaTest and ShardingTest objects in the mongo shell. Although a little foggy in documentation they have a clear advantage over doing it all by hand. Then I saw there’s a Bulk command in the shell. This also helps to speed things up, but there’s still a need for some loops to fill up the set of commands. The technology to test and teach really has improved in the last year.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s