Dependent Origination

Archive for September 2012

I have encountered a ‘cannot find reference to function’ error from the linker today and couldn’t figure out why. Thanks to a colleague who recently worked on the build system — we figured out why.

library file:

namespace A {

namespace {

struct D { definition; }


return_type example_function(const vector<D>&);


test file:

#include “library.h”

vector<D> v;


this test cannot pass linker because linker would complain it couldn’t find reference to example_function.

How did we figure out?

1. grep through the actual compile command to see the *.o file is in it

2. use nm on the *.o file so we can see the function symbol is in the object file

3. clearly this is about parameter types now — i confirmed it passing a simple type

4. then colleague realizes that it is because of the unnamed space — it is assigned to a random number while compiling the library — and another random number assigned while compiling the test — so the two symbols won’t match up in the symbol table and linker couldn’t find the prototype of the function


1. take out the unnamed space or give it a name

php> $a = ”

php> = count($a)


php> $a = array()

php> = count($a)


I was trying to add a new shard in the mongos, but came across this error repeatedly. From google, it says mongos (before a certain version, 2.2.11?) refuse to take in the new information if your first try missed a port. The solution? Google says quit and restart mongos. But in my case, I think you have to change some configuration of the current replica set to make mongos think it needs to reconsider the information it receives and compare with existing information it knows — it is not enough only restart and give it the correct information, you have to change the replica set/shard somehow too. In fact, in my case, that didn’t exactly work immediately either. But after a while, twenty minutes at least, I restarted mongos without doing anything different from the previous run but successfully added the shard/replica set.

Moral of the story: when you add a shard/replica set, do it right the first time, to save yourself all these frustrations. Here is the post where I figured out what I should do. It also has lots of commands you can learn from, definitely not a newbie.

I did something stupid yesterday morning, apparently after only five or six hours’ sleep, that I terminated some EC2 instances that are still being used as shard servers — that is their ip’s are still stored in the main config database of our MongoDB setup. This has caused a major headache that took me an entire day to resolve.

First of all, restarting the instance with its ebs (luckily I did not delete the disk image 🙂 will not get back the original IP’s.

Apparently this post says you can go to the config database to change the ip addresses of the shard servers so when you run db.runCommand({listShards:1}) you will see shards with newly updated IP’s. Unfortunately it didn’t work for me because the config db (on the shard that is up and running) for some reason refused to show any existing collections (or data) to me. Not sure why. Otherwise from the post it sounded like the author went into those collections and modified them directly.

Other sites mentioned other ways to fix it — such as adding your new instance’s IP into the old replica set and then remove the old one — however, this requires that the old replica set is still running by this time. Since I terminated the instances and their ip changed, I had no way of restarting the replica set without mongo complaining about mismatched config, for example i cannot give a new name as another replica set  (it complains that the new name and old name mismatches), i cannot start it as the old name either (it complains that self is not in the replica set). Despite it being a replica set, with data directories only, apparently it stores some config information with itself.

So I had only one way out: I restarted the good shard as a single-sharded (one replica set) setup so production traffic can go ahead without any more write errors (since mongos is still distributing writes to the missing shard it has lots of write error — more on this later, in fact, i enabled sharding without actually carrying out the sharding command so there were no real data on the missing shard but still mongo refused to operate).

About the missing shard, it is really hard to start a fresh setup from their data directories (I originally planned to start new instances on them and use mongoexport to export the data and then mongoimport into the good shard). The plan never worked because, as described before, some config information was stored together with the data. Luckily after three hours, I figured out that I never really sharded the data so I don’t have to restore the data. Without this revelation, this would still be a disaster. But now it is merely a mishap 🙂

Here is another post I found useful while trying to modify IP’s in the existing config database. There are other suggestions in the thread, which was not applicable to my case but could be useful to yours.

Found in the log of the mongo shards after some config mishap and prolonged write error caused by a missing shard.

Do as suggested — either drop index and re-create them or use the ReIndex method.

mongo –port 10000 –host your_mongo_instance



September 2012
« Aug   Oct »


  • couldn't get over the jetlag, sleeping during the day from seven to four, for days. 1 month ago
  • is emptying trash and happily discovering the available disk space now ranks at 100G+. 2 months ago
  • is looking at other people's intentions, not their capabilities, and feeling much happier every day :) 1 year ago
  • is planning on how to spend the next two weeks until the new year, at home. 1 year ago
  • is going to have human company for Thanksgiving; a rare event for the past like twenty years. 1 year ago

Flickr Photos











More Photos