Dependent Origination

A Cool Bug

Posted on: September 10, 2013

I got a request that the reporting weekly email is missing numbers from its users. Turned out a few days’ logs are missing in the system. I checked the cron jobs and cron running system that the sync of logs should be fine — I have manually run the sync logs scripts and it is working out fine too.

What is the problem that these syncing didn’t happen for the past few days?

From talking to other people, we realized here is what is going on. Our cron running system will run all executable scripts in a directory based on its frequency requirement (per directory). It could be the scripts before the one I need are having problems. We checked the first script using the cron log system. Nothing. Then the second script. Turned out it has been hanging for all these few days — all the cron processes are still there in the system and they couldn’t make progress at all. This is because the script needs to access some production machines but the production setup (dns service etc) has been changed in the past few days.

Solution is instead of saving files to a certain machine in production, we are uploading to s3. That way it will have nothing to do with future production upgrades.

Quite a cool experience when figured out, starting with so much confusion 🙂


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

September 2013
« Aug   Oct »


Flickr Photos

%d bloggers like this: