Dependent Origination

A Cool Bug

Posted on: September 10, 2013

I got a request that the reporting weekly email is missing numbers from its users. Turned out a few days’ logs are missing in the system. I checked the cron jobs and cron running system that the sync of logs should be fine — I have manually run the sync logs scripts and it is working out fine too.

What is the problem that these syncing didn’t happen for the past few days?

From talking to other people, we realized here is what is going on. Our cron running system will run all executable scripts in a directory based on its frequency requirement (per directory). It could be the scripts before the one I need are having problems. We checked the first script using the cron log system. Nothing. Then the second script. Turned out it has been hanging for all these few days — all the cron processes are still there in the system and they couldn’t make progress at all. This is because the script needs to access some production machines but the production setup (dns service etc) has been changed in the past few days.

Solution is instead of saving files to a certain machine in production, we are uploading to s3. That way it will have nothing to do with future production upgrades.

Quite a cool experience when figured out, starting with so much confusion 🙂


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

September 2013
« Aug   Oct »


  • is reflecting on the past three years and reading the article on divided self aptly showing up at the top of my tweeter feed. 5 months ago
  • is having a nice feeling of connecting dots all over the place. they are now illuminating the same point. 6 months ago
  • Style Transfer is fun! TensorFlow rocks! #WTM17 8 months ago
  • couldn't get over the jetlag, sleeping during the day from seven to four, for days. 10 months ago
  • is emptying trash and happily discovering the available disk space now ranks at 100G+. 11 months ago

Flickr Photos

%d bloggers like this: