Dependent Origination

A Cool Bug

Posted on: September 10, 2013

I got a request that the reporting weekly email is missing numbers from its users. Turned out a few days’ logs are missing in the system. I checked the cron jobs and cron running system that the sync of logs should be fine — I have manually run the sync logs scripts and it is working out fine too.

What is the problem that these syncing didn’t happen for the past few days?

From talking to other people, we realized here is what is going on. Our cron running system will run all executable scripts in a directory based on its frequency requirement (per directory). It could be the scripts before the one I need are having problems. We checked the first script using the cron log system. Nothing. Then the second script. Turned out it has been hanging for all these few days — all the cron processes are still there in the system and they couldn’t make progress at all. This is because the script needs to access some production machines but the production setup (dns service etc) has been changed in the past few days.

Solution is instead of saving files to a certain machine in production, we are uploading to s3. That way it will have nothing to do with future production upgrades.

Quite a cool experience when figured out, starting with so much confusion🙂

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

September 2013
M T W T F S S
« Aug   Oct »
 1
2345678
9101112131415
16171819202122
23242526272829
30  

Twitter

  • is looking at other people's intentions, not their capabilities, and feeling much happier every day :) 11 months ago
  • is planning on how to spend the next two weeks until the new year, at home. 11 months ago
  • is going to have human company for Thanksgiving; a rare event for the past like twenty years. 1 year ago
  • living by myself again; a strange feeling in a bustling city with thirteen million people 1 year ago
  • just came back home after sixteen days on the road; when i set out, the plan was a mere three-day trip 1 year ago

Flickr Photos

IMG_2057

IMG_2051

IMG_2048

IMG_2044

IMG_2042

IMG_2035

IMG_2033

IMG_2030

IMG_2028

IMG_2024

More Photos
%d bloggers like this: