Dependent Origination

A Cool Bug

Posted on: September 10, 2013

I got a request that the reporting weekly email is missing numbers from its users. Turned out a few days’ logs are missing in the system. I checked the cron jobs and cron running system that the sync of logs should be fine — I have manually run the sync logs scripts and it is working out fine too.

What is the problem that these syncing didn’t happen for the past few days?

From talking to other people, we realized here is what is going on. Our cron running system will run all executable scripts in a directory based on its frequency requirement (per directory). It could be the scripts before the one I need are having problems. We checked the first script using the cron log system. Nothing. Then the second script. Turned out it has been hanging for all these few days — all the cron processes are still there in the system and they couldn’t make progress at all. This is because the script needs to access some production machines but the production setup (dns service etc) has been changed in the past few days.

Solution is instead of saving files to a certain machine in production, we are uploading to s3. That way it will have nothing to do with future production upgrades.

Quite a cool experience when figured out, starting with so much confusion 🙂

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

September 2013
M T W T F S S
« Aug   Oct »
 1
2345678
9101112131415
16171819202122
23242526272829
30  

Twitter

  • couldn't get over the jetlag, sleeping during the day from seven to four, for days. 1 month ago
  • is emptying trash and happily discovering the available disk space now ranks at 100G+. 2 months ago
  • is looking at other people's intentions, not their capabilities, and feeling much happier every day :) 1 year ago
  • is planning on how to spend the next two weeks until the new year, at home. 1 year ago
  • is going to have human company for Thanksgiving; a rare event for the past like twenty years. 1 year ago

Flickr Photos

IMG_3517

IMG_3515

IMG_3505

IMG_3497

IMG_3261

IMG_3260

IMG_3255

IMG_2736

IMG_2733

IMG_2629

More Photos
%d bloggers like this: