I could easily mark this as the worst morning in as far back as I can remember. Without the first cup of coffee I sat down to scan our servers like I do everyday, just looking for anything out of the ordinary, like services that failed to run. For the most part it is a ten minute job that rarely varies day to day. This morning was an exception.

Nearly every nightly job failed. Worse than that there was an hour and ten minute hole in the logs, 0155 to 0305 was completely unaccounted. I scanned every log from authentication to our application logs and every single one of them showed this hole but checking our external monitoring service showed that we had zero downtime. What the hell happened?

A cold hand of desperation and fear gripped my stomach leaving me dizzy. I ran chkrootkit but came up clean so I mentally prepared myself to rebuild the server and possibly be eviscerated by my bosses. How would I explain this? How could I protect us from it happening again, that is if I still have my job?

Sitting helpless I realized, “Spring Ahead”.

Tags: Hack, Linux, Security, Spring, Sysadmin

This entry was posted on Sunday, March 9th, 2008 at 7:44 am and is filed under Sysadmin. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.