Friday, 16th o January 2004
The Perfect Storm
Laura was out in a meeting today, so I was alone with the kids, the two little delightful munchkins. They were getting antsy to get out, so around four thirty, we decided to go to the park. I put on their shoes and got them their clothes. Jaimito was particularly impatient. "Go go go gogo....waaaaaaaaa."
"Jaimito, Daddy has to go pee pee. We're still going to the park. I'll be right back."
Suddenly from out of nowhere, a breeze picked up. I felt the back of my neck tingle. It was a sinking feeling. I was miles from home and I knew there was no one to save me.
I answered my frantically ringing cell phone. "James O'Malley."
«Hey James, Jose Camacho»
«How are you, Jose, how can I help you.»
«There seems to be a problem with our bandwidth. I'm not getting access to the Internet, and my provider said that they are not showing any connectivity problems with our site.»
«Hmmm,» I said as I did the pee pee dance. Damn! I lamented, tech support.
By then, the wind was blowing like a 2 year old lying on the floor throwing a tantrum. I knew it was bad. Really bad. None of the hatches was battened... or whatever you do with hatches. «Let's take a look,» I said as I logged in via secure shell. «You look good from here. I don't see a problem.» I was puzzled. I asked him about his machines, checked IP's, checked the web caching proxy. Nothing. I sent him running around checking cables, different machines, different web-sites. Nothing. Jaimito was still wailing.
Oh my God, the Cape wasn't where I thought it was. I realized I was in the wrong machine. «Jose, you won't believe this. I was logged into another client's machine. Forget everything I said.» I felt like an idiot, but we quickly got back to work. No time to fret. This was life or death. I got him to go to the machine and log in. I wanted him to run an IP traffic program so we could analyse from where the flood of traffic originated. «Log in, type i-p-t-r-a-f and hit enter.» He typed it wrong. I spelled it again and I got him into the program.
Turn into the storm, I kept repeating to myself, but it's tough. When you are being buffeted from all sides, you can't tell from where the winds are blowing. Suddenly a big one came crashing over the port side. My cell phone started to beep. Arrgh, low battery. I quickly found a charger cord and plugged myself in. I was now tied down on deck. I wasn't going anywhere. If the boat sank, I was going to go with it. I still had to use the bathroom and I could no longer see what the kids were doing, and the mosquitos were ravaging my legs.
I continued to lead him through menus and commands, having him read to me the program's output letter by letter. Every mistake costs valuable time. What could be going on here, I thought? The machine is spewing out tons of traffic, but I can't log in. Normally, even when traffic is heavy, the encrypted shell will at least give me a login prompt. It looks like a ICMP flood from a virus worm, but I have that blocked at the firewall. What is going on!
«Check the protocols on the outgoing connections,» I said. «I bet it's a large email attachment. And I bet it's going over the IP-sec encrypted tunnel.» Sure enough that's what it was, encrypted communications between nodes on their VPN. IP-sec packets take precedence over normal Internet traffic, so if you flood the tunnel, traffic outside receives a lower priority. «I bet someone sent a huge email attachment and it's stuffing up the tunnel.»
Jaimito is quiet now. I don't hear him, but as any sailor knows, that is the time to be afraid, be very afraid. «Hold on,» I told Jose. I put the phone down and made my way to our bedroom. The floor was sticky. Odd. The devastation I saw would bring a grown man to tears: lipstick, makeup orange juice everywhere. The storm wrought more damage than I could have imagined. "Jaimito!!" He looked up at me with guilty eyes and handed me the smashed lipstick as if to say, I was just looking to give this to you, Daddy. "Oh, you little boy!" I said in a stern voice, which as any parent knows is more than sufficient to initiate a full blown guilt cry. Jaimito started bawling again. Such a good little repentant boy. I love him. But now was not the time, the storm was coming again, and I needed to get back to my station.
I picked up the phone. «Jose, are you there? Good, now we see that it's a large email attachment. Let's try to delete it from the mail queue.» Type the command /root/qmHandel -l | less, then look for the unique id of the email in the queue. Then type /root/qmHandel -dxxxx. Rinse, lather and repeat until the queue is empty. I quickly realized that the task was too complicated to attempt in such a tumult. «Let's just kill the whole email server, » I said. «Type k-i-l-l-a-l-l -9 q-m-a-i-l---s-e-n-d... no, not sent... s-e-n-d. Okay, got it. Did it return an error. No? Okay, hit the up arrow and repeat the command until you see it say "no process found." » This carried on for a while. Mistakes, corrections, return to the command. I would leave the phone make a quick circle of the house to see that hatches were battened and attempt the log in remotely. «I'm in!» I exclaimed. «Okay, that's what it was, Jose. Let me go into the queue and delete the e-mails. I'll have it fixed in a few minutes.» I hung up the phone and breathed a sigh of relief. I ran to the bathroom before I exploded and Jaimito started wailing again as if to say, I waited all that time and you're still not going to take me to the park?! Poor little thing. Of course, Olaia was a little angel, she stayed calm throughout the perfect storm.