16 Feb 09 All the right things in a week of hell.
Two weeks ago today is when reality caught up with my family and myself. Two weeks ago my dad had a heart attack on board flight DL162 over the middle of Montana. The crew on board that plane is what saved his life. The flight was JFK bound but was instead diverted to Great Falls, MT.
Flight DL162 diverted to Great Falls, MT.
What came next were hours of getting the entire family out to Great Falls. After dropping everything at work I used my iPhone to find out he was in Great Falls, then did a Google search to look up hospitals in that area. In my favor, they only had one major hospital, which I called right when I got out of the car. By 3pm (plane landed in GTF at 2:02pm PST) I knew where he was and that they were taking care of him.
Made all the phone calls / flight arrangements while getting updates from the hospital which wasn’t forecasting a good recovery. By 6pm I had my brother driving from Pullman, WA in a rental (his car wouldn’t make it) from Hertz and 3 plane tickets ordered from Alaska Airlines with two hotel rooms at a local Marriott hotel.
By 11pm Mountain time (10pm Pacific) we were in Montana next to my Dad’s bed seeing him in probably the worst state I had ever seen.
Thankfully over the next few days he got better (faster then the hospital predicted) and we were once again reminded how lucky he was that he was at hospital (Benefis Healthcare) that is able to perform primary angioplasty.
On Thursday the media in Montana got wind of what was going on in their state and my dad was over the news. We also got to meet the entire staff that saved him.
The staff at Benefis.
By Friday he was allowed to go home and we took a Friday afternoon flight back to Seattle from Helena, MT.
I’m incredibly thankful for all the things that went right. Right flight crew, right doctor on board and an incredibly good hospital (all in the middle of Montana).
This experience also taught me how precious family is and that the time on this earth can be cut off at any moment but I’m glad to still have my Dad for what I hope is a lot more years.
Tags: family, iphone, life, technology, work
04 Dec 08 It’s 5:00pm and your flight leaves at 7:30…
and you’re literally in Rockefeller center on the night Christmas tree lighting ceremony.
Imagine the following situation:
- Gigantic crowd, closed subway station at 47-50th because of the chaos at Rockefeller center and streets closed off
- Hotel is 9 street blocks north of you with your luggage (gigantic suitcase)
- The plane at JFK will leave with or without you.
That is the situation I found myself in yesterday. I literally had to push my way through a crowd of thousands of people to get to the hotel to pickup my crud. Finally got there at 5:25. Rushed upstairs got my stuff and thought about options on how to get to JFK.
* Taxi cab – $45, takes about 80 minutes.
* Subway Line A to Howard Beach station – $2 + $5 for Airtrain, takes about 70 minutes
* Subway Line E to Jamaica Station – $2 + $5 for Airtrain, takes about 60 minutes + time to get to Line E
* LIRR to Jamaic Station – ~$8 + $5 for Airtrain, takes about 35 minutes + time to get to Penn Station
Cab was out of the question. Battle came down to Line A (since it was right next to the hotel) or a trek to Penn Station. With the fact that having a giant suitcase on a subway at rush hour is *no* fun at all, I decided on the railroad. Made it to Penn Station at 5:40pm, train left at 5:45pm and I was pulling into Jamaica station at 6:15pm.
I thought to myself the entire way that I can’t even make it in 35 minutes by car or any form of public transportation to Seatac when stuck in Bellevue at 5pm… whereas in a city just barely under 10 million people and about the same distance to the airport I have options and the ability to do exactly that.
Soon, very soon, options wil become available in Seattle….
14 Sep 08 A new direction
Starting this Monday my already adventurous but short career is going to take me in a new direction. For years I’ve been building up my skills on the *NIX platform becoming increasingly better by the day to the point where I feel comfortable hacking up C code in order for it to compile on say… Solaris 8 (not by choice 
). I remember it was only 3 years ago that I was answering phone calls and performing tech support and since then have been doing system administration, with the last 6 months being more storage (Netapp & Hitachi USP) re-implementation & administration.
After my relationship ended in May I thought to myself “If a company offered me a job to travel 50% or more of my time, I’ll take.”. It would be the only reason I’d leave Classmates. Such an event happened in August. An usability study in June turned into a job offer by August after various delays on my side. So this Monday I start. First I need to get acclimated with the product (which is called Likewise) and hopefully I’ll be soon traveling all over. This is a big change career wise for me. Going from system administration to deploying software is a pretty big change but it’s a huge challenge for me. My fear is that I’ll get rusty when it comes to system administration but I’m going to do my best to keep my “A” game.
One thing I hate from all of this is way my resume is starting to look. Sure my last two gigs were over a year but not anywhere near 2 years and that makes me fairly unhappy with myself. That will be my next thing to improve…..
Tags: classmates, life, likewise, traveling, work
04 Aug 08 Buh bye Dell
So today I had the pleasure of a recycling company picking up our old Dell inventory (pe X550 through pe X850). It felt great throwing them on a giant stack. Nothing better then going office space on a bunch of crappy pos servers.
Tags: dell sucks, kent, work
03 Jul 08 Deploying Zabbix once again
So 18 months ago I deployed Zabbix at McGraw-Hill. Now I am doing exactly the same, only with a bit more hardware at Classmates.com
The setup.
Run Zabbix 1.5.3 in all 3 environments. Don’t customize so we can upgrade to 1.6 when that’s released in Q4 08.
Web / zabbix_serverd process server
1 for each environment (corp, prod, va). Machines are HL DL385 with an exception in VA where is a DL365. They are Opterons with RAID 1+0 and about 8gb of ram with dual nics connected to 1gbit ports. The work load isn’t here, its just the messenger. Runs RHEL 4.6.
Database server
3x Sun Fire T2000 servers, one for each environment. Running Solaris 10 05/08 and an source compiled optimized coolthreads build of MySQL 5.0.51b. Specs are 16 core Ultrasparc T1 procs with 8GB of ram again. Chuck Goolsbee would love these servers due to the amount of lower power they use but yet are so very functional These make perfect MySQL database servers. With about 200-300 hosts doing about 20-30 checks per host in each environment, they’ll need it too.
All three are in a distributed monitoring environment, where two child/slaves report back to the parent/master node. The big kahuna node will be in my corp environment with the child/slaves reporting back; similar to how our BI dept collects data.
Rolled out zabbix agent installation via cfengine packaged in a RPM for our RHEL4 based installations and added auto discovery rules as such in Zabbix. With 10 minutes I had an entire environment reporting various items back to the server. Unfortunately the configuration of server wasn’t this quick .
Things left over to do:
- The T2000s come with two filler trays for two more SAS drives. Plan to get 2x 146gb 10K SAS drives and run a ZFS pool on them for the MySQL db.
- Perform nightly db dumps to each environment’s Sun Fire X4500, Sun’s not so eco friendly 48TB disk drive monster.
Observations
-With corp completely done. The load on the mysql database server is about 7.x-8.x, thus there is still a full 50% left for expansion.
-Database is growing at a rate of 100MB a day, in part because historical data takes up a ton of space. This should slow down to a crawl on day 90 when it takes trending data (about 10% the size of history data) until day 180. The total DB size I am guess will be around 10-12GB.
07 Mar 08 simply put: ilo rules
It wasn’t too long ago (2 years) where somehow it was required to be in front of a machine if it were to go down (as in it was no longer accessible via the OS) or needed an OS installed. Thankfully the most of the servers I work with now have remote capability like HP’s ILO which has a ‘virtual serial port’ interface or sun’s console access (ok prompt / ilom). It is how my colleagues and I can administer servers 3000 miles away or when we don’t feel like hanging out in the office we can do it from home. Like me tonight finishing up a kickstart (which also simply rules) and configuration.
Now if only we could remotely replace physical hardware without paying $150 per hour for a tech to do it.
05 Dec 07 Back to Virginia
Going back to good ol’ Virginia for 3 days this time instead of 3 weeks. Hopefully HP will get their sh… I mean act together so all the parts I need are there when I get there. One thing is for sure, navigating the HP support tree is helluva lot harder then getting someone from Sun on the phone.
20 Nov 07 The Virginia transformation
The last 3 weeks I’ve spent my time in Virginia, specifically in one of the Equinix datacenters in Ashburn. I was there to help build out a datacenter. When we arrived it was a box galore everywhere (165 HP DL365s, 50 Cisco 3750s, 30 Cisco 2960s, 2 Cisco 6500s) and the accompanying accessories for everything (supervisor cards, memory, the works). The racks and pdus had been setup, nothing else had. So over the course of 3 weeks we basically unboxed, racked, cabled and did basic configuration on everything. Hindsight is a bitch and the fact that pictures were frowned upon made it difficult to make a photo journey. However I do have 6 photos that represent what has been completed in the end.
All in all a very good trip. Got to see some of Washington DC too. WSDOT could truly learn something from VADOT concerning the implementation of roads, highways and what not.
