So today I had the pleasure of a recycling company picking up our old Dell inventory (pe X550 through pe X850). It felt great throwing them on a giant stack. Nothing better then going office space on a bunch of crappy pos servers.
Life would be boring without change and lately I’ve gone through a lot of it. Other than my lack of updating of this blog, things have changed.
Some of the things that have changed, in no specific order:
- In May, after almost 4 years, Cam and I broke up. While she may have written extensively about this in her blog, I feel that there is nothing left to say on it. She cheated on me and that is just simply a deal breaker. Attempts to justify this is simply wrong as there is no excuse for that type of behavior.
- The colors of my condo have finally changed. The living and dining room was this nasty mustardy dark dark yellow, bordering light brown, now its bright yellow with a tint of orange. The bedroom went from a dark brown (with green) to a ice blue. Yum. Pics will be made soon.
- As of today a new couch. This thing kicks ass; fantastic to lay on.
My focus at work has also changed. Previously I was doing just system administration stuff but over the course of the last few months I’ve taken on more responsibility in the area of storage. So now I work with HDS, Netapp and a few Sun X4500s. Each have their own specific uses and it truly is exciting stuff, especially since who used to administer it left room for some major improvements to be made.
I have realized that I’d like to travel more. I am definitely not looking for a new job as I’m actually having a good time at Classmates with the one exception that if a job landed on my doorstep that would require me to travel a good amount (like 50%+) then I’d go for it.
Welp thats all for now.
So 18 months ago I deployed Zabbix at McGraw-Hill. Now I am doing exactly the same, only with a bit more hardware at Classmates.com
The setup.
Run Zabbix 1.5.3 in all 3 environments. Don’t customize so we can upgrade to 1.6 when that’s released in Q4 08.
Web / zabbix_serverd process server
1 for each environment (corp, prod, va). Machines are HL DL385 with an exception in VA where is a DL365. They are Opterons with RAID 1+0 and about 8gb of ram with dual nics connected to 1gbit ports. The work load isn’t here, its just the messenger. Runs RHEL 4.6.
Database server
3x Sun Fire T2000 servers, one for each environment. Running Solaris 10 05/08 and an source compiled optimized coolthreads build of MySQL 5.0.51b. Specs are 16 core Ultrasparc T1 procs with 8GB of ram again. Chuck Goolsbee would love these servers due to the amount of lower power they use but yet are so very functional These make perfect MySQL database servers. With about 200-300 hosts doing about 20-30 checks per host in each environment, they’ll need it too.
All three are in a distributed monitoring environment, where two child/slaves report back to the parent/master node. The big kahuna node will be in my corp environment with the child/slaves reporting back; similar to how our BI dept collects data.
Rolled out zabbix agent installation via cfengine packaged in a RPM for our RHEL4 based installations and added auto discovery rules as such in Zabbix. With 10 minutes I had an entire environment reporting various items back to the server. Unfortunately the configuration of server wasn’t this quick :-).
Things left over to do:
- The T2000s come with two filler trays for two more SAS drives. Plan to get 2x 146gb 10K SAS drives and run a ZFS pool on them for the MySQL db.
- Perform nightly db dumps to each environment’s Sun Fire X4500, Sun’s not so eco friendly 48TB disk drive monster.
Observations
-With corp completely done. The load on the mysql database server is about 7.x-8.x, thus there is still a full 50% left for expansion.
-Database is growing at a rate of 100MB a day, in part because historical data takes up a ton of space. This should slow down to a crawl on day 90 when it takes trending data (about 10% the size of history data) until day 180. The total DB size I am guess will be around 10-12GB.
So with a certain fruit company announcing Exchange support for their one button phone 5-6 weeks ago, I signed up my department and we got approved even though I sponsored myself on everything (corporate didn’t and they did all the proper sponserships, heheheh. I guess the fruit company still likes the rule breakers :-)).
I can’t talk about it specifically because I can’t talk about it so it. However so far so good but it definitely is *beta* software. Some quirks like it wiping out your existing contacts when you enable the exchange calendar and contact sync (the calendar sync works great… the contact sync it self not so much). However the ability to get your email and acknowledge pages without opening up your laptop is fantastic. Once I disabled contact sync and forced sync my contacts back over I was good to go.
Haven’t tried the Cisco VPN yet, have to find my profile file to load that one.
All we need now is landscape SMS and MMS support :-).
It wasn’t too long ago (2 years) where somehow it was required to be in front of a machine if it were to go down (as in it was no longer accessible via the OS) or needed an OS installed. Thankfully the most of the servers I work with now have remote capability like HP’s ILO which has a ‘virtual serial port’ interface or sun’s console access (ok prompt / ilom). It is how my colleagues and I can administer servers 3000 miles away or when we don’t feel like hanging out in the office we can do it from home. Like me tonight finishing up a kickstart (which also simply rules) and configuration.
Now if only we could remotely replace physical hardware without paying $150 per hour for a tech to do it.
It occurred to me that I should write more on this blog. Alas a lot of times I don’t have something interesting to say or I have to rant about something. Then there are those moments where I truly do want to post something up but another event pops up. I guess this will be a post where those that check in once a year or blue moon will still see I’m alive and doing well.
So what’s going on with me lately? I’m still working at Classmates. Its a good job. Its different working in a larger team then I’ve had in the past. I can definitely see some good things that this already has taught me. The biggest being that everything needs to be discussed and reviewed before implemented or changed. You also have to appreciate criticism instead of hating it. The job has also taken me to Virginia twice already since I started in August. One ended up being a three week trip, the other only 3 days. I can definitely see myself continue to grow there.
Cam and I are also doing well. Christmas came and then passed leaving us with only a small debt as opposed to larger ones in the past. I gave her a nice digital SLR like camera and she got me a TiVo HD. I was finally able to retire the Comcast cable box and replace it for a $1.79 per month cable card. The relationship Cam and I have is certainly a lot better then it was in the past. I’m glad we overcame our difficulties and problems we had earlier this year.
I hope that this year I can start taking part time classes at BCC or some sort of school. While I am gaining a lot of hands on experience in working with systems and year over year I see a dramatic improvement in my skills. However I really wish to improve my communications skills and managerial skills plus I’d be nice if I could get a degree out of it eventually so I have a backbone to my experience.
Hopefully I’ll be successful in my goal and I’m sure I’ll write about it here eventually. For those that do their yearly checks to see what I’m up to… hope all is going well in your life.
Yvo
So I was on Sun’s website reading up on ZFS (see post below) and it seems they are now demonstrating a data center in a box. That is actually pretty intriguing. They’ll be in Seattle on April 1st and 2nd.
This is what their marketing department has to say:
“Code-named Project Blackbox, this Sun innovation is the first virtualized datacenter, optimized for extreme energy, space, and performance efficiency. Project Blackbox enables instant-on expansion and rapid deployment while maximizing savings and providing operational flexibility to address two of today’s most critical IT issues — soaring energy prices and space constraints.”
Off topic and random… but i’ll end up going to one of their sessions.
More info:
Project Blackbox
On the Debian servers I administer recently an update to the apt & gpg package has caused me to get this message when ever I run apt-get update:
There are no public key available for the following key IDs: A70DAF536070D3A1
To fix this you must either be
a. root
or
b. have proper sudo privileges.
I will use sudo in my example.
sudo gpg --recv-key --keyserver wwwkeys.eu.pgp.net A70DAF536070D3A1
sudo gpg --export A70DAF536070D3A1 | apt-key add -
So far it only seems to be affecting my debian installs that use testing as the favored repository but I’d imagine that once etch goes stable that my stable favored servers will complain (or the debian team will have fixed it by then).
So about a month ago I wrote about Zabbix. Well it has been a month and after getting the corporate firewall team in Hightstown, NJ to open up specific ports between the two data centers we got Zabbix rolled out to our production environments. So far so good, if I even stop the Apache process for 30 seconds I get paged on my phone. We have severity levels setup so for example if the partition is 80% full it’ll email us as opposed to send SMS messages to our phone. However if a ping fails 3 times in a row then it goes nuts and hits everyone with a SMS.
The “Gooeyness” is useful to show management our uptimes and loads, plus it just looks purty. I love the CLI but I don’t mind a pretty GUI.
Screenies, click to blow up.
Note: This graph helped us actually discover a configuration problem as we were hitting our max connection limit (1500 per machine) frequently.
Edited: more screenshots.
Over the last few weeks I’ve been looking for a decent monitoring system that would monitor the 40 odd servers at work. Now anyone with even a small foot hold in the open source world has heard of nagios. Nagios is very open ended system that is very flexible but at the same time can be a behemoth to configure. Not to mention that it primarily depends on the SNMP protocol (yes it supports others as well) and there are a ton of plugins to chose from but hardly any clear documentation as everyone has *their* way of running Nagios.
So enter Zabbix. Another open source product that is also backed by the same company with a support contract. Easily to get support via the forums located on that website (Nagios does *not* have official support forums, but links to unofficial support forums) and the main product manager, Alexei, is very easy to contact. Not to mention that the product is stable and after you get the interface down (which is a blessing / hateful considering how entirely segregated the “configuration” section is from the “monitoring” section) its very easy to configure and setup. On the client side I can either do A. SNMP or B. an agentd that runs on the client machine. I have chosen B for all our Linux machines (Fedora and Debian). Easy to setup and it only requires two ports to open on the firewall (makes it easy for the corporate red tape).
It does everything I need: monitor logs, check processes (like httpd/apache, (x)inetd, mysqld, oracle, jboss), monitor critical files for any alterations (via checksum), monitor network bandwidth (out and in), process load (1m, 5m, 15m), memory usage (swap & physical) and uptime. The agent hardly produces any additional load on the machine nor is it a memory hog.
Currently its monitoring our internal machines (3 development, 1 proxy and itself) and it does a mighty fine job. Oh and load on the zabbix server machine. Well monitoring 5 machines is producing a 0.05 - 0.10 load. This is after upgrading the debian box from the 2.4.27 kernel to 2.6.17… when it was running the 2.4.27 kernel its load was 0.8 - 1.0. The thing that takes up the most load is mysqld (mysql 5.0.24) which isn’t too surprising since it does approximately 30,000 queries every hour for the 5 machines currently being monitored.
You are currently browsing the archives for the work related category.