The setup.
Run Zabbix 1.5.3 in all 3 environments. Don’t customize so we can upgrade to 1.6 when that’s released in Q4 08.

Web / zabbix_serverd process server
1 for each environment (corp, prod, va). Machines are HL DL385 with an exception in VA where is a DL365. They are Opterons with RAID 1+0 and about 8gb of ram with dual nics connected to 1gbit ports. The work load isn’t here, its just the messenger. Runs RHEL 4.6.

Database server
3x Sun Fire T2000 servers, one for each environment. Running Solaris 10 05/08 and an source compiled optimized coolthreads build of MySQL 5.0.51b. Specs are 16 core Ultrasparc T1 procs with 8GB of ram again. Chuck Goolsbee would love these servers due to the amount of lower power they use but yet are so very functional These make perfect MySQL database servers. With about 200-300 hosts doing about 20-30 checks per host in each environment, they’ll need it too.

All three are in a distributed monitoring environment, where two child/slaves report back to the parent/master node. The big kahuna node will be in my corp environment with the child/slaves reporting back; similar to how our BI dept collects data.

Rolled out zabbix agent installation via cfengine packaged in a RPM for our RHEL4 based installations and added auto discovery rules as such in Zabbix. With 10 minutes I had an entire environment reporting various items back to the server. Unfortunately the configuration of server wasn’t this quick .

Things left over to do:
- The T2000s come with two filler trays for two more SAS drives. Plan to get 2x 146gb 10K SAS drives and run a ZFS pool on them for the MySQL db.
- Perform nightly db dumps to each environment’s Sun Fire X4500, Sun’s not so eco friendly 48TB disk drive monster.

Observations
-With corp completely done. The load on the mysql database server is about 7.x-8.x, thus there is still a full 50% left for expansion.
-Database is growing at a rate of 100MB a day, in part because historical data takes up a ton of space. This should slow down to a crawl on day 90 when it takes trending data (about 10% the size of history data) until day 180. The total DB size I am guess will be around 10-12GB.