Homo-Adminus Blog

If you’ve ever worked in companies with 5-10+ servers and it was your responsibility to install new boxes, change some configuration files and install new software on many boxes you definitely know how painful this work is. Every time you need to change something on 3-5-100 boxes, you go there and make those changes. Most experienced of us used some weird scripts to perform some task on many boxes or used some stuff like dsh. Even with those tricks I’d never wish this work to anyone.

While I was working in Galt, I’ve asked our junior admin to check out puppet and try to use it on our servers. After a week of screaming he’s managed to install and configure it and said “Wow! This is what we definitely need!”. Since then he was using it and was happy :-). But, unfortunately (for me) I’ve never tried it and when I’ve changed my job and started doing the same old (but not good) things like dsh and for+ssh I’ve started thinking: “Stop! Try puppet - maybe it is what you need!”. And what can i say? Yes! This is what every admin needs! Today we’ve deployed our next web server (we do it pretty often) and it was done with puppet - pretty painlessly and quickly.

So, what is puppet? Let me quote their site:

Puppet lets you centrally manage every important aspect of your system using a cross-platform specification language that manages all the separate elements normally aggregated in different files, like users, cron jobs, and hosts, along with obviously discrete elements like packages, services, and files.

Puppet’s simple declarative specification language provides powerful classing abilities for drawing out the similarities between hosts while allowing them to be as specific as necessary, and it handles dependency and prerequisite relationships between objects clearly and explicitly. Puppet is written entirely in Ruby.

Basically, puppet is a tool which lets you describe whatever you’d like about your server(s) and deploy your configuration (configs, packages, files, directories, cron jobs, user accounts, ssh keys, etc) to any number of servers you need. You can describe some simple parts of your system (like “apache”, “nginx”, “rails”, “useful_gems”, “admin-ssh-keys”, etc) and then describe your servers using those terms (like machine X = apache+rails+useful_gems+admin_ssh_keys). When it is done, you’ll be able to deploy all this stuff with one command!

So, if you’re struggling with deploying some changes on many servers or, even, have only a few of them, I’d recommend to try puppet - it could save you a lots of time and nerves.

P. S. When you’ll try to use it - do not give up for a few days - it is hard to change your way of thinking and start writing correct and convenient scripts using puppet’s language.

This will be one of those posts I’d like to publish primarily to be able to coma back later and check it out instead of reading docs again

So, we have a server with two (or more) network interfaces are we need to be able to use more than one interface in our VDS machines. How do we set it up?

(more…)

It’s been a long time since we’ve started this project and it is time to make a checkpoint. So, I’ve decided to release final 1.0 version and make 1.X branch stable while all serious development with deep architectural changes will be done 2.X branch (trunk at this moment).

Changes from previous release:

• Perl semaphores implementation caused huge memory leaks (mmmd_mod).
• Now we do not send any commands to hard offline hosts with dead TCP/IP stack to prevent mointoring problems for other hosts.
• Removed legacy StartSlave method from agent code which caused problems on some Perl versions
• Added a few fixes to prevent non-exclusive roles from moving. This caused internal status structures to be corrupted.
• Made all mysql checks properly report errors occurring (previously they were resulting in an OK response). Thanks to Phillip Pearson.
• Some memory leaks found in mysql checkers and as a quick fix we’ve added an idea of “Maximum Checks Before Restart” to all checkers. If you want some checker to restart after N checks, simply add “restart after N” to your checker declaration.
• Added some more docs to the project site.

New version can be obtained here or from the project’s SVN repository.

New alpha release 1.0-pre4 of the MySQL Master-Master Replication Manager. This release has lots of major fixes and I’m glad to announce first sponsored port of mmm to non-linux platform - it has been ported to Solaris 10. So, here are our changes in this version:

• Real checks timeouts - I’ve found and fixed lots of problems in checks timeout code and now if you specified in your mmm_mon.conf, that some check should timeout in 5 sec, it would timeout correctly on all supported platforms.
• External third-party tools using - On all supported non-linux platforms mmm will use system binaries for fping and arp_ping so porting to another platforms would be much easier.
• Agent notifications fix - Now we don’t try to notify dead agents about cluster changes and additionally we have 10 sec timeout on notification sends to prevent monitoring from lagging on network connection timeouts.
• Bundled fping and send_arp - We have both used third-party tools used in mmm bundled in our distribution as separate build trees (you could find and build/install them from contrib directory).
• Flexible perl binary location - We’re using “#!/usr/bin/env perl” as shebang line in our perl scripts so you can use any perl interpreter just by placing it earlier in your PATH variable.

Notice: If you’ll try to install this version, try to run bin/sys/fping and bin/sys/send_arp on your server before installation. If you’ll notice any errors, feel free to build binaries for your platform using contrib/* sources (you’ll need gcc and libnet installed).

So, as you can see, mmm development goes forward and we’re fixing some problems trying to make this software mature. If you want to help us, you can send your comments to mmm-devel mailing list, post your bug reports to our bug tracking or sponsor any changes you need

So, first two days of mysqlconf’07 are finished now. What can I say is that without any doubts: It worth it! If you’re working on some high traffic projects, some high-loaded database driven systems, etc, you definitely should attend such conferences - you’d never be able to get such big amounts of information from the best people in the industry as you can get here.

I’ve been attending mostly practical MySQL scale-out sessions and BOFs and I’ve got really controversial impressions. No - everybody was great, controversial thing is my own level - I never before was so sure that I know nothing at all! I see people here for which most of new things for me (especially in MySQL scaling) are pretty obvious and it is hard to keep myself from some kind of self-beating because I don’t know these things.

Interesting thing happened today - I clearly realized why YouTube guys got their billion and we or someone else didn’t - IMHO - one of the major factors was their ability to handle all these users traffic efficiently. They’ve got one of the best (imho) technical people in their command. I was really impressed by one of their guys (sorry, but I’m too bad in remembering people’s names) - one of the two founders - who participated in MySQL replication and scale out BOF as attendee. He clearly showed me that they’re able to grow almost without any limits because of their skills which is one of the major requirements in today’s fast changung world.

Another interesting thing was that I realized that my spoken English could be really good when people speak clearly and I understand their questions. I spoke with some guys about our mmm project, about maby things and there was no problems at all!

So, its going pretty late and tomorrow going to be a long day so I want to repeat - if you have any chance to get on such conferences, DO IT! If your company can’t help you, spend your own money or find some other solutions, but DO IT! It is the best thing could happen to YOU as professional!

P.S. Just remembered - I saw some women-DBAs today! Really smart girls! I never thought that pretty girl can become such great IT prefessional and now I know - I was really wrong.

P.P.S. Sorry to russian-speaking readers - I will definitely translate my last posts later - just when I’ll come back home from this crazy (but the best) event with 14hrs of load a day

This blog turned 1 year old last month and I think all of my readers were glad to read it. I’ve never asked for help and I offered help to people who needed it. But today I’m forced to ask my readers for help because it is the first time in my long practice when I really don’t know how to solve my problem.

(more…)

So, I’s been a long time since I wrote my last post here. Lots of work and almost no new interesting technologies in my work caused such delay. But today I’m proud to announce release of really interesting project: MySQL Master-Master replication manager - set of flexible scripts for management different MySQL deployment schemes with master-master replication involved.

More information about this software could be found in detailed announce in Peter Zaitsev blog (actually this software was created by me for his company’s client) or at project page. All your questions and suggestions welcomed in MMM development group on Google Groups. If you’d like to support this software on Digg.com, you’re welcome.

This small article is about Rails+Mongrel setup and its performance. List of other tested deployment schemes, description of testing methodology and, of course, all benchmark results you can find on “Ruby On Rails Benchmark Summary and Findings” page.

(more…)

While I’ve been doing performance testing of different Ruby on Rails deployment schemes, I came across very interesting software - HAProxy.

HAProxy is a TCP/HTTP reverse proxy which is particularly suited for high
availability environments. Indeed, it can :
- route HTTP requests depending on statically assigned cookies ;
- spread the load among several servers while assuring server persistence
through the use of HTTP cookies ;
- switch to backup servers in the event a main one fails ;
- accept connections to special ports dedicated to service monitoring ;
- stop accepting connections without breaking existing ones ;
- add/modify/delete HTTP headers both ways ;
- block requests matching a particular pattern ;

It needs very little resource. Its event-driven architecture allows it to easily
handle thousands of simultaneous connections on hundreds of instances without
risking the system’s stability.

As for me, I’m really impressed by HAProxy’s performance and I will suggest to try to use it in some high-availability environments because of its performance and set of features. Take a look at this software - it is really good tool!

Because of not fully correct testing methodology, benchmark results are not fully correct. So, I decided to redo all tests. New benchmark results you can get in “Looking For Optimal Solution” series Summary post.

This week we have started one new project with Ruby on Rails as primary framework. My first task was to prepare runtime environment for it on one of our development servers. When I have tried to research how people doing it, I noted that there is no information about how to deploy rails application with nginx as frontend and what is performance of such solution. Before blindly make any decisions about future platform I’ve decided to make some performance tests of the some most popular rails web servers/frontends. Results of these tests you can find here with configuration samples for all software that I have used.

(more…)