This is why when I received an email from Packt company asking if I’d like to read and review their book on Lighttpd I decided to give it a shot (I usually do not review any books because I do not always have enough time to read a book thoroughly to be able to write a review). So, here are my impressions from this book.

First, when I received the book, I was in doubt: how such a small book could cover so flexible and multi-purpose piece of software like Lighttpd? But after reading a few first chapters I understood that the book is written as any professional book should be written - no weird long chapters explaining what is Internet, web server, HTTP or HTML. From the first pages you dive to the world of Lighttpd starting from installation and going through the most popular and useful configuration options to really interesting virtual hosting configurations.

Pretty interesting security-related chapters explain typical process of obtaining and installing SSL certificates. This always was the most painful process for me in almost any web setup I worked on, don’t know why though, this process seemed kind of messy for me, but I think those chapters could help novice web server administrators understand the process of securing a web server.

For many new lighttpd users the most interesting chapters should be ones that explain typical problems and use cases of using apache with or instead of apache servers and describe how to set up lighttpd to be used with the most popular backend web software (Ruby on Rails, wordpress, phpmyadmin and more). From my consulting experience I should say that this is the most popular lighttpd deployment scheme (lighttpd frontend + some backends) and this is why these chapters should be the most interesting for new users. As a RoR server administrator I wish authors would explain more different deployment options for Rails, but of course I understand that the book is dedicated to lighttpd, not Rails so it is OK to miss some options.

And finally, for experienced users like me who understands all the internals of the web traffic handling and needs to get more control over the traffic there is just awesome chapters on using Lua with lighttpd and writing custom lighttpd modules. I really wish I had such an explanations when I was doing nginx modules development for our projects! If I’d write some lighttpd module I’d definitely buy this book just because of those Lua and modules-related chapters - there is not so much info on these topics on the Net and such a solid explanation of the basics of web server extension could help a lot.

So, as the final words on this book, I’d like to recommend it to all web services administrators (even if they do not use lighttpd yet) - this book explains many fundamental things that you’d really like to know. As for professional developers/admins, this books could be useful if they are going to work on some advanced configurations or modules for lighttpd (yes, I loved those lua and modules sections).

All this time (few months now) I was using starling noticed really bad thing in how it works: if workers die (really die, or lock on something for a long time, or just start lagging) and queue start growing, the thing could kill your server and you won’t be able to do something about it - it just eats all your memory and this is it. Since then I’ve started looking for a better solution for our queuing, the technology was too cool to give up. I’ve tried 5 or 6 different popular solutions and all of them sucked… They ALL had the same problem - if your queue grows, this is your problem and not queue broker’s :-/ The last solution I’ve tested was ActiveMQ and either I wasn’t able to push it to its limits or it is really so cool, but looks like it does not have this memory problem. So, we’ve started using it recently.

In this small post I’d like to describe a few things that took me pretty long to figure out in ruby Stomp client: how to make queues persistent (really!) and how to process elements one by one with clients’ acknowledgments.

First, we need to connect to the queue server and this code is the same for clients and servers:

Second form looks better for me because it allows you to specify if you want client to be reliable (lock on errors and try to reconnect, etc) or just want client to raise an error in case of any problems.

When you’ve connected to the server, you can push your data to a queue:

And this is where you have an ability to specify if you want your data to be persistent (survive server crashes, etc): headers is a hash that could have an element :persistent => true, which would do the thing. So, your code would looks like this (for example):

Now, when you have your data submitted to the queue, you need to be able to read it and process with some script. This is as simple as the following code:

Again, we wanted to receive messages one by one and acknowledge successful processing in our code. This is simple too. You need:

• Pass :ack => :client as an element in your headers hash
• Call client.acknowledge(msg) in the loop if you’re sure that an element could be removed from the queue

This is basically it with the stuff I wanted to explain today. If you’ve never tried to work with any queue servers, try today and maybe tomorrow you won’t be able to imagine your systems architecture without such a component

Few times I was creating a git branches and hacking this action trying to implement some sort of page-level caching to make things faster. But all the time results weren’t as good as I’d like them to be. So, branches were sitting there and waiting for a better idea.

Few months ago my good friend has joined Scribd and we’ve started thinking on this problem together. As the result of our brainstorming we’ve managed to figure out what were the problems preventing us from doing efficient caching:

• First of all, a lots of code in the action is changing the page view if our visitor is a bot (no, not a cloaking, just some minor adjustments of the view).
• Second problem was a set of differences in the view for anonymous and logged in users.
• And finally, third problem was the fact that the page has a few blocks that change pretty dynamically: document stats pane and comments lists.

All these problems when combined were creating a lots of pain when I was trying to cache a whole page. When we’ve figured them out, we’ve started thinking on how could we generalize possible combinations of those factors and possible approaches to caching.

There is a well known idea in web applications development: the fastest web app action is an action that does not require any code to be executed on your application server. So, first idea we’ve tried to think about was some approach that would definitely reduce the number of hits on our app servers. This idea was based on HTTP protocol features related to Last-Modified and E-Tag headers. But there was a problem - not so many users go to the same page twice so even if we’d make the page cacheable, it wouldn’t help too much. But the idea of full page caching outside of the application was really good and we’ve started playing with it to figure out how to use it in production.

Long time ago, when Internet was slow and expensive many ISPs and large companies were trying to reduce their traffic w/o hurting users’ experience. Then caching proxy servers were born. The idea of those servers was to handle all web requests going from a network (ISP or a company office) and try to cache as much content as possible so when the same or some other user would request a cached page, proxy server would return it really fast. If we’d implement support for those Last-Modified headers, all proxy servers would be happy to cache our pages. But there was a problem - no one uses caching proxies in 2008 So, we’ve got an idea - why can’t we place such a server in front of our application and make it cache content for all users in the world? (Yes, we knew about a caching reverse proxies before - I’m just trying to explain the flow of our thoughts and words when we were brainstorming the problem).

The only problem with this approach would be to differentiate logged in users, anonymous users and bots. Considering the fact that our proxy server could be placed between the app and our web servers (nginx), we’ve decided to create a nginx module that would translate the same document page URLs to a set of URLs, which would be different for all those 3 kinds of users.

When all those problems with different kinds of users were solved, we’ve decided to solve the last one - non-cacheable dynamic stats pane. The solution was pretty simple - we’ve added a small ajax call to the page which would update stats on the cached version of our page for all real users while bots will see the same page, but with a bit stale stats pane.

Long story short, the results is really great. Application servers load reduced by 50-70%, database servers load is reduced by 30-60%, response times dropped down to 150-200 msec from 500-750 msec. As an additional positive effect of the caching we’ve managed to remove all fragments caches from the application and free more of memcached resources for data caches. Here are a few cacti graphs of our servers load/traffic (the caching was introduced on Oct 9th at night):

Main MySQL command counters:

Uploaded with plasq’s Skitch!

One of our Application Servers CPU Usage:

Uploaded with plasq’s Skitch!

One of our Application Servers Load Average:

Uploaded with plasq’s Skitch!

Unfortunately there are a lot of things to share related to this caching experience, so I’ve decided to make a series of posts that would explain all the problems we had and solutions we’ve found for each of the following parts of the caching system:

• Nginx module development
• Squid Server setup (configs and hardware)
• Rails code to support Last-Modified headers and how we purge caches

So, if you’re interested in details, subscribe to this blog’s RSS feed and in a few days you’ll see the first article from this series.

Bounces-handler package is a simple set of scripts to automatically process email bounces and ISP‘s feedback loops emails, maintain your mailing blacklists and a Rails plugin to use those blacklists in your RoR applications.

This piece of software has been developed as a part of more global work on mailing quality improvement in Scribd.com, but it was one of the most critical steps after setting up reverse DNS records, DKIM and SPF.

The package itself consists of two parts:

• Perl scripts to process incoming email:
  • bounces processor — could be assigned to process all your bounce emails
  • feedback loops messages processor — more specific for Scribd, but still - could be modified for your needs (will be released soon).
• Rails plugin to work with mailing blacklists

For more information, please check our README file. If you have any questions, comments or suggestions, please leave them here as a comments and I’ll try to reply as soon as possible.

Notice: The box can’t be restarted to check with clean caches and stuff, but I was doing full reads from this huge table on a machine with 16Gb RAM so all caches were washed out by this load anyways.

As you can see, less work linux does on its side to optimize disk I/O, slower it works Actually it was pretty expected, but still - surprising result. The problem (as guys from Youtube explained on the last year MySQL Conf) is there because Linux knows nothing about RAID’s internals and specific drives’ queues so when it is trying to re-arrange requests in its own queue it wastes CPU resources and could potentially prevent RAID controller from doing its own queue optimizations.

After this test I’ve tried it on a few other I/O-bound servers (both read and write bound) and the result was the same - noop (do nothing) I/O scheduler gave me the best results. Long story short, I’ve decided to try this scheduler on all our boxes for a week and look at the results in cacti graphs to see how it works.

We are looking for a data charmer. A mysql magician. A code hack. A funny man. A mad man. A passionate man. Or perhaps a woman who does all these things and more.

Here’s what you gotta do:

• Pro-active and reactive performance analysis, monitoring and general database plumbing of all leaky issues.
• Work with others on the team to help maintain/improve and support the infrastructure for a high traffic, high growth site
• Optimize and tune the database day to day
• Algorithmic bent. Develop algos to quicken search times, response times, find shortest paths between various connections on site.
• Have solid low level networking/protocol/computer security skills
• Log everything. Usage stats, search stats, user behaviour stats. Draw conclusions. Constantly refine and tinker.
• Help with periodic large storage migrations
• Work intimately with operations, development, and strategy team to ensure smooth deployments of new iterations, high availability of database services.
• Understand capacity planning. Always thinking 10 steps ahead. (Whether it means looking at distributed systems services, cloud computing options, evaluating HA models used in other industries etc)
• Have a pulse on the state of the web, social media, social networking, different scalability architectures, benefits/negatives of each.
• Interest in high concurrency, distributed systems architectures.
• General low level hacking/scripting/optimizations in perl/python.
• Evaluate changing conditions in the archi
• Think creatively. No dogmatists.

Ideal skillset:

• BS in Comp Sci or equivalent
• 5+ years experiene with Linux/Unix systems
• 3+ years with MySQL in production environment
• Knowledge and experience with partitioned architectures and a database sharding techniques
• Capacity planning/high growth planning/emergency planning experience
• Passion, bordering on paranoia, for hunting bottlenecks, and optimizing IO operations
• Experience with MySQL replication
• Deep experience with MySQL internals
• Experience with performance analysis tools, storage engines, backup methodologies for MySQL
• Great perl/shell scripting experience
• Team player, self motivated, able to handle high stress situations while maintaining a calm disposition
• Great communication skills, attention for detail, and an interest in the business side of the equation of systems/scale planning
• Eat/sleep/breathe the web, startups, and the landscape of the social web
• Insomniac

We’re ready to offer an aggressive salary with tremendous upside by way of stock options, commensurate with your experience, your drive and your results.

Apply directly to:

net ‘dot’ startup ‘at’ googles mail service dot com

by sending us a CV/resume, and optionally, a link to your blog or Linkedin profile.

“My family got bad news - doctors said allogenic bone marrow transplantation is the only chance for my son Ivan.

“8 months of heavy and expensive immune suppression brought some positive results so we hoped that recovering is just question of time.

“Ivan is very brave boy - not every human meets so much suffering during whole life, like Ivan already met in his 2,5 years. But long road is still in front of us to get full recover - we are ready to come it through.

“Ukrainian clinics have no technical possibility to do such complex operation, so we need 150-250K EUR for Israel or European or US clinic. The final decision will be made considering amount we able to find. Perhaps my family is able to get ~60% of that by selling the flat where parents leave and some other goods, but we still require external help.”

– Andrii Nikitin, MySQL Engineer

For donation: Donation can be made through PayPal (via MySQL/Sun website)

The problem with browsing was pretty simple to describe and as hard to fix - we have large data set which consists of a few tables with many fields with really bad selectivity (flag fields like is_deleted, is_private, etc; file_type, language_id , category_id and others). As the result of this situation it becomes really hard (if possible at all) to display documents lists like “most popular 1-10 pages PDF documents in Italian language from the category “Business” (of course, non-deleted, non-private, etc). If you’ll try to create appropriate indexes for each possible filters combination, you’ll end up having tens or hundreds of indexes and every INSERT query in your tables will take ages.

So, we were doing many weird things to solve this problem, like having really huge covering indexes for most popular browsing situations, having powerful (and expensive) dedicated servers for browsing queries, etc. Nothing helped and we ended up reducing filters number to 1-2 per query. But with continuously growing documents base we’d really love to help people find interesting content and filters is what we’d really want to add to the project.

Long story short, we’ve decided to try to use Sphinx search engine for really unusual thing - we decided to move all SELECT queries for our documents to a small (proof of concept) sphinx installation on one of our DB servers and use Sphinx’ full-scan feature. After a few hours of development I’ve created a simple to use module which mimics ActiveRecord’s find method interface and allows us to perform queries like the following (sometimes even more powerful than AR):

After a short testing we’ve got our performance benchmarks results and here are some facts:

• documents count - 3000000+
• re-indexing time - 20-30 seconds for an entire data set (4 large joined tables)
• longest query (surprisingly - a query without any filters) takes about 0.5 sec on one core
• generic query with 1-2 filters takes 0.01-0.05 sec on one core
• queries with many filters take 0.01 sec and less on one core
• queries time could be reduced almost linearly when adding more cores (splitting index on many pieces).
• sphinx scales really easily on more than one box if we’ll need it to.

As you can see here, it is really useful sometimes to think “out of the box” when you work with some tools - we were using sphinx for a quite a long time now, but this solution for browsing was like a blessing and really helped us to keep up with service popularity growth.