Tag Archives: open source

Back from Amsterdam and OHM

I’m back from a great week in the Netherlands, where I, in the company of good friends, visited Amsterdam and attended OHM. Before completing the lists of talks that I started in my previous post, I will share a few pictures.

Thomas and yours truly enjoying a beer at Café Captein & Co on Sunday, July 28:

20130728-thomas-and-yours-truly-at-cafe-captein-and-co

It was one of the first beers that day, and it was the beginning of a small pub crawl…

Edward, yours truly and others (hmm, I don’t exactly remember who) getting ready for dinner with a 9.5% Westmalle at OHM on Friday, August 2:

20130802-getting-ready-for-dinner-at-ohm

Some of us visited the OHM lounge in the evening:

20130802-chilling-in-the-ohm-lounge

Thomas on the floor, Jacob and Kim on the couch. A very cozy place.

While waiting for dinner to be served at the camp site restaurant just outside OHM, we had some random, geeky fun:

20130803-geek-fun

The picture shows Jacob, yours truly, Thomas and Georg. Edward is behind the camera. It was on Saturday, August 3. Note the funny name of the beer – Texels Skuumkoppe. It is a local beer, as it is brewed on the island Texel.

Awesome and semi-awesome talks I attended at OHM:

Missed talks that I will watch later:

The video recordings from the tents are not processed yet, but the OHM guys have promised that they will be, in a not so distant future…

It feels good to be home again. I think I’ve socialized plenty for the time being and now I need some privacy. Also, I’ve missed my favorite music genre – metal. We have mostly been listening to pop and light rock in the car.

Let me end by mentioning that I find Amsterdam a lot more cozy and inviting than London. I compare Amsterdam to London, as I visited the latter in February. Do not forget to rent a boat and sail through the canals when you visit Amsterdam! It’s certainly worth it.

OHM – Observe, Hack, Make

Yours truly having a cold beer in a very hot tent:

mt-ohm

My small, blue, crappy, non-waterproof tent in front of Georg’s gigantic one:

mt-ohm-tents

Awesome talks I’ve attended:

Missed talks that I should watch later:

Update: The two lists are complete in my next post.

High Availability with CARP and HAProxy

Let me start by setting the scene. You have five mail servers providing a relatively large customer base with SMTP (relay to the world), POP3(S), IMAP(S) and webmail (HTTP(S)). Other servers, which are not part of the scene, receive mails from the world to the customers and do spam filtering. Mail from the customers to the world are also piped through this filtering – you can stop your facepalm – but it is also not a part of the scene. At the moment, the five mail servers are loadbalanced using DNS round robin, which provides load sharing but no redundancy. The protocols are in charge of the redundancy.

Since only SMTP provides redundancy (delivery is retried a number of times), the customers will have a bad experience with POP3, IMAP and webmail in the event of a server crash. If their MUA does not have an outgoing queue from which it retries delivery, they will also notice problems with SMTP and risk losing outgoing mails. Additionally, many customers do not hesitate to call the service desk, if their newly written mail does not disappear from their MUA’s queue immediately after they have clicked send. This increases the service desk load, which we also want to avoid.

It is clear that the situation is not ideal. There is no redundancy. Both uncontrolled and controlled downtime (scheduled maintenance) will disturb the customers’ use of mail services. The servers run FreeBSD, so we will utilize CARP and HAProxy to add redundancy without adding extra mail servers next to the existing ones or adding loadbalancer servers in front of the mail servers. Originally, my mind was set on keepalived, as I have some experience with it on Linux, but it seems that CARP is the weapon of choice in the *BSD world. As a curiosity, the FreeBSD port of keepalived expired in November 2011. The idea behind the improved setup is that CARP lets us have a number of virtual IP addresses that float between the servers. HAProxy provides service monitoring on each server and directs connections to the other servers if the local services are down.

It is important for me to stress that the presented setup in this post is only one out of many setups that can achieve load sharing and redundancy. Its pros and cons must be compared to the pros and cons of other ways that could be chosen in a given situation. Before mentioning a few cons, I want to start with the big, obvious pro. In a typical CARP/keepalived and HAProxy setup, in my experience, you have two dedicated loadbalancer servers in active/passive mode, i.e. at a particular point in time, one server sits idle, while all traffic flows through the other server. Failover to the idle server provides redundancy, and loadbalancing over a number of application servers provides load sharing. The single active server, however, is still a bottleneck. If your infrastructure provides 1 or even 10 Gbps connectivity, and your setup only handles e.g. 200 Mbps of traffic, this might not be a problem. Nonetheless, the dedicated loadbalancer servers are still at least two extra servers that can be avoided, if the application servers themselves, in coorporation with the underlying network infrastructure, are able to perform load sharing and redundancy. The setup presented here enables the application servers to do so.

The cons are that the solution is a bit complex, the procedures for maintenance and scaling are a bit complex as well, and the chosen loadbalancer, HAProxy, lacks some useful features. I have only looked at version 1.4 of HAProxy, so the features might not be lacking in newer versions. In addition, I might have missed something. I operate a Riverbed Stingray Traffic Manager on a daily basis, which means that I am used to many possibilities for session persistence, extensive traffic scripting, built-in checks for many types of services, SSL offloading, etc. It would be nice to offload the SSL to HAProxy and to have built-in, deep checks for POP3 and IMAP. We have to do without these things.

The setup consists of six servers:

Hostname Address Virtual address 1 Virtual address 2
mail01.test.netic.dk 192.168.42.11 192.168.42.21 192.168.42.26
mail02.test.netic.dk 192.168.42.12 192.168.42.22 192.168.42.27
mail03.test.netic.dk 192.168.42.13 192.168.42.23 192.168.42.28
mail04.test.netic.dk 192.168.42.14 192.168.42.24 192.168.42.29
mail05.test.netic.dk 192.168.42.15 192.168.42.25 192.168.42.30
client01.test.netic.dk 192.168.42.100

We have DNS round robin for the ten virtual addresses for the records {relay, mail, smtp, pop3, pop3s, imap, imaps, webmail}.test.netic.dk. Example:

[root@client01 ~]# dig mail.test.netic.dk a | grep ^mail | sort
mail.test.netic.dk.	900	IN	A	192.168.42.21
mail.test.netic.dk.	900	IN	A	192.168.42.22
mail.test.netic.dk.	900	IN	A	192.168.42.23
mail.test.netic.dk.	900	IN	A	192.168.42.24
mail.test.netic.dk.	900	IN	A	192.168.42.25
mail.test.netic.dk.	900	IN	A	192.168.42.26
mail.test.netic.dk.	900	IN	A	192.168.42.27
mail.test.netic.dk.	900	IN	A	192.168.42.28
mail.test.netic.dk.	900	IN	A	192.168.42.29
mail.test.netic.dk.	900	IN	A	192.168.42.30

It might be okay to use a greater TTL. Caching resolvers should cache the entire record sets and answer clients in a round robin fashion. We are not interested in changing the records frequently. The six servers are FreeBSD 9.1 amd64 guests in VirtualBox on my laptop (click on the images to view them unscaled):

20130615-high-availability-with-carp-and-haproxy-01

The terminal multiplexer tmux makes it easy to give an overview:

20130615-high-availability-with-carp-and-haproxy-02

In FreeBSD 9.1, CARP is available as a kernel module. I just added the line ‘if_carp_load=”YES”‘ to /boot/loader.conf and rebooted. The device configuration takes place in /etc/rc.conf. The handbook has an example – I will post configuration details in the end of the post.

I started out with one virtual address per server rather than two. This is shown on the following two images:

20130615-high-availability-with-carp-and-haproxy-03   20130615-high-availability-with-carp-and-haproxy-04

The advskew values are chosen such that the server “to the right” takes over in case of a crash, i.e. mail02 takes over for mail01, mail03 for mail02, …, mail01 for mail05. Think of the five servers being placed in a ring. One virtual address per server, however, has the disadvantage that a lot of traffic forwarding work might be put on one server, which might cause a domino effect. The second image shows the uneven distribution after mail01, mail02 and mail03 have crashed. Their virtual addresses have moved to mail04, while mail05 still only has one address.

Rather than building some sort of service which continuously distributes the virtual addresses evenly between operational servers, I decided to upgrade to two virtual addresses per server. Both solutions are complex, but the one that I have chosen does not require an extra server or any scripting. In return it probably does not scale as well as the other solution.

The following six images show the resulting setup and test the most obvious failover scenarios. The two servers “on each side” of a server takes over in the event of a crash. In this way, the traffic forwarding work is divided somewhat more evenly.

20130615-high-availability-with-carp-and-haproxy-05   20130615-high-availability-with-carp-and-haproxy-06   20130615-high-availability-with-carp-and-haproxy-07

20130615-high-availability-with-carp-and-haproxy-08   20130615-high-availability-with-carp-and-haproxy-09   20130615-high-availability-with-carp-and-haproxy-10

Note that a CARP interface starts out in the backup state, when it is brought up. We want virtual addresses to float back to their original server, when the server becomes operational. My solution, which is not thoroughly tested at this point, is to have a cronjob at boot and every tenth minute that sets the state, e.g. on mail01:

[root@mail01 ~]# tail -n 2 /etc/crontab 
@reboot root /bin/sleep 30; /sbin/ifconfig carp0 state master; /sbin/ifconfig carp5 state master
*/10 * * * * root /sbin/ifconfig carp0 state master; /sbin/ifconfig carp5 state master

If you decide to create the aforementioned virtual address distribution service, these cronjobs becomes unnecessary (and disturbing).

The following image shows that HAProxy has entered the scene and that Postfix answers on the virtual addresses:

20130615-high-availability-with-carp-and-haproxy-11

The haproxy.conf shown on the image has been changed a bit. It turned out that checks every tenth second, coming from five HAProxy instances, generate many useless lines of log. So far, I have chosen only to monitor the backup servers/services once every minute. (Correction: The uploaded haproxy.conf files at the bottom of this post still checks all servers every tenth second. It is left as an exercise to the reader to adjust this for his/her particular setup.)

HAProxy on a given server is configured such that it only forwards to services on the other servers, if its own services are down. If one of its own services is down, it loadbalances in a round robin fashion over the corresponding service on the other servers. The idea is that we do not want to generate extra traffic between the servers, if we do not have to. It should be more the exception than the rule that a local service is down.

The following two images show tests of CARP failover, from a client’s point of view:

20130615-high-availability-with-carp-and-haproxy-12   20130615-high-availability-with-carp-and-haproxy-13

The following two images show a HAProxy failover test and the HATop ncurses client, respectively:

20130615-high-availability-with-carp-and-haproxy-14   20130615-high-availability-with-carp-and-haproxy-15

Two things about the image to the right: a) HATop are useful for gathering statistics and for toggling service maintenance. b) The glimpse of Postfix’ lines in /var/log/maillog reveals that services no longer see the real source address of a client – they only see the source addresses of the different HAProxy instances.

Especially the remark about source addresses is important. Many connections/requests from one source address might trigger an abuse detection/prevention mechanism in a service. Lookups in e.g. RBLs will not make sense. Most protection mechanisms must be migrated from the services to HAProxy. Finally, the HAProxy access log is necessary to link internal and external addresses. Intelligent log collection tools like Splunk can be configured to do this, which means that it might not be a problem.

The services only listen on 127.0.0.1 and 192.168.42.1[1-5], while HAProxy listens on all interfaces, i.e. also on the floating virtual addresses:

[root@mail01 ~]# netstat -an | grep LISTEN | sort
tcp4       0      0 *.110                  *.*                    LISTEN
tcp4       0      0 *.143                  *.*                    LISTEN
tcp4       0      0 *.25                   *.*                    LISTEN
tcp4       0      0 *.587                  *.*                    LISTEN
tcp4       0      0 10.0.3.15.22           *.*                    LISTEN
tcp4       0      0 127.0.0.1.22           *.*                    LISTEN
tcp4       0      0 127.0.0.1.8025         *.*                    LISTEN
tcp4       0      0 127.0.0.1.8110         *.*                    LISTEN
tcp4       0      0 127.0.0.1.8143         *.*                    LISTEN
tcp4       0      0 127.0.0.1.8587         *.*                    LISTEN
tcp4       0      0 192.168.42.11.22       *.*                    LISTEN
tcp4       0      0 192.168.42.11.8025     *.*                    LISTEN
tcp4       0      0 192.168.42.11.8110     *.*                    LISTEN
tcp4       0      0 192.168.42.11.8143     *.*                    LISTEN
tcp4       0      0 192.168.42.11.8587     *.*                    LISTEN

I know I also mentioned the protocols POP3S, IMAPS and HTTP(S) (webmail) in the beginning, but these services are not yet installed, and the corresponding frontends and backends in HAProxy are disabled. In haproxy.conf, “balance source” is added to HTTP(S) backends, as session persistence is needed in the event of round robin loadbalancing over backup servers/services. As is evident from the current, non-redundant setup, persistence is not needed with pure DNS round robin, as a browser gets a single DNS reply from the operating system and remembers that for at least some time (15-30 minutes have been observed). This is yet another reason for keeping addresses alive rather than editing DNS records when performing scheduled maintenance or when working around a crashed server.

Let me end this saga with links to scripts and configuration files:

Easter egg on the occasion of Copenhell:

As I Lay Dying – An Ocean Between Us

Back from Puppet training in Belgium

A colleague and I attended the three-day course Advanced Puppet Training in Belgium from Tuesday to Thursday, and we are now back in Denmark. Actually, we reached Aalborg around midnight between Thursday and Friday. The course instructor was Ger Apeldoorn, a freelance Puppet consultant from the Netherlands. He had put together a very interesting course and we look forward to applying our new knowledge at Netic. The course was held at Open-Future, a Belgian company specializing in Linux and open source in general.

I highly recommend the course. It was a well-balanced mix of lectures and lab exercises. Puppet now impresses me even more.

On Wednesday we visited a nearby shopping center, where I managed to pick up a few Belgian treats 🙂

We drove the approx 2100 km (roundtrip), but due to a pretty decent car (the Netic VW Touran) and the many long stretches of German highway it wasn’t a problem at all.

New hardware for dotsrc.org

Having just expressed how much I needed a relaxing weekend after two stressful weeks, I ended up doing voluntary work most of Sunday for the biggest open source software mirror in Denmark 🙂

Georg Sluyterman and yours truly configured, transported, mounted and connected six donated Dell servers. Okay, to be precise, one of them wasn’t mounted and connected, as it’s a backup server that will be moved to another location. Additionally, we threw out a number of old servers corresponding to half a full-size rack. I’m not revealing the donor of the new servers, as I presently don’t know whether they want the publicity.

New hardware for dotsrc.org on February 17, 2013 - picture 1 of 7 - the 125-150 kg of hardware was transported in my Aygo   New hardware for dotsrc.org on February 17, 2013 - picture 2 of 7 - yours truly getting intimate with the new hardware in a rather small elevator...   New hardware for dotsrc.org on February 17, 2013 - picture 3 of 7 - the two Dotsrc.org racks before we got started

New hardware for dotsrc.org on February 17, 2013 - picture 4 of 7 - the two Dotsrc.org racks with five of the new servers mounted in the bottom of the left rack   New hardware for dotsrc.org on February 17, 2013 - picture 5 of 7 - five of the new servers   New hardware for dotsrc.org on February 17, 2013 - picture 6 of 7 - many kilograms of old hardware that is about to be thrown out

New hardware for dotsrc.org on February 17, 2013 - picture 7 of 7 - the old hardware and yours truly

Speaker placement and fighting with XBMC etc.

As mentioned in my post yesterday, I have been moving my speakers. My first plan was to build angled mounting brackets out of wood, which the speakers could hang from, like when they hang flat on a wall. I found out, however, that my skills with a handsaw on my dining table without a clamp are non-existent. I kept the screws and will donate the pieces of wood to my parents’ stove.

Instead, I pointed my Aygo towards Ikea and bought two shelves and four brackets. I like the result. Judge for yourself. Before and after shots:

Old placement of my speakers (January 12, 2013)   New place of my speakers (January 13, 2013)

My media center setup with Raspbmc on a Raspberry Pi currently has a problem. I tend to believe the problem mainly lies with XBMC, which is the media center part of the the Linux distribution Raspbmc. Without XBMC’s AudioEngine (AE) enabled, i.e. with default settings, playback always stops after one song. I have already enabled “automatically play next song” (!!!). If I want it to play the next song, I have to press x (to stop playback mode) and press enter on the next song (to start playback mode on that song). When it is done playing the next song, the problem repeats. It doesn’t make a difference whether I choose HDMI or analog audio output.

With AE enabled and the kernel module snd-bcm2835 loaded (AE will not use HDMI output without it), the problem actually gets worse. Before I reveal why, I must admit that AE for a short while made me happy, as its digital to analog conversion (done by PulseAudio) is clearly better than the DAC in my tv. At the moment, my old fashioned, analog, stereo amplifier is connected to the headphone output on my tv. The DAC in my tv is almost as bad as the analog output on the Raspberry Pi itself. Running with AE enabled is worse, because XBMC crashes when it is done playing a song. It automatically restarts, though, so it is not unavailable for long. Well, the point is that XBMC crashing is worse than having to manually start the next song. Which is worse doesn’t really matter, however, as both makes the platform unusable for simple audio playback (in my humble opinion…). Hopefully, both problems will be solved in a not so distant future. Until then I’ll probably just connect a simple mp3 player to my amplifier.

I have ordered a Cablesson HDelity HDMI Audio Extractor (ARC) from the German Amazon. It should provide me with excellent digital to analog conversion, but it will not solve the next-song-problem. Let me cry a bit more – it almost costs the same as two Raspberry Pi boards.

On top of all, I have tried to use a Terratec Aureon Dual USB DAC, which I got for Christmas. I haven’t had any luck getting it to work with Raspbmc, and I’ve tried both inside and outside XBMC. I have also tried limiting USB to 1.0/1.1 (lowest speed), but that made Raspbmc stop working, and I had to put the memory card into another computer and remove the boot option. Of course, I have tested with and without a powered USB hub.

Update, 2013-01-14 19:25:42: The next-song-problem was fixed by installing the latest nightly build of XBMC.

Helping people

Okay, I know I promised to write some stories to the pictures in Pictures tell stories, but that will either happen later or never…

Here is a feel-good episode from work – or at least related to work.

A guy asks on the channel #bsd-dk on EFnet for a piece of software to receive UDP packets and forward them to a number of hosts:

09:28:46 <some_guy> hmm, can anybody recommend something from
                    /usr/ports that can receive udp packets and
                    forward them to multiple hosts?

Nobody suggests anything, and some people, myself included, jokes that it will only take a “few” lines of Perl, Python or C to do the trick.

While waiting – at work – for nodes to be drained in our loadbalancer and Tomcats to deploy, I read up on sockets, putting sockets (and file handles) in an array/hash and registering a signal handler, and I write this proof of concept UDP forwarder:

A few moments later the guy responds:

11:49:19 <some_guy> toft: perfect, it works just fine. directly
                    into production it goes. thanks :-)

Of course, in retrospect, the operating system’s kernel does housekeeping of sockets when a process dies, so the lines 6, 15-23 and 31-33 are not strictly necessary. In conclusion, a “few” lines of Perl in this case is equal to 31.

The forwarder in action:

In the cloud

  

This blog is now being served by the software WordPress, Apache HTTP Server, MariaDB and Ubuntu Linux, all of which are running in a free instance in Amazon Elastic Compute Cloud.

So don’t tell me that I’m not into the game! 😛

Edit: Now I’m also using the web application accelerator Varnish. The main culprit with regard to the blog’s speed, however, seems to be HTML-resized images. This means that you always download full-size images, even though you only need small/medium versions. I haven’t found a solution for this yet. Well, at least not a free solution – the WordPress plugin ImageScaler isn’t free.

A picture of me moving the blog and blogging about it (beware of white skin!):

Silence!

I have been really happy with my Sennheiser CX-380 earphones, since I bought them approx one year and a half ago. They have been great for running, weight training and commuting with bus, train and airplane. It was therefore a sad moment when I recently discovered a loose connection in their jack. As is evident from the above picture, there are only a few copper wires for each of the four inner cables in their wiring. I am not saying that it is impossible to attach a new jack, but it requires a very steady hand and probably a magnifying glass.

As Copenhagen Marathon is approaching, and I really need good earphones for my long training runs, I have temporarily given up on fixing my broken ones and have ordered a new pair. They actually arrived today. I chose the relatively cheap Sennheiser CX-300 II Precision – approx 300 DKK including postage. Well, as long as they do a decent job of eliminating noise and staying in my ears… I hope so 🙂

Oh, and what else is new?

Saturday, April 28, I set a new record with regard to how far I have ever run (running distance PR?). Unfortunately, I’m behind on my marathon training, so don’t expect anything close to the 42,195 meters. I ran 28.2 km. I was completely fried the rest of the day. Originally, I aimed at finishing the marathon in 3h 30m, requiring a pace of 4m 59s per km, but I might have to accept 3h 45m. My worst fear is cramp in my legs and other pains that cannot be ignored. My fitness will probably not let me down. A couple of excuses:

  1. It will be my first marathon, and
  2. I ran the 28.2 km without water and other supplies.

I really hope for relatively cold weather on the big day. Enough about running for this post…

At work I constantly learn new cool things. I have started to regularly perform tasks on our Cisco network infrastructure, and I have learnt about (and started to use) Splunk, which is a great tool for log analysis and reporting. I know it will provoke hatred in some of my geeky friends, when I say this, but together with the Riverbed Stingray Traffic Manager, Splunk is a member of the (though very small) set of closed source software that I feel good about. Both products work very well, they have active communities, and I haven’t yet seen open source replacements.

Last but not least, I have received OpenBSD 5.1 today. Thanks to the developers for yet another release of my favorite operating system. After a MacBook-only period of four months, I have once again started using OpenBSD -current on a daily basis on my Thinkpad X60 laptop. It just works[tm].

Update at 22:25 CEST: I have just used the new earphones in the gym, together with my new music player armband and sweatband. The earphones lacked a “clothespin” to secure their wires to my shirt, but I just used the one that came with my old ones. Sennheiser should consider supplying one with the CX-300’s too – or with all models where the lengths of the wire for each channel differ, i.e. the models where you probably will have the wire for the right channel behind your neck. The earphones fit nicely in my ears, deliver good music quality, eliminate noise from the surroundings and don’t fall out during exercises. Compared to the crappy Apple earphones, I used for a while, it is very clear that I don’t have to turn up the volume as much. That equals less stress on my ears and longer battery life for my player. All in all I am quite happy with the purchase.