Tag Archives: it

A Visit from Thy and Vice Versa

Friday, August 30, I was visited by two friends from Thy. I served something simple and something that was at the top in my freezer – venison from roe deer. I remember having shot it with my own rifle in the winter season of 2012. Very delicious. It was accompanied by crispy bacon, scalloped potatoes and cold lager.

Most days I don’t really need a dishwasher, but it’s a really nice invention, when you have people over for dinner, and you want to spend time with them rather than stand in the kitchen. If you remember to open it while the dishes etc. are still quite hot, they’ll even get dry without your intervention. The view on Saturday morning:


Saturday I visited my parents and brother in Thy, as my father turned 54. We celebrated his birthday by taking the ferry to Feggesund and eating lots and lots of eel at Feggesund Færgekro. The eel were accompanied by potatoes, parsley sauce, Porse Guld (beer) and Rød Aalborg (schnapps). We often joke that the only good thing about the small island Mors is the view of Thy, and in fact the view is not something to complain about:


On Sunday, my father and I got up at 3:45 in the morning and went hunting for geese and ducks together with three other hunters. My father’s plan was to let me and the other guys have the best chances, as he lives closer to the hunting grounds and can go there any other day. The geese hadn’t heard his plan though, so they flew mostly over him, enabling him to shoot three of them. I helped pluck their feathers and got two of them with me home in return. In the following picture I’m trying to hold the geese up high, but they are actually quite heavy.


The weekend was concluded by a few hours of extremely serious work, where a colleague and I were on site at Netic performing a planned upgrade of MySQL from version 5.1 to 5.6 on a very important database cluster.


Back from Amsterdam and OHM

I’m back from a great week in the Netherlands, where I, in the company of good friends, visited Amsterdam and attended OHM. Before completing the lists of talks that I started in my previous post, I will share a few pictures.

Thomas and yours truly enjoying a beer at Café Captein & Co on Sunday, July 28:


It was one of the first beers that day, and it was the beginning of a small pub crawl…

Edward, yours truly and others (hmm, I don’t exactly remember who) getting ready for dinner with a 9.5% Westmalle at OHM on Friday, August 2:


Some of us visited the OHM lounge in the evening:


Thomas on the floor, Jacob and Kim on the couch. A very cozy place.

While waiting for dinner to be served at the camp site restaurant just outside OHM, we had some random, geeky fun:


The picture shows Jacob, yours truly, Thomas and Georg. Edward is behind the camera. It was on Saturday, August 3. Note the funny name of the beer – Texels Skuumkoppe. It is a local beer, as it is brewed on the island Texel.

Awesome and semi-awesome talks I attended at OHM:

Missed talks that I will watch later:

The video recordings from the tents are not processed yet, but the OHM guys have promised that they will be, in a not so distant future…

It feels good to be home again. I think I’ve socialized plenty for the time being and now I need some privacy. Also, I’ve missed my favorite music genre – metal. We have mostly been listening to pop and light rock in the car.

Let me end by mentioning that I find Amsterdam a lot more cozy and inviting than London. I compare Amsterdam to London, as I visited the latter in February. Do not forget to rent a boat and sail through the canals when you visit Amsterdam! It’s certainly worth it.

OHM – Observe, Hack, Make

Yours truly having a cold beer in a very hot tent:


My small, blue, crappy, non-waterproof tent in front of Georg’s gigantic one:


Awesome talks I’ve attended:

Missed talks that I should watch later:

Update: The two lists are complete in my next post.

High Availability with CARP and HAProxy

Let me start by setting the scene. You have five mail servers providing a relatively large customer base with SMTP (relay to the world), POP3(S), IMAP(S) and webmail (HTTP(S)). Other servers, which are not part of the scene, receive mails from the world to the customers and do spam filtering. Mail from the customers to the world are also piped through this filtering – you can stop your facepalm – but it is also not a part of the scene. At the moment, the five mail servers are loadbalanced using DNS round robin, which provides load sharing but no redundancy. The protocols are in charge of the redundancy.

Since only SMTP provides redundancy (delivery is retried a number of times), the customers will have a bad experience with POP3, IMAP and webmail in the event of a server crash. If their MUA does not have an outgoing queue from which it retries delivery, they will also notice problems with SMTP and risk losing outgoing mails. Additionally, many customers do not hesitate to call the service desk, if their newly written mail does not disappear from their MUA’s queue immediately after they have clicked send. This increases the service desk load, which we also want to avoid.

It is clear that the situation is not ideal. There is no redundancy. Both uncontrolled and controlled downtime (scheduled maintenance) will disturb the customers’ use of mail services. The servers run FreeBSD, so we will utilize CARP and HAProxy to add redundancy without adding extra mail servers next to the existing ones or adding loadbalancer servers in front of the mail servers. Originally, my mind was set on keepalived, as I have some experience with it on Linux, but it seems that CARP is the weapon of choice in the *BSD world. As a curiosity, the FreeBSD port of keepalived expired in November 2011. The idea behind the improved setup is that CARP lets us have a number of virtual IP addresses that float between the servers. HAProxy provides service monitoring on each server and directs connections to the other servers if the local services are down.

It is important for me to stress that the presented setup in this post is only one out of many setups that can achieve load sharing and redundancy. Its pros and cons must be compared to the pros and cons of other ways that could be chosen in a given situation. Before mentioning a few cons, I want to start with the big, obvious pro. In a typical CARP/keepalived and HAProxy setup, in my experience, you have two dedicated loadbalancer servers in active/passive mode, i.e. at a particular point in time, one server sits idle, while all traffic flows through the other server. Failover to the idle server provides redundancy, and loadbalancing over a number of application servers provides load sharing. The single active server, however, is still a bottleneck. If your infrastructure provides 1 or even 10 Gbps connectivity, and your setup only handles e.g. 200 Mbps of traffic, this might not be a problem. Nonetheless, the dedicated loadbalancer servers are still at least two extra servers that can be avoided, if the application servers themselves, in coorporation with the underlying network infrastructure, are able to perform load sharing and redundancy. The setup presented here enables the application servers to do so.

The cons are that the solution is a bit complex, the procedures for maintenance and scaling are a bit complex as well, and the chosen loadbalancer, HAProxy, lacks some useful features. I have only looked at version 1.4 of HAProxy, so the features might not be lacking in newer versions. In addition, I might have missed something. I operate a Riverbed Stingray Traffic Manager on a daily basis, which means that I am used to many possibilities for session persistence, extensive traffic scripting, built-in checks for many types of services, SSL offloading, etc. It would be nice to offload the SSL to HAProxy and to have built-in, deep checks for POP3 and IMAP. We have to do without these things.

The setup consists of six servers:

Hostname Address Virtual address 1 Virtual address 2

We have DNS round robin for the ten virtual addresses for the records {relay, mail, smtp, pop3, pop3s, imap, imaps, webmail}.test.netic.dk. Example:

[root@client01 ~]# dig mail.test.netic.dk a | grep ^mail | sort
mail.test.netic.dk.	900	IN	A
mail.test.netic.dk.	900	IN	A
mail.test.netic.dk.	900	IN	A
mail.test.netic.dk.	900	IN	A
mail.test.netic.dk.	900	IN	A
mail.test.netic.dk.	900	IN	A
mail.test.netic.dk.	900	IN	A
mail.test.netic.dk.	900	IN	A
mail.test.netic.dk.	900	IN	A
mail.test.netic.dk.	900	IN	A

It might be okay to use a greater TTL. Caching resolvers should cache the entire record sets and answer clients in a round robin fashion. We are not interested in changing the records frequently. The six servers are FreeBSD 9.1 amd64 guests in VirtualBox on my laptop (click on the images to view them unscaled):


The terminal multiplexer tmux makes it easy to give an overview:


In FreeBSD 9.1, CARP is available as a kernel module. I just added the line ‘if_carp_load=”YES”‘ to /boot/loader.conf and rebooted. The device configuration takes place in /etc/rc.conf. The handbook has an example – I will post configuration details in the end of the post.

I started out with one virtual address per server rather than two. This is shown on the following two images:

20130615-high-availability-with-carp-and-haproxy-03   20130615-high-availability-with-carp-and-haproxy-04

The advskew values are chosen such that the server “to the right” takes over in case of a crash, i.e. mail02 takes over for mail01, mail03 for mail02, …, mail01 for mail05. Think of the five servers being placed in a ring. One virtual address per server, however, has the disadvantage that a lot of traffic forwarding work might be put on one server, which might cause a domino effect. The second image shows the uneven distribution after mail01, mail02 and mail03 have crashed. Their virtual addresses have moved to mail04, while mail05 still only has one address.

Rather than building some sort of service which continuously distributes the virtual addresses evenly between operational servers, I decided to upgrade to two virtual addresses per server. Both solutions are complex, but the one that I have chosen does not require an extra server or any scripting. In return it probably does not scale as well as the other solution.

The following six images show the resulting setup and test the most obvious failover scenarios. The two servers “on each side” of a server takes over in the event of a crash. In this way, the traffic forwarding work is divided somewhat more evenly.

20130615-high-availability-with-carp-and-haproxy-05   20130615-high-availability-with-carp-and-haproxy-06   20130615-high-availability-with-carp-and-haproxy-07

20130615-high-availability-with-carp-and-haproxy-08   20130615-high-availability-with-carp-and-haproxy-09   20130615-high-availability-with-carp-and-haproxy-10

Note that a CARP interface starts out in the backup state, when it is brought up. We want virtual addresses to float back to their original server, when the server becomes operational. My solution, which is not thoroughly tested at this point, is to have a cronjob at boot and every tenth minute that sets the state, e.g. on mail01:

[root@mail01 ~]# tail -n 2 /etc/crontab 
@reboot root /bin/sleep 30; /sbin/ifconfig carp0 state master; /sbin/ifconfig carp5 state master
*/10 * * * * root /sbin/ifconfig carp0 state master; /sbin/ifconfig carp5 state master

If you decide to create the aforementioned virtual address distribution service, these cronjobs becomes unnecessary (and disturbing).

The following image shows that HAProxy has entered the scene and that Postfix answers on the virtual addresses:


The haproxy.conf shown on the image has been changed a bit. It turned out that checks every tenth second, coming from five HAProxy instances, generate many useless lines of log. So far, I have chosen only to monitor the backup servers/services once every minute. (Correction: The uploaded haproxy.conf files at the bottom of this post still checks all servers every tenth second. It is left as an exercise to the reader to adjust this for his/her particular setup.)

HAProxy on a given server is configured such that it only forwards to services on the other servers, if its own services are down. If one of its own services is down, it loadbalances in a round robin fashion over the corresponding service on the other servers. The idea is that we do not want to generate extra traffic between the servers, if we do not have to. It should be more the exception than the rule that a local service is down.

The following two images show tests of CARP failover, from a client’s point of view:

20130615-high-availability-with-carp-and-haproxy-12   20130615-high-availability-with-carp-and-haproxy-13

The following two images show a HAProxy failover test and the HATop ncurses client, respectively:

20130615-high-availability-with-carp-and-haproxy-14   20130615-high-availability-with-carp-and-haproxy-15

Two things about the image to the right: a) HATop are useful for gathering statistics and for toggling service maintenance. b) The glimpse of Postfix’ lines in /var/log/maillog reveals that services no longer see the real source address of a client – they only see the source addresses of the different HAProxy instances.

Especially the remark about source addresses is important. Many connections/requests from one source address might trigger an abuse detection/prevention mechanism in a service. Lookups in e.g. RBLs will not make sense. Most protection mechanisms must be migrated from the services to HAProxy. Finally, the HAProxy access log is necessary to link internal and external addresses. Intelligent log collection tools like Splunk can be configured to do this, which means that it might not be a problem.

The services only listen on and[1-5], while HAProxy listens on all interfaces, i.e. also on the floating virtual addresses:

[root@mail01 ~]# netstat -an | grep LISTEN | sort
tcp4       0      0 *.110                  *.*                    LISTEN
tcp4       0      0 *.143                  *.*                    LISTEN
tcp4       0      0 *.25                   *.*                    LISTEN
tcp4       0      0 *.587                  *.*                    LISTEN
tcp4       0      0           *.*                    LISTEN
tcp4       0      0           *.*                    LISTEN
tcp4       0      0         *.*                    LISTEN
tcp4       0      0         *.*                    LISTEN
tcp4       0      0         *.*                    LISTEN
tcp4       0      0         *.*                    LISTEN
tcp4       0      0       *.*                    LISTEN
tcp4       0      0     *.*                    LISTEN
tcp4       0      0     *.*                    LISTEN
tcp4       0      0     *.*                    LISTEN
tcp4       0      0     *.*                    LISTEN

I know I also mentioned the protocols POP3S, IMAPS and HTTP(S) (webmail) in the beginning, but these services are not yet installed, and the corresponding frontends and backends in HAProxy are disabled. In haproxy.conf, “balance source” is added to HTTP(S) backends, as session persistence is needed in the event of round robin loadbalancing over backup servers/services. As is evident from the current, non-redundant setup, persistence is not needed with pure DNS round robin, as a browser gets a single DNS reply from the operating system and remembers that for at least some time (15-30 minutes have been observed). This is yet another reason for keeping addresses alive rather than editing DNS records when performing scheduled maintenance or when working around a crashed server.

Let me end this saga with links to scripts and configuration files:

Easter egg on the occasion of Copenhell:

As I Lay Dying – An Ocean Between Us

One of those weeks

I’m slacking on my couch thinking back on the past week… I have been quite busy, at least work-wise. For some reason, however, my list of assigned tasks/changes/service requests/etc. has grown from 18 to 34. *sigh* 🙂

I’m not really accustomed to saying no or saying “Listen, that sounds mighty exciting, and I would love to set that up, but I’m swamped as it is.” or “What’s the priority? Is it allowed preempt task X and Y, because it sort of has to, if you want it done.”. In my experience, most people want to please others, and I’m no different. Saying no hurts. You also feel a bit inadequate. People are poking at your limits.

Maybe the title of this post is misleading. “One of those weeks” sounds like a recurring event, and this week has actually been somewhat special. My musings above should indicate that. The only recurring thing, which probably everybody can relate to, is that I’m a little worn out and need the weekend 🙂

Being busy isn’t all bad. It indicates that I have something to do, which isn’t a given in any industry these days. Also, during the week I got the chance to play with FreeBSD and CARP for a customer. Lovely software. Easy to set up and it just works. No matter what you think about virtualization in the server room, it’s an awesome invention for testing stuff. In this case I installed VirtualBox and created three guests with FreeBSD 9.1. Using the camera in my phone, I created the following two videos:

Simple test of FreeBSD 9.1 and CARP – part 1 of 2

Simple test of FreeBSD 9.1 and CARP – part 2 of 2

Have a nice weekend wherever you are! I will be cleaning my apartment, washing clothes, getting a visit from my parents (and maybe my brother) and looking at a house. The house is located in the small village Store Brøndum (a funny name for a rather small village…), which is in the countryside. It’s approximately midway between Aalborg and Hadsund. I don’t want to unveil more at this point.

Party on if you’re at Copenhell \m/

Back from Puppet training in Belgium

A colleague and I attended the three-day course Advanced Puppet Training in Belgium from Tuesday to Thursday, and we are now back in Denmark. Actually, we reached Aalborg around midnight between Thursday and Friday. The course instructor was Ger Apeldoorn, a freelance Puppet consultant from the Netherlands. He had put together a very interesting course and we look forward to applying our new knowledge at Netic. The course was held at Open-Future, a Belgian company specializing in Linux and open source in general.

I highly recommend the course. It was a well-balanced mix of lectures and lab exercises. Puppet now impresses me even more.

On Wednesday we visited a nearby shopping center, where I managed to pick up a few Belgian treats 🙂

We drove the approx 2100 km (roundtrip), but due to a pretty decent car (the Netic VW Touran) and the many long stretches of German highway it wasn’t a problem at all.

Feaster in F-klubben

Friday, April 5, I attended the Feaster event in F-klubben. It was a traditional Easter dinner with marinated herring, warm liver pate, meatballs, assorted salads, cheese, schnapps, beer, etc. Sometimes cause for confusion, in F-klubben, the name of all events must start with F, if in any way possible. Hence, Easter becomes Feaster 😉 F-klubben is a social club at the scientific faculty of Aalborg University, and it covers students and employees at the departments for computer science, math and physics. When I studied at, and later was employed by, Aalborg University, I regularly both (co-)arranged and attended events in the club. I was particularly active in F-kult, which is F-klubben’s cult film club.

As witnessed by F-klubbens sales system (“stregsystemet”), it was my first real visit in almost three years! At least I bought something very sensible on my previous visit… Limfjords-porter, of course. The brew for the elite 🙂

It was nice to say hi to a number of familiar faces, see the computer science building, and hear what is going on in the department. The evening and night also included a visit to the bar at the dormitory I lived at while being a PhD student (Nordjyske Kollegium) and a visit to the home of my old friend, classmate and colleague, Thomas Bøgholm, where we tasted some good whiskys.

Daylight saving time

Here’s a glimpse of my Saturday morning:


Freshly pressed Aeropress coffee made from freshly ground beans, a car magazine, a hunting magazine and a Puppet book.

The coming night, Denmark, together with many other countries, will switch to daylight saving time. This means we will advance our clocks one hour and therefore go from UTC+1 to UTC+2, or, alternatively, from CET to CEST. Less sun in the morning, more sun in the evening. Let’s say you get up at 6 AM on Monday. If the clock hadn’t been advanced, it would only show 5 AM. I’m just thinking out loud here… 🙂

I’m on call for my workplace when the clocks are advanced. That actually worried me a bit, but then I realized that I had mixed up two events – the insertion of a leap second on Saturday, June 30, 2012, and the switch to standard time on Sunday, October 28, 2012. It was the former that caused massive problems. As far as I know we (system administrators) rarely see problems related to the rather regular switches between standard time and summer time. Phew.

Renewed license to kill

In pure James Bond style, I’ve just received my license for the coming hunting season 🙂

I look forward to beautiful, warm summer mornings and evenings from May 16 to July 15, the summer season in Denmark for roebucks. I plan on keeping mosquitos from biting me using a hat with a net and several layers of clothes. Sunset, the smell of spruce trees, coffee, my rifle, silence… Sweet.

Since this is a public blog, let me just point out that I do not store my hunting weapons on my home address.

Totally unrelated: The chat service Microsoft Messenger is closing down, and I’m not switching to Skype for personal use. This means that if you want to chat with me after April 8, you have two options: mt@martintoft.dk on Google Talk or toft on IRC (EFnet, freenode and IRCnet). Non-geeky people should probably disregard the IRC option 😉

New hardware for dotsrc.org

Having just expressed how much I needed a relaxing weekend after two stressful weeks, I ended up doing voluntary work most of Sunday for the biggest open source software mirror in Denmark 🙂

Georg Sluyterman and yours truly configured, transported, mounted and connected six donated Dell servers. Okay, to be precise, one of them wasn’t mounted and connected, as it’s a backup server that will be moved to another location. Additionally, we threw out a number of old servers corresponding to half a full-size rack. I’m not revealing the donor of the new servers, as I presently don’t know whether they want the publicity.

New hardware for dotsrc.org on February 17, 2013 - picture 1 of 7 - the 125-150 kg of hardware was transported in my Aygo   New hardware for dotsrc.org on February 17, 2013 - picture 2 of 7 - yours truly getting intimate with the new hardware in a rather small elevator...   New hardware for dotsrc.org on February 17, 2013 - picture 3 of 7 - the two Dotsrc.org racks before we got started

New hardware for dotsrc.org on February 17, 2013 - picture 4 of 7 - the two Dotsrc.org racks with five of the new servers mounted in the bottom of the left rack   New hardware for dotsrc.org on February 17, 2013 - picture 5 of 7 - five of the new servers   New hardware for dotsrc.org on February 17, 2013 - picture 6 of 7 - many kilograms of old hardware that is about to be thrown out

New hardware for dotsrc.org on February 17, 2013 - picture 7 of 7 - the old hardware and yours truly