LVS with HA for Win2k Terminal Servers
Preface
This document describes how to build, install, and configure Linux Virtual Server Direct Routing (LVS-DR) with High Availability (HA) to load balance a Windows 2000 Terminal Server (Win2kTS) cluster. Short names in the previous parenthesis will be used throughout the document.
I highly suggest you take a look at the non-static version of this document here: http://footon.jheslop.com/howto/LVS-HA-win2k-howto.html
This document is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY, either expressed or implied. While every effort has been taken to ensure the accuracy of the information documented herein, the author(s)/editor(s)/maintainer(s)/contributor(s) assumes NO RESPONSIBILITY for any errors, or for any damages, direct or consequential, as a result of the use of the information documented herein.
Introduction
This is an attempt to collect knowledge needed to setup two (or more) LVS-DR/HA nodes for a Win2kTS cluster. The entire process of getting, compiling, installing, and configuring this process will be covered. Pointers to LVS-DR, HA, and other projects will be included as needed. This version of the HowTo is for IPVS Netfilter 1.1.8 http://www.linuxvirtualserver.org/software/ipvs.html for the 2.6 linux kernel, ipvsadm 1.24, and heartbeat 1.0.4. HowTo will be Gentoo specific, but will have alternate distribution information as needed.
All previous versions of IPVS, ipvadm, and heartbeat are either obsolete or not guaranteed. This document makes no promises as to the success of getting any of the previous versions working.
The main purpose of this setup is to connect Microsoft Windows clients using Remote Desktop Connection tool to Windows 2000 Terminal Servers. You can alter use this information for other purposes as well.
Latest version
The latest versions of this document will be kept in this location:
http://footon.jheslop.com/howto/
Disclaimer
This document is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY, either expressed or implied. While every effort has been taken to ensure the accuracy of the information documented herein, the author(s)/editor(s)/maintainer(s)/contributor(s) assumes NO RESPONSIBILITY for any errors, or for any damages, direct or consequential, as a result of the use of the information documented herein.
Authors
List of everyone who has put words into this file.
[1] Jeremy Heslop
Please notify the HowTo maintainer if you believe you should be listed above
What is an LVS?
A Linux Virtual Server (LVS) is a cluster of servers which appears to be one server to an outside client. In this situation it allows for many Windows 2000 Terminal Servers to be accessed by using one IP address and to balance the load across them. For more detailed information please consult the LVS HowTo http://www.austintek.com/LVS/LVS-HOWTO/
What is DR?
You can choose from three setups when using LVS: NAT, TUN, or DR. DR stands for Direct Routing. To find out more about the other two setups consult the LVS documentation http://www.linuxvirtualserver.org/Documents.html We will use DR because it allows higher bandwidth. Once the client makes a connection to the LVS server and gets redirected to a Win2KTS the client directly communicates with the terminal server and therefore does not use the LVS bandwidth. There are some limitations and advantages to all three setups so please make sure which one will work for you best.
Why do I want LVS?
In this setup LVS will balance out the load between a cluster of Windows 2000 Terminal Servers. Typically when a user has a list of IPs or computer they choose the last one they connected to or the first in the list. This leads to having an over abundance of clients on one or two servers with the other servers being relatively open. With LVS and it's weighted least-connection scheduling we will be able to connect clients to servers that have the least load thus increasing clients productivity because server response will be better.
What is HA?
High Availability Cluster (HA) is a cluster that allows a host (or hosts) to become Highly Available, that means if one node goes down (or a service on that node goes down) another node can pick up the service or node and take over from the failed machine. http://linux-ha.org
Why do I want HA?
The HA cluster will allow your LVS cluster to continue to function even if the main HA cluster node goes down. This means a high amount of redundancy is introduced into the equation allowing for more uptime if a hardware or software component fails.
Setting up LVS
Setting up LVS is a straight forward procedure. We will need to assume a few things before we can continue: A working Linux computer such as Gentoo Linux 1.4, all computers including the terminal servers on the same network, and the appropriate hardware on all the computers (NIC cards for each machine).
Compiling the kernel with LVS
The appropriate code we need to setup LVS is included in the Linux kernel 2.6.0 and above. There are patches for 2.2 and 2.4 kernels here: http://www.linuxvirtualserver.org
Do this in gentoo to get the latest 2.6.0 sources:
emerge sys-kernel/gentoo-dev-sources
You will need to download a Linux 2.6.0 kernel from http://www.kernel.org uncompress it and then type in either:
make menuconfig make xconfig
or
make gconfig
Configuring the kernel is out of the scope of this document. There are plenty of resources available on the web that explain it in more detail.
Here are the kernel compile options needed for LVS:
Code maturity level options --->
[*] Prompt for development and/or incomplete code/drivers
Device Drivers --->
Networking support --->
Networking options --->
[*] Network packet filtering (replaces ipchains) --->
[ ] Network packet filtering debugging
IP: Netfilter Configuration --->
IP: Virtual Server Configuration --->
<M> IP virtual server support (EXPERIMENTAL)
[*] IP virtual server debugging
(12) IPVS connection table size (the Nth power of 2)
--- IPVS transport protocol load balancing support
[*] TCP load balancing support
[*] UDP load balancing support
[ ] ESP load balancing support
[ ] AH load balancing support
--- IPVS scheduler
<M> round-robin scheduling
<M> weighted round-robin scheduling
<M> least-connection scheduling scheduling
<M> weighted least-connection scheduling
<M> locality-based least-connection scheduling
<M> locality-based least-connection with replication scheduling
<M> destination hashing scheduling
<M> source hashing scheduling
--- IPVS application helper
<M> FTP protocol helper
This will include all the LVS modules you need for this setup. Then compile your kernel:
make && make modules_install
If you have lilo or grub under Gentoo you can do a make install and get the kernel copied into /boot. Just remember to make sure you mount /boot first. Also make sure to reflect the changes of your new kernel's name and location under
/etc/lilo.conf or /boot/grub/grub.conf
Then restart your computer.
Build and Install Ipvsadm
This is relatively easy under gentoo, but because we are using the 2.6 kernel we will need to make sure we use the latest ebuild.
cd /usr/portage/sys-cluster/ipvsadm emerge ipvsadm-1.24.ebuild
For those without gentoo head on over to the LVS main website and download the latest version of ipvasdm then follow the LVS instructions on building it for your system.
You should now have a fully functional LVS system without HA functionality.
Setting up HA
Setting up HA has a few steps. We need to compile and install the heartbeat and ldirectord programs, configure the ha.cf, haresource, and ldirectord.cf files and make sure we can get a heartbeat of some kind between the HA nodes. Remember each HA node needs to be mostly identical in setup. That is each needs LVS setup and HA setup as well to function.
Compiling heartbeat with ldirectord
Again Gentoo makes this easy with an ebuild, but make sure we have ldirectord built in:
USE='ldirectord' emerge heartbeat
The only thing I found that gentoo did not properly setup was the /var/lig/heartbeat directory. So lets make sure we create that:
mkdir /var/lib/heartbeat
For other Linux distributions please check the HA website, but it is only a matter of compiling and installing the heartbeat program.
Configuring HA: ha.cf, haresources, authkeys, ldirectord.cf
This was the hardest part of the setup for me even though I had done it one time in the past. First we will copy the example files into the /etc/ha.d/ folder.
cd /usr/share/doc/heartbeat-1.0.4/ cp ha.cf haresources /etc/ha.d
Here are the options I have in ha.cf with comments stripped out:
logfacility local0 keepalive 2 deadtime 30 warntime 10 initdead 120 ucast plip0 192.168.0.1 node lvs1 node lvs2
To get a more detailed explanation of each line please consult the HA documentation. I had to make sure I had the correct IPs for my lvs1 and lvs2 nodes. The ucast option will be the IP address for the other node in the parallel port network. I also made references to the plip IPs for readability. Here is what I put in /etc/hosts:
127.0.0.1 localhost 10.140.20.7 lvs1 10.140.20.8 lvs2 10.140.20.9 lvs # plip entries 192.168.0.1 localplip 192.168.0.2 remoteplip
The above /etc/hosts file was from the lvs1 node. You will need to switch the plip ips in the lvs2 node's /etc/hosts file. I used parallel cables to create the heartbeat between the two nodes and setup a private network for just these two computers. You will need to use another setup if you plan on using more than two. I could have put extra NIC cards and used crossover cables as well, but there where no extra NIC cards at the time. I would advise using extra NIC cards above this solution and putting the extra NIC cards on a private network using a Hub or switch.
I then setup the heartbeat program to do ucast which will send out udp pings to each node over the parallel link. I configured the parallel port network referring to this site: http://www.tldp.org/HOWTO/Net-HOWTO/x1650.html It was for kernel 2.0, but I was able to adapt it to my configuration. Here is what I had to setup to get the parallel port network working in kernel 2.6 and gentoo:
Added this to /etc/modules.d/i386:
options parport_pc io=0x378 irq=7
Make sure you put in the correct io address and irq found in your bios. Then issue modules_update command to regenerate the /etc/modules.conf file.
I then added these lines to /etc/conf.d/local.start:
/sbin/ifconfig plip0 localplip pointopoint remoteplip \ netmask 255.255.255.0 /sbin/route add remoteplip plip0
And these lines to /etc/conf.d/local.stop:
/sbin/route del remoteplip plip0 /sbin/ifconfig plip0 down
I'm sure there are other ways to do this, but this is how I did it. I now have a working heartbeat private network. So the computers can talk to each other and when one goes down the other one takes over LVS duties.
Here is the only line I have in the haresources file. Remember this file has to be the same on all HA nodes.
lvs1 10.140.20.9 ldirectord
This lets us know that lvs1 is the primary HA node and will answer to this virtual IP address and will start ldirectord when taking over HA duties.
For the authkeys file I just used the simple crc flag because I was using parallel port networking. If you are using a public connection for the heartbeat you will want to use a better authentication. Here is the contents of my authkeys file on both nodes:
auth 1 1 crc
Remember that the authkeys file must be 600 permission:
chmod 600 /etc/ha.d/authkeys
Here is the ldirectord.cf file:
logfile="local0" virtual=10.140.20.9:3389 real=10.140.20.11 gate real=10.140.20.13 gate real=10.140.20.15 gate real=10.140.20.17 gate real=10.140.20.19 gate real=10.140.20.21 gate scheduler=wlc
We will log to standard syslog output, set our virtual ip to the same as the one in haresources with the port we want (terminal services), setup our real servers with direct routing (gate) and turn on the weighted least-connections scheduler (wlc). All of these files should be under /etc/ha.d.
Setting up the Windows 2000 Terminal Servers
We will need to make sure that we have the Windows 200 Terminal Servers setup properly so that we don't encounter any arp issues with the virtual IP address. We need to setup the Windows machines like so:
- Add the Microsoft Loopback Adapter by going to the Control Panel and Add/Remove Hardware. Then add a new networking device. Go under Microsoft and select the Loopback Adapter device.
- Go into the new network configuration by going into the Control Panel and then Network and Dialup Connections.
- Go to the properties of the new Local Area Connection
- Configure it with your virtual IP address (ie 10.140.20.9)
- Un-check Client for Microsoft Networks
- Un-check File and Printer Sharing for Microsoft Networks.
- Go to Advanced and set interface metric to 254
This completes the setup. You should now have two or more computers that act as an HA cluster to serve a LVS cluster of Windows 2000 Terminal Servers to load balance the connections from your clients.