Dual internet load balancing

From LQWiki
Jump to navigation Jump to search

Introduction

***This tutorial is created for gentoo, If someone could come through and modify it for generic linux distributions I would welcome it***

Sometimes you may have access to multiple internet connections on a single machine. It is very handy to be able to use both internet connections for various reasons:

  • Dual internet = more bandwidth* most tasks will only use a single connection, such as a file download, the speed is seen when downloading several files and having half download on one connection and the other half using the other connection. Similar to multi-cpu systems seeing speed increases typically when multitasking.
  • Redundancy, if one connection goes down you have another.
  • Bragging rights
  • in my case static IP's are only available through dsl, but dsl is slower than the cable internet option in my area. So I have both, the dsl provides static ip and the cable provides the speed.

Packages

As far as I can tell the only package you need for this is iproute2.

Kernel Options

You do need to set the following kernel options to make use of dual connections. These options are for the 2.6.x kernel.

CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_IP_ROUTE_MULTIPATH=y

Configuration

** My configuration makes use of a new init.d startup script seperate from the gentoo networking scripts. I would welcome it if anyone would modify this for standard gentoo systems.

Step 1: Make sure all the interfaces are correctly configured for their own networks

For each connection you need to first make sure the connection is there, this can be achieved through the gentoo standard network scripts (just follow the gentoo guide for configuring your network on each interface). Another option is to use the ifconfig utility to create each interface:

/sbin/ifconfig ethX xxx.xxx.xxx.xxx netmask xxx.xxx.xxx.xxx'

Step 2: Create the dualnet rc script

Use your favorite text editor (nano -w) to open/create /etc/init.d/dualnet

Initial empty script

First lets create a general empty script:

#!/sbin/runscript

start() {
ebegin "Configuring Dualnet"
#Interface 1

#Interface 2

#Other interfaces .....

#Final Routing instructions

}

restart() {
svc_stop
svc_start
}

Device sections

Basic device section

Now we need to flesh it out, each interface will have it's own section that will follow this template:

/sbin/ip route add <NETWORK> dev <DEVICE> src <IPADDRESS> table <#>
/sbin/ip route add default via <GATEWAY> table <#>
/sbin/ip rule add from <IPADDRESS> table <#>
  • Definitions:
Line1: Initial configuration for the table
Line2: Specify the gateway this table should use
Line3: Make sure this interface is used to respond to any connection made to it, this ensures that if someone connects tot hat interface, the responce will go out using the same interface. We do not want a connection to come in on eth0 and have the responce sent out on eth1.
<DEVICE> is the device used for this connection, ex: eth0
<IPADDRESS> is the ip address of this specific interface, ex: 192.168.0.2
<#> is the table number, each interface will have it's own table, so for eth0 you might use 1 for <#>
<GATEWAY> is the ip address of the gateway this interface uses
<NETWORK> is the network the interface is conencted to, example: 192.168.0.0/24

You need one section per interface.

Help finding some information
  • Calculating /xx for netmask

Each section of the ip address is 8 bits, netmask speciofies how many bits are used to identify the network, and how many are used to identify the specific machine. for instance a netmask of 255.255.255.0 has the first 3 sections of an ip address used for the network, and the last section for the machine. For netmasks that have only 255's and 0's as their sections it is easy to calculate. Each section is 8 bits, /xx is the total bits used for thr network portion, so simply add the 255's and multiple by 8:

255.0.0.0        255's: 1   1 * 8 = 8  /8
255.255.0.0      255's: 2   2 * 8 = 16  /16
255.255.255.0    255's: 3   3 * 8 = 24  /24
255.255.255.255  255's: 4   4 * 8 = 32  /32 *This will never happen.

Of cource there is something called subnetting. This is said to let you borrow bits from one side or another. I have no idea how to calculate when you get a situation like this: Netmask = 255.255.255.248. You can spend your time online and either find it already calculated for you, or you can spend a lot of time reading about subnetting until you know all there is to know about it leaving you with a lot of information you do not need. Or alternatively you can use the following trick assuming you have the connection up and running.

If you, like me are lazy, or have no idea how to calculat the /xx for strange netmasks there is an easy way, simply type 'ip route' as root, it will show you the proper /xx value for each interface assuming you already have it up and running:

# ip route
192.168.0.0/24 dev eth1  proto kernel  scope link  src 192.168.0.2 
127.0.0.0/8 via 127.0.0.1 dev lo  scope link

My guess on subnets:
I have an onterface with 255.255.255.248 it gives me /29. I know that 255 is the largest number you can store in 8 bits, 255-248 = 7 I know that you need 3 bits to store a 7 (binary code/math here) 32 is the total bits in an ip address, 32 - 3 = 29 same number as the /xx, so if all else fails you can try to calculate it this way, but if you do not know anything about binary code for storing numbers this does not help you. * Further thought also reveals to me that this 7 is the number of static ip's my connection provides: 2 for the router and isp connection, and 5 for the systems on my network. Of cource there may very well be no connection here, this is just observation on my part.

Example: 2 interfaces
#Interface 1:
/sbin/ip route add 192.168.0.0/24 dev eth0 src 192.168.0.2 table 1
/sbin/ip route add default via 192.168.0.1 table 1
/sbin/ip rule add from 192.168.0.2 table 1

#Interface 2:
/sbin/ip route add 192.168.1.0/24 dev eth1 src 192.168.1.2 table 2
/sbin/ip route add default via 192.168.1.1 table 2
/sbin/ip rule add from 192.168.1.2 table 2

Final Routing Instructions

Single Outbound Route

Normally a system will have a single default gateway. This gateway is used for any connection where no pre-defined routing path can be found. If you simply want each interface to respond to it's own connections, but only have one connection used for outbound then you simply need specify he default gateway:

# route add default gw <GATEWAY> <DEVICE>

However If you want to utilise the bandwidth of more than one connection you specify the route differently.

Load Balancing

First you want to make sure there are no existing default routes. This means go into your startup scripts and make sure you never specify a gateway for the interfaces you have configured. Alternatively you can manually remove them.

To manually remove default gateways, you need to find them first:

# route
192.168.0.0     *               255.255.255.0   U     0      0        0 eth1
loopback        Abydos.open-exo 255.0.0.0       UG    0      0        0 lo
default         xxx.xxx.xxx.xxx   0.0.0.0       UG    0      0        0 eth0
default         xxx.xxx.xxx.xxx   0.0.0.0       UG    0      0        0 eth1 

To remove a gateway:

# route del default <DEVICE>

Make sure they are all removed. *NOTE: This will not be perminent, if your default gateway is specified in your startup scripts it will be restored on reboot. If you do nto know how to remove the default gateway from your startup script you can force the issue, but it is dirty. To force the issue edit your dualnet rc script we wrote above, above each section insert this line:

/sbin/route del default <DEVICE> > /dev/null

This will work, but it is dirty, if the default route is there it will be deleted, if not the command will generate an error. ' > /dev/null' might prevent you from seeing the error (though it still occurs) but I have not tested this.

Now to specify our default route.

/sbin/ip route add default scope global\
nexthop via <GATEWAY> dev <DEVICE> weight <W>\
nexthop via <GATEWAY> dev <DEVICE> weight <W>

Defenitions:

All previosuly defined terms are still used.
<W> Weight, Numerical value 1 or greater.  A higher weight will result in that connection being used mroe often. if both your connections are the same speed use 1 for each, if one is twice as fast as the other use 2 for it's weight.

Each device will have it's own line:

nexthop via <GATEWAY> dev <DEVICE> weight <W>

Each line except the last should have the '\' escape character to tell the script that the next line is part of the same command.

*NOTE: Make sure you use the correct device/gateway pairings.

The last piece of information the system needs is the nameserver to use when resolving net addresses. the nameserver is stored in the /etc/resolve.conf and is an entry like this:

nameserver 192.168.0.1

To specify the nameserver simply run the following:

echo "nameserver <GATEWAY>" >> /etc/resolv.conf # Will add the line

If you do not have anything that creates this line in the file that is all you need. However if you already have the nameserver line in there for some reason on reboot, I recommend you find out why and prevent it from happening, but if you can't or are just lazy you can try the following, but be warned it wipes out the file completely, make sure you do not have other important lines in there after a reboot.

echo "nameserver <GATEWAY>" > /etc/resolv.conf # Will wipe everything out then add the line

All you do is remove one of the '>' symbols changing it from append to create.

You must have a nameserver specified in order to resolve internet addresses. Most gateways have a dns server of some kind, so you can probably use the IP address of any of your internet gateways. Optionally you can specify all of them:

echo "nameserver <GATEWAY1>" >> /etc/resolv.conf
echo "nameserver <GATEWAY2>" >> /etc/resolv.conf
echo "nameserver <GATEWAY3>" >> /etc/resolv.conf

Example of final script

Here is my script as an example, it uses dsl, and cablemodem connections, I have stripped the ip's for my own security:

#!/sbin/runscript

start() {
	ebegin "Configuring Network"

	/sbin/ifconfig eth0 <IP REMOVED> netmask 255.255.255.248 #I do not use the gentoo net scripts, so I start the interface here
	/sbin/ip route add <NET REMOVED>/29 dev eth0 src <IP REMOVED> table 1
	/sbin/ip route add default via <GW IP REMOVED> table 1
	/sbin/ip rule add from <IP REMOVED> table 1

	/sbin/ifconfig eth1 192.168.0.2 netmask 255.255.255.0 #I do not use the gentoo net scripts, so I start the interface here
	/sbin/ip route add 192.168.0.0/24 dev eth1 src 192.168.0.2 table 2
	/sbin/ip route add default via 192.168.0.1 table 2
	/sbin/ip rule add from 192.168.0.2 table 2

	/sbin/ip route add default scope global nexthop via <REMOVED> dev eth0 weight 1\
						nexthop via 192.168.0.1 dev eth1 weight 3 #My cablemodem is 3 times faster than my dsl

	echo "nameserver 192.168.0.1" > /etc/resolv.conf #Wipe out the file and give it the first nameserver
	echo "nameserver <REMOVED>" >> /etc/resolv.conf #Add the second nameserver afterwords

	eend $?

}

restart() {
svc_stop
svc_start
}

Note on DHCP and dynamic ip's

The Note

If any of your connections are dynamically assigned via dhcp you have a problem. Basiaclly everything I layed out is for static ip. One solution is to have your dynamically assigned interfaces initilize before the script, then grab the info from the system and use it in the script. I do not have this issue and have not worked out all the fine details of doing this. Basically you need to follow these steps though:

  • Establish the dynamic connections
  • grab the information (IP, Network, and GW)
  • remove the default route created by the connection
  • run the script using the information that has been grabbed.

Helpful script

I am a nice guy, while I have not gone through and written the script for dynamic addresses I have created a utility to help anyone that wants to try.

The perl script below allows you to grab the ip and gateway of an established interface.

#!/usr/bin/perl

use strict;

my $IP;
my $GW;
my $IF = $ARGV[0];
my $Data;

exit(1) if ((!($ARGV[0])) || (!($ARGV[1])));

if ($ARGV[1] eq 'ip')
{
        $Data = `/sbin/ifconfig $IF`;
        $Data =~ m/inet addr:[0-9\.]+/i;
        my $IP = $&;
        $IP =~ s/inet addr://ig;
        print("$IP");
} elsif ($ARGV[1] eq 'gw')
{
        $Data = `/sbin/route`;
        $Data =~ m/default.*$IF/ig;
        $GW = $&;
        $GW =~ s/default\s*//ig;
        $GW =~ s/\s+.*//ig;
        print("$GW");
}

Simply save the script to a file such as info.pl, and give it the proper permissions using chmod.

Syntax:

./Script.pl <DEVICE> <Info to get>

Example: Getting IP of eth0

./Script.pl eth0 ip

Example: Getting Gateway of eth0

./Script.pl eth0 GW

To use this in a script:

$IP = `/path/to/script.pl eth0 ip`

the ip address of eth0 will now be inside the $IP variable, you can then use that wherever you need it. The script does not return a newline, so the variable should be uncluttered.

Hope that helps.

Sources/Links

http://lartc.org/howto/lartc.rpdb.multiple-links.html - 4.2 Routing for multiple uplinks/providers section of Linux Advanced Routing & Traffic Control HOWTO