Virtual Server via Direct Routing

This page contains information about how to use the Direcet Routing request dispatching technique to contruct a Linux virtual server.

Direct Routing request dispatching technique

This request dispatching approach was first implemented in IBM's NetDispatcher. The virtual IP address is shared by real servers and the load balancer. All real servers have their loopback alias interface configured with the virtual IP address, and the load balancer has an interface configured with the virtual IP address too, which is used to accept the request packets. The load balancer and the real servers must have one of their interfaces physically linked by a HUB/Switch. The architecture of virtual server via direct routing is illustrated as follows:

When a user accesses the service provided by the server cluster, the packet destined for virtual IP address (the IP address for the virtual server) arrives. The load balancer(LinuxDirector) examines the packet's destination address and port. If they are matched for a virtual service, a real server is chosen from the cluster by a scheduling algorithm, and the connection is added into the hash table which records the established connection. Then, the load balancer directly forwards it to the chosen server. When the incoming packet belongs to this connection and the chosen server can be found in the hash table, the packet will be again directly routed to the server. When the server receives the forwarded packet, the server finds that the packet is for the address on the loopback alias interface and processes the request, finally return the result directly to the user. After the connection terminates or timeouts, the connection record will be removed from the hash table.

The direct routing workflow is illustrated in the following figure:

The load balancer siimply changes the MAC address of the data frame to that of the chosen server and restransmit it on the LAN. That's the reason that the load balancer and each server must be directly connected to one another by a single uninterrupted segment of a LAN.

How to build the kernel

1. The VS patch for kernel 2.0.36

First, get a fresh copy of the Linux kernel source of the right version. Second, apply the virtual server patch (version 0.9 or later) to the kernel. Third, make sure the following kernel compile options at least are selected.

Kernel Compile Options:

Code maturity level options --->

[*] Prompt for development and/or incomplete code/drivers

Networking options --->

[*] Network firewalls
....
[*] IP: forwarding/gatewaying
....
[*] IP: firewalling
....
[*] IP: masquerading
....
[*] IP: ippfvs(LinuxDirector) masquerading (EXPERIMENTAL)
Virtual server request dispatching technique---
( ) VS-NAT
( ) VS-Tunneling
(X) VS-DRouting

And, you have to choice one scheduling algorithm.

Virtual server scheduling algorithm
(X) WeightedRoundRobin
( ) LeastConnection
( ) WeightedLeastConnection

[ ] IP: enabling ippfvs with the local node feature

Fourth, rebuild the kernel. Once you have your kernel properly built, update your system kernel and reboot.

2. The VS patch for kernel 2.2.9

Kernel Compile Options:

Code maturity level options --->

[*] Prompt for development and/or incomplete code/drivers

Networking options --->

[*] Network firewalls
....
[*] IP: forwarding/gatewaying
....
[*] IP: firewalling
[*] IP: always defragment (required for masquerading) (NEW)
....
[*] IP: masquerading
....
[*] IP: masquerading virtual server support (EXPERIMENTAL)(NEW)
(12) IP masquerading table size (the Nth power of 2)(NEW)
<M> IPVS: round-robin scheduling(NEW)
<M> IPVS: weighted round-robin scheduling(NEW)
<M> IPVS: weighted least-connection scheduling(NEW)
<M> IPVS: persistent client connection scheduling(NEW)

My example for testing virtual server via direct routing

Here is my configure example for testing virtual server via direct routing. The configuration is as follows. I hope it can give you some clues.

1. For kernel 2.0.x

The load balancer (LinuxDirector), kernel 2.0.36

ifconfig eth0 172.26.20.111 netmask 255.255.255.0 broadcast 172.26.20.255 up
route add -net 172.26.20.0 netmask 255.255.255.0 dev eth0
ifconfig eth0:0 172.26.20.110 netmask 255.255.255.255 broadcast 172.26.20.110 up
route add -host 172.26.20.110 dev eth0:0
ippfvsadm -A -t 172.26.20.110:23 -R 172.26.20.112

The real server 1, kernel 2.0.36

ifconfig eth0 172.26.20.112 netmask 255.255.255.0 broadcast 172.26.20.255 up
route add -net 172.26.20.0 netmask 255.255.255.0 dev eth0
ifconfig lo:0 172.26.20.110 netmask 255.255.255.255 broadcast 172.26.20.110 up
route add -host 172.26.20.110 dev lo:0

When I am on other hosts, 'telnet 172.26.20.110' will actually connect the real server 1.

2. For kernel 2.2.x

The load balancer (LinuxDirector), kernel 2.2.9

ifconfig eth0 172.26.20.111 netmask 255.255.255.0 broadcast 172.26.20.255 up
route add -net 172.26.20.0 netmask 255.255.255.0 dev eth0
ifconfig eth0:0 172.26.20.110 netmask 255.255.255.255 broadcast 172.26.20.110 up
route add -host 172.26.20.110 dev eth0:0
ipvsadm -A -t 172.26.20.110:23 -s  wlc   (WLC scheduling is used for this virtual service)
ipvsadm -a -t 172.26.20.110:23 -R 172.26.20.112 -g

The other example of servers having different network routes

In the virtual server via direct routing, the servers can follows the different network routes to the clients. Here is a configuration example.

The load balancer (LinuxDirector), kernel 2.0.36

ifconfig eth0 <an IP address> ...
...
ifconfig eth0:0 <VIP> netmask 255.255.255.255 broadcast <VIP> up
route add -host <VIP> dev eth0:0
ifconfig eth1 192.168.0.1 netmask 255.255.255.0 broadcast 192.168.0.255 up
route add -net 192.168.0.0 netmask 255.255.255.0 dev eth1
ippfvsadm -A -t <VIP>:23 -R 192.168.0.2 (For kernel 2.2, you should alter this command)

The real server 1, kernel 2.0.36

ifconfig eth0 <a seperate IP address> ...
Follow the different network route
...
ifconfig eth1 192.168.0.2 netmask 255.255.255.0 broadcast 192.168.0.255 up
route add -net 192.168.0.0 netmask 255.255.255.0 dev eth1
ifconfig lo:0 <VIP> netmask 255.255.255.255 broadcast <VIP> up
route add -host <VIP> dev lo:0


Last updated on June 24, 1999

Created on May 1, 1999