lvs-direct-routing

 INTRODUCTION:

In Fedora, CentOS, and Rehat Enterprise Linux, IP Load Balancing solution is provided by using a package called ‘Piranha’.

 

Piranha offers the facility for load balancing inward IP network traffics (requests) and distribution of this IP traffic among a farm of server machines. The technique that is used to load balance IP network traffic is based on Linux Virtual Server tools.

 

This High Availability is purely software based provided by Piranha. Piranha also facilitates system administrator with a cool Graphical User Interface tool for management.

 

The Piranha monitoring tool is responsible for the following functions:

 

  • Heartbeating between active and backup load balancers.
  • Checking availability of the services on each of real servers.

 

 

Components of Piranha Cluster Software:

 

  • IPVS kernel, LVS (manage the IPVS routing table via the ipvsadm tool)
  • Nanny (monitor servers & services on real servers in a cluster)
  • Pulse (control the other daemons and handle failovers between IPVS routing boxes).

 

 

 

 INSTALLATION:

We will configure our computers or nodes as following:

 

Our load balancing will be done using 2 Linux Virtual Server Nodes or routing boxes.

We will install two or more Web servers for load balancing.

 

 

We will modify our hosts configuration file at /etc/hosts on each of the nodes in our setup

 

[root@arthar1 ~]# vim /etc/hosts

127.0.0.1 localhost.localdomain localhost

::1 localhost6.localdomain6 localhost6

 

##### Web Servers IPs #####

192.168.31.100             arthar1.sathish.com    arthar1

192.168.31.150             arthar2.sathish.com    arthar2

 

##### Load Balancing Nodes IPs #####

192.168.31.175                 arthar3.sathish.com      arthar3

192.168.31.155                 arthar4.sathish.com       arthar4

 

##########  Virtual IP/Service IP of Webserver ##########

192.168.31.180             server.sathish.com            server

 

1

 

Copy the /etc/hosts file to all the servers (This step is not required if you have DNS)

 

[root@arthar1 ~]# scp /etc/hosts arthar2:/etc

[root@arthar1 ~]# scp /etc/hosts arthar3:/etc

[root@arthar1 ~]# scp /etc/hosts arthar4:/etc

 

 

2

After copying to host file to all the nodes, we need to generate SSH keys.

 

[root@arthar1 ~]# ssh-keygen –t rsa

[root@arthar1 ~]# ssh-keygen –t dsa

 

3

[root@arthar1 ~]# cd /root/.ssh/

[root@arthar1 .ssh]# cat *.pub > authorized_keys

 

 

Now copy ssh keys to all other nodes for password less entry which is required by pulse daemon.

 

[root@arthar1 .ssh]# scp -r /root/.ssh/ arthar2:/root/

[root@arthar1 .ssh]# scp -r /root/.ssh/ arthar3:/root/

[root@arthar1 .ssh]# scp -r /root/.ssh/ arthar4:/root/

4

 

 

We can build up a global finger print list as following:

 

[root@arthar1 .ssh]# ssh-keyscan -t rsa arthar1 arthar2 arthar3 arthar4

[root@arthar1 .ssh]# ssh-keyscan -t dsa arthar1 arthar2 arthar3 arthar4

 

 

5

Now we will configure NTP service on all the nodes. We will make the ARTHAR3 as our NTP Server.

 

[root@arthar3~]# rpm -qa | grep ntp

ntp-4.3.3p1-9.el6.centos

chkfontpath-1.20.1-1.1

 

 

 

[root@arthar3 ]# vim /etc/ntp.conf

###Configuration for NTP server###

restrict 127.0.0.1

server 127.127.1.0 # local clock

fudge 127.127.1.0 stratum 10

 

 

 

[root@arthar3 ~]# service ntpd start

[root@arthar3 ~]# chkconfig ntpd on

 

Now we will configure client side configuration in arthar1.

 

[root@arthar1 ~]# vim /etc/ntp.conf

#restrict 127.0.0.1

#restrict -6 ::1

server 192.168.31.175

#server 0.centos.pool.ntp.org

#server 1.centos.pool.ntp.org

#server 2.centos.pool.ntp.org

#server 127.127.1.0 # local clock

#fudge 127.127.1.0 stratum 10

 

 

 

[root@arthar1 ~]# service ntpd start

[root@arthar1 ~]# chkconfig ntpd on

[root@arthar1 ~]# ntpdate -u 192.168.31.175

 

[root@arthar1 ~]# scp /etc/ntp.conf arthar2:/etc

[root@arthar1 ~]# scp /etc/ntp.conf arthar4:/etc

 

[root@arthar2 ~]# service ntpd start && chkconfig ntpd on

[root@arthar4 ~]# service ntpd start && chkconfig ntpd on

 

Copy the same configuration or the file /etc/ntp.conf to other 2 nodes arthar2, arthar4. After copying restart the ntp service on these nodes.

 

Now we will update the time on all the nodes by typing following command:

 

 

[root@arthar2 ~]# ntpdate -u 192.168.31.175

[root@arthar4 ~]# ntpdate -u 192.168.31.175

 

 

Now we will setup our Linux Virtual Server (ARTHAR3 & ARTHAR4) by installing Piranha package. We already know that Piranha includes ipvsadm, nanny and pulse demon.

 

 

We will use Yum to install Piranha on the both nodes.

 

[root@arthar3 ~]# yum install piranha -y

[root@arthar4 ~]# yum install piranha -y

 

Now we will configure Linux Virtual Server configuration file at /etc/sysconfig/ha/lvs.cf

 

 

[root@arthar3 ]# vim /etc/sysconfig/ha/lvs.cf

serial_no = 1

primary = 192.168.31.175

service = lvs

rsh_command = ssh

backup_active = 1

backup = 192.168.31.155

heartbeat = 1

heartbeat_port = 1050

keepalive = 2

deadtime = 10

network = direct

debug_level = NONE

monitor_links = 1

virtual server1 {

active = 1

address = 192.168.1.180 eth0:1

port = 80

send = “GET / HTTP/1.1\r\n\r\n”

expect = “HTTP”

load_monitor = uptime

scheduler = rr

protocol = tcp

timeout = 10

reentry = 180

quiesce_server = 0

server arthar1 {

address = 192.168.31.100

active = 1

weight = 1

}

server arthar2 {

address = 192.168.31.150

active = 1

weight = 1

}

}

6

Now we will copy this configuration file to arthar4.

 

[root@arthar3 ~]# scp /etc/sysconfig/ha/lvs.cf  arthar4:/etc/sysconfig/ha/

 

[root@arthar3 ~]# vim /etc/sysctl.conf

net.ipv4.ip_forward = 1

net.ipv4.conf.eth0.arp_ignore = 1

net.ipv4.conf.all.arp_announce = 2

net.ipv4.conf.eth0.arp_announce = 2

 

 

[root@arthar3 ~]# scp /etc/sysctl.conf arthar4:/etc/

 

Run this command on both nodes

7

[root@arthar3 ~]# sysctl -p

net.ipv4.ip_forward = 1

net.ipv4.conf.eth0.arp_ignore = 1

net.ipv4.conf.all.arp_announce = 2

net.ipv4.conf.eth0.arp_announce = 2

net.ipv4.conf.default.rp_filter = 1

net.ipv4.conf.default.accept_source_route = 0

kernel.sysrq = 0

kernel.core_uses_pid = 1

net.ipv4.tcp_syncookies = 1

kernel.msgmnb = 65536

kernel.msgmax = 65536

kernel.shmmax = 68719476736

kernel.shmall = 268435456

 

8

[root@arthar4~]# sysctl -p

net.ipv4.ip_forward = 1

net.ipv4.conf.eth0.arp_ignore = 1

net.ipv4.conf.all.arp_announce = 2

net.ipv4.conf.eth0.arp_announce = 2

net.ipv4.conf.default.rp_filter = 1

net.ipv4.conf.default.accept_source_route = 0

kernel.sysrq = 0

kernel.core_uses_pid = 1

net.ipv4.tcp_syncookies = 1

kernel.msgmnb = 65536

kernel.msgmax = 65536

kernel.shmmax = 68719476736

kernel.shmall = 268435456

 

We will start httpd on both web servers.

 

[root@arthar1]#/etc/init.d/httpd start && chkconfig httpd on

[root@arthar2 ]#/etc/init.d/httpd start && chkconfig httpd on

 

We will start pulse service on both lbs nodes:

 

[root@arthar3 ~]# service pulse start

[root@arthar3 ~]# chkconfig pulse on

[root@arthar3 ~]# tail -f /var/log/messages

10

Now we will install and configure our web servers and arptables_jf package for direct routing.

 

[root@arthar1 ~]# yum install httpd arptables_jf -y

[root@arthar1 ~]# echo “Web Server 1” > /var/www/html/index.html

 

Now we will configure the Ethernet interfaces for virtual IP on first web server node.

 

[root@arthar1 ~]# ifconfig eth0:1 192.168.31.180 netmask 255.255.255.0 broadcast 192.168.31.255 up

[root@arthar1 ]# echo “ifconfig eth0:1 192.168.31.180 netmask 255.255.255.0 broadcast 192.168.31.255 up” >> /etc/rc.local

 

Now we will do it on the second web server node.

 

[root@arthar2 ~]# yum install httpd arptables_jf -y

[root@arthar2 ~]# echo “Web Server 2” > /var/www/html/index.html

 

Now we will configure the Ethernet interfaces for virtual IP on second web server node.

 

[root@arthar2 ~]# ifconfig eth0:1 192.168.31.180 netmask 255.255.255.0 broadcast 192.168.31.255 up

[root@arthar2 ~]# echo “ifconfig eth0:1 192.168.31.180 netmask 255.255.255.0 broadcast 192.168.31.255 up” >> /etc/rc.local

 

Now we will configure our arptables on our first web server node

.

[root@arthar1 ~]# arptables -A IN -d 192.168.31.180 -j DROP

[root@arthar1 ~]# arptables -A OUT -d 192.168.31.180 -j mangle –mangle-ip-s 192.168.31.175

[root@arthar1 ~]# arptables -A OUT -d 192.168.31.180 -j mangle –mangle-ip-s 192.168.31.155

[root@arthar1 ~]# service arptables_jf save

[root@arthar1 ~]# chkconfig arptables_jf on

 

Now we will configure our arptables on our first web server node.

 

 

[root@arthar2 ~]# arptables -A IN -d 192.168.31.180 -j DROP

[root@arthar2 ~]# arptables -A OUT -d 192.168.31.180 -j mangle –mangle-ip-s 192.168.31.175

[root@arthar2 ~]# arptables -A OUT -d 192.168.31.180 -j mangle –mangle-ip-s 192.168.31.155

[root@arthar2 ~]# service arptables_jf save

[root@arthar2 ~]# chkconfig arptables_jf on

 

 

 

We have managed to setup our LVS and webserver nodes now its time to test if everything is working or not.

 

 

 

[root@arthar3 ]# ipvsadm -L

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

TCP server.sathish.com:http rr

-> arthar1.sathish.com:http Route 1 0 0

-> arthar2.sathish.com:http Route 1 0 0

11

 

Finally open a web browser from any machine and type http://server.sathish.com and keep on refreshing the page, we will get output of page contents from Webserver 1 and Web Server 2.

12 13