virtuozzo

In this article, I wish to introduce the various commands that can be used to troubleshoot, load or performance issues, within a Virtuozzo node

To begin with, we will discuss the various situations that can degrade performance of a Virtuozzo node. As is obvious, heavy usage of either/all of CPU, Memory, Disk or Network will degrade the performance of node. The same case applies with Virtuozzo nodes too, but with an additional complexity; it could also happen as a result of the processes running inside a container. In this case we need to identify the problem container and deal with the processes inside that container.

I have found these commands are quite useful for finding out the problem container while troubleshooting Virtuozzo alerts.

 

Troubleshooting load issues caused by high CPU activity

Display the list of containers sorted based on cpu usage,

#/usr/sbin/vzlist   -o  ctid,laverage

OR

#/usr/sbin/vzstat   -t  -s  cpu|awk  ‘NF==10{print$0}’

 

Sometimes the above one-liners will not show the actual cpu usage inside the containers(possibly due to some delay in updating the stats), but still, the load inside node will be high. In this situation, running the command pasted below will help find out the cpu intensive containers.

#for i in `/usr/sbin/vzlist -H -o ctid`; do echo “CTID: ${i} `/usr/sbin/vzctl exec ${i} cat /proc/loadavg`”; done

 

 

List out all containers for which the status is not in “OK” status. This is quite helpful while troubleshooting load issues when the load average in the node is super-high(above 1000)

#/usr/sbin/vzstat -t|awk ‘{if(NF==10 && $2!=”OK” && $1!=”CTID”)print $0}’

 

Lists the top 10 containers based on number of processes running inside the container.

#/usr/sbin/vzlist -H -o ctid,numproc|sort -r -n -k2|head

cpuvz

Troubleshooting load issues caused by n/w activity

Sorts containers based on socket usage

#/usr/sbin/vzstat -t -s sock|awk ‘NF==10{print $0}’

 

Sorts containers based on TCP sender buffer usage,

#/usr/sbin/vzlist -H -o ctid,tcpsndbuf |sort -r -n -k2

 

 

Sorts containers based on TCP receive buffer usage,

#/usr/sbin/vzlist -H -o ctid,tcprcvbuf |sort -r -n -k2

 

 

Sorts containers based on the highest inbound traffic(quite useful while troubleshooting n/w related attacks),

#/usr/sbin/vznetstat -r |awk ‘$3 ~ /G/ {print $0}’|sort -r -nk3

 

 

 

Sorts containers based on the highest oubound traffic(quite useful while troubleshooting n/w related attacks) ,

#/usr/sbin/vznetstat -r |awk ‘$5 ~ /G/ {print $0}’|sort -r -nk5

Troubleshooting performance issues caused by memory utilization

 

The ‘dmesg‘ command displays containers which has resource shortages. There is possibility that it could be the abusive one. So there is a need to check the process inside that container.
#dmesg|egrep -v ‘(SMTP-LOG|INPUT-DROP|LIMIT-PPS-DROP|FORWARD-DROP)’

 
TTL=64 ID=0 PROTO=UDP SPT=68 DPT=67 LEN=556
[1101732.300833] __ratelimit: 44  callbacks suppressed
[1101732.310531] Fatal resource shortage: kmemsize, UB 12215.
[1101742.294179] Fatal resource shortage: kmemsize, UB 12215.
[1101752.277368] Fatal resource shortage: kmemsize, UB 12215.
[1101752.393226] Fatal resource shortage: kmemsize, UB 12215.
[1105092.458621] __ratelimit: 101 callbacks suppressed
[1105092.468411] Fatal resource shortage: kmemsize, UB 12215.

Troubleshooting load issues caused by high disk I/O activity.
You can install the ‘atop’ command and spot problem processes at the top of the list when sorting by disk usage (‘D’). To get more information on using ‘atop’ refer url.

The above one-liners will help you identify the problem CTID or the process(PID) responsible for performance issue. In the second case after finding the process-id, you can use ‘vzpid’ command to spot the container inside which the process is running and either renice or stop that process. And in the first case, you can view the processes running inside the container using either ‘vzps’ or ‘vztop’ command. The usage of which is given below,

#vztop -b -c -n 1 -E <CTID>

 

OR

 

#vzps auxfww -E <CTID>

 

I have written a small python script that will automate the ssh login and execution of the one-liners mentioned above. You can view this script at our script reposiory under the following URL:

http://scriptmantra.info/2013/03/python-script-for-virtuozzo-one-liners/