1 / 26

Virtuozzo Linux Advanced Q&A: deployment and optimization

Virtuozzo Linux Advanced Q&A: deployment and optimization. Andrey Moruga Product Manager May 9, 2007. Why Should I Care?. Want to get most out of your servers? Better density per physical server More computing power to the applications, not to the internal system tasks

torin
Download Presentation

Virtuozzo Linux Advanced Q&A: deployment and optimization

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Virtuozzo Linux Advanced Q&A: deployment and optimization Andrey Moruga Product Manager May 9, 2007

  2. Why Should I Care? • Want to get most out of your servers? • Better density per physical server • More computing power to the applications, not to the internal system tasks • Better performance and response time • Better uptime • Review and optimize • Hardware configuration • Software tuning • VE parameters • And get the result • More • Competitive pricing • Faster ROI customers per physical servers happy Virtuozzo Linux Advanced Q&A: deployment and optimization

  3. But my Servers Perform Already Well! They can always do better! Maybe they do But… 3 Virtuozzo Linux Advanced Q&A: deployment and optimization

  4. Sometimes the Problems are Obvious # time lsbin dev home lib media mnt proc sbin tmp var boot etc initrd lost+found opt root usr vzreal 0m1.205suser 0m0.001ssys 0m0.004s • Simple web pages are loaded several seconds • The file system is very slow • Swap space always in use for more than 50-80% # free -m total used free shared buffers cached Mem: 752 740 12 0 145 230 Swap: 1537 1470 67 • “uptime” showing load average tens (30 and more) # uptime 15:45:47 up 29 days, 2:04, 1 user, load average: 71.40, 68.27, 70.79 4 Virtuozzo Linux Advanced Q&A: deployment and optimization

  5. Lack of Resources: Symptoms CPU shortage CPU(s) have almost 0 idle cycles – always are busy Applications are slow, bad response time in interactive mode A lot of processes are in “R” – running state High load average (reported by “uptime” command): 30 and more Disk IO shortage Disk access it very slow. Simple command like “ls” takes forever Applications are slow, bad response time in interactive mode A lot of processes are in “D” – blocked (“D”isk) state High load average (reported by “uptime” command): 30 and more Memory Swap space is heavy used; intensive swap-in/out activity OOM (Out-Of-Memory) kills happens often 5 Virtuozzo Linux Advanced Q&A: deployment and optimization

  6. CPU Shortage: Problem Confirmation • Check CPU idle and number of the processes running # vmstat 2 procs ----------memory--------- --swap-- ----io--- -system- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 38 0 0 218444 148948 235916 0 0 0 5 15 16 0 0 100 0 49 0 0 218444 148948 235916 0 0 0 0 1006 23 100 0 0 0 37 1 0 218444 148948 235916 0 0 0 20 1008 28 100 1 0 0 34 0 0 218444 148948 235916 0 0 0 0 1004 26 100 1 0 0 35 0 0 218444 148948 235916 0 0 0 0 1005 23 100 0 0 0 38 0 0 218444 148948 235916 0 0 0 0 1005 25 100 0 0 0 33 0 0 218444 148948 235916 0 0 0 0 1004 23 100 0 0 0 32 0 0 218444 148948 235916 0 0 0 0 1006 26 100 0 0 0 6 Virtuozzo Linux Advanced Q&A: deployment and optimization

  7. CPU Shortage: Problem Resolution Find the processes that are always running, and prevent them from running again ps ax| grep “ R ” or top to find running processes vzpid {PID} to locate a VE where they are running vzstat to see CPU usage on per-VE basis Check for possible misconfiguration Compare CPUUNITS assigned to different servers using vzcpucheck -v.Avoid a significant imbalance of CPUUNITS (by orders of magnitude) between VEs on the same server Check CPULIMITS applied to VEs. Avoid setting CPULIMUT to less than 50% of single CPU Examine /proc/user_beancounters file. Failure in the resources allocation may provoke CPU consuming Buy more powerful server Just kidding  7 Virtuozzo Linux Advanced Q&A: deployment and optimization

  8. Disk IO Shortage: Problem Confirmation • Check the number of the processes blocked (in “D” state), the disk IO block counts and CPU wait/idle time # vmstat 2 procs ----------memory--------- --swap-- ----io--- -system- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 23 592 19040 2932 601296 0 0 4730 11530 1138 100 2 15 0 84 0 22 592 2768 2876 636860 0 0 5672 17506 1144 119 2 29 0 70 0 21 592 1080 2836 640420 0 0 3820 10368 1135 95 1 10 0 89 0 20 592 2856 2820 634004 0 0 3242 17344 1158 118 3 31 0 67 0 20 592 2756 2812 634092 0 0 5612 10866 1134 109 2 19 0 80 0 20 592 2796 2776 633836 0 0 2112 17392 1176 126 1 15 0 83 0 21 592 2692 2756 639380 0 0 4200 18534 1157 124 3 28 0 69 0 22 592 928 2796 645092 0 0 1110 11908 1143 111 3 18 0 80 • Check if extra disk activity increases the load average dramatically • E.g. backup process increases load average from 30 to 60 8 Virtuozzo Linux Advanced Q&A: deployment and optimization

  9. Disk IO Shortage: Problem Resolution • Make sure the “noatime” option for /vz partition set # cat /etc/fstab|grep /vz LABEL=/vz /vz ext3 noatime 1 2 • Make sure slocate doesn’t provoke extra load • In the /etc/updatedb.conf file, the PRUNEFS variable should include “vzfs”, PRUNEPATHS variable should include “/vz” • Check that VEs are not running updatedb by default • In case you run 2.6.8 kernel, CFQ disk scheduler may provide better performance • Add “elevator=cfq” line to your kernel boot parameters and restart the server • Review the /vz partition block size • Though 4096 bytes block size may waste some additional disk space, it gains slightly better performance than 1024 bytes 9 Virtuozzo Linux Advanced Q&A: deployment and optimization

  10. Disk IO Shortage: Problem Resolution (Contd.) • Check the file system journal size # debugfs -c /dev/sda3 debugfs 1.35 (28-Feb-2004) /dev/sda3: catastrophic mode - not reading inode or group bitmaps debugfs: stat <8> Inode: 8 Type: regular Mode: 0600 Flags: 0x0 Generation: 0 User: 0 Group: 0 Size: 8388608 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16452 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4421c312 -- Wed Mar 22 16:35:14 2006 atime: 0x00000000 -- Wed Dec 31 19:00:00 1969 mtime: 0x4421c312 -- Wed Mar 22 16:35:14 2006 BLOCKS: (0-11):1426-1437, (IND):1438, (12-267):1439-1694, (DIND):1695, (IND):1696, (268-523):1697-1952, (IND):1953, (524-779):1954-2209, (IND):2210, (780-1035):2211-246 [...output skipped...] 10 Virtuozzo Linux Advanced Q&A: deployment and optimization

  11. Disk IO Shortage: Problem Resolution (Contd.) needs_recovery • If needed, increase the file system journal size to the recommended size 64-128MB # tune2fs -l /dev/sda3|egrep "Filesystem state|Filesystem features" Filesystem features: has_journal ext_attr resize_inode filetype sparse_super Filesystem state: clean # tune2fs -O ^has_journal /dev/hda3 tune2fs 1.35 (28-Feb-2004) # tune2fs -J size=64 /dev/hda3 tune2fs 1.35 (28-Feb-2004) Creating journal inode: done # debugfs -c /dev/hda3 debugfs 1.35 (28-Feb-2004) /dev/hda3: catastrophic mode - not reading inode or group bitmaps debugfs: stat <8> Inode: 8 Type: regular Mode: 0600 Flags: 0x0 Generation: 0 User: 0 Group: 0 Size: 67108864 [...skip...] 11 Virtuozzo Linux Advanced Q&A: deployment and optimization

  12. Disk IO Shortage: Problem Resolution (Contd.) • Check that your VEs have an option to omit log files syncing # cat /etc/syslog.conf # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none -/var/log/messages # Log all the mail messages in one place. mail.* -/var/log/maillog • Review your hardware configuration • SCSI controllers usually provide the better performance due to their hardware ability to rearrange disk request in the most efficient order • Stripe volumes allow you to significantly increase disk IO bandwidth due to their ability to distribute single operation to multiple volumes simultaneously • RAID10 disk arrays may be used in order to have both striping and mirroring for high performance and data redundancy 12 Virtuozzo Linux Advanced Q&A: deployment and optimization

  13. Memory Shortage: Problem Confirmation • Check the memory, swap space usage and swap activity # vmstat 2 procs ----------memory--------- --swap-- ----io--- -system- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 6 110084 1112 508 10600 100 14422 316 14434 1200 249 1 39 0 60 4 3 52464 1184 1060 10952 108 512 1369 531 1144 229 5 95 0 0 0 21 77692 944 440 5172 60 13244 864 13244 1145 304 0 11 0 88 0 26 98264 968 520 6308 180 10286 778 10286 1181 271 0 5 0 95 0 25 110884 976 580 7560 480 6310 1132 6319 1133 278 0 5 0 96 0 20 90944 972 664 9008 384 13264 1155 13273 1151 318 1 31 0 68 • Search for OOM (Out-Of-Memory) kills • OOM kills are reported as kernel messages and may be found in /var/log/messages file, indicated by string like “Apr 26 15:45:31 testbox kernel: OOM killed process httpd (pid=19245, ve=10000) (mm=db1a9100) exited, free=5619 gen=3429.” 13 Virtuozzo Linux Advanced Q&A: deployment and optimization

  14. Memory Shortage: Problem Resolution Check your system configuration Hosting production servers with Virtuozzo usually should have at least 4GB of physical memory installed Make sure the swap size is not less than the memory size Make sure the proper kernel is used (run “uname -a” to see the kernel version used) Small UniProcessor systems should use “up” version of kernel SMP system with up to 4GB of memory should use “smpnosplit” kernel If the memory size exceed 4GB, “entnosplit” kernel should be used If the average number of processes on your system exceeds 2,000 – the “split” version of kernel (“smp” and “enterprise” accordingly) is recommended 64bit kernels have no memory limitations, so “smp” kernel fits systems with any amount of memory installed and number of processes running 14 Virtuozzo Linux Advanced Q&A: deployment and optimization

  15. Memory Shortage: Problem Resolution (Contd.) Tune your default applications configuration Switch off unused application features and modules Place rarely used network services under xinet.d based startup (vzsetxinetd tool allows you to switch a service startup mode) Fine tune applications parameters. Typically, applications tuned for a dedicated server are more “pre-allocate” oriented than their VE counterparts, so the default application configuration may not be optimal for the VE. For example, number of apache startup processes may be modified using StartServers, MinSpareServers and MaxSpareServers configuration variables Use benefits of Virtuozzo templates Template based applications may provide great memory sharing and saving. Avoid installing applications in your VEs directly from binary packages 15 Virtuozzo Linux Advanced Q&A: deployment and optimization

  16. Summary CPU shortage Symptoms Poor performance, high load average, high number of running processes Solution Kill abusing processes, review VE CPU and system resources settings Disk IO shortage Symptoms Poor performance, high load average, high number of blocked processes, minimal extra disk activity provokes load spikes Solution Review server hardware and software configuration, minimize heavy disk IO operations Memory shortage Symptoms High swap usage and swap out activity, often OOM kills Solution Add more RAM, review VE applications settings, use benefits of Virtuozzo templates 16 Virtuozzo Linux Advanced Q&A: deployment and optimization

  17. Q&A: Deployment and Troubleshooting-1 Q: What the recommended server partitioning is? A: Use the following general guidelines: Three partitions: / (root), /vz, swap /vz has highest performance requirements, / (root) has minimal ones Assign 4 to 10 GB of disk space for / (root), swap space is equal to the RAM size, the rest of the disk space given to /vz Q: Can I have multiple disk partitions mounted under /vz? A: Yes, though keep in mind that: Templates (/vz/template) should reside on a single disk partition Single VE private area (/vz/private/{VEID}) should reside on a single disk partition If you need to use multiple partition for a VE, mount it into the VE’s root (/vz/root/{VEID}/mnt) instead of the private area 17 Virtuozzo Linux Advanced Q&A: deployment and optimization

  18. Q&A: Deployment and Troubleshooting-2 Q: How can I leverage SAN on Virtuozzo? A: You can build automatic Virtuozzo failover cluster using Red Hat Cluster Suite software VE VE VE VE RH Cluster Node 1 Node 2 Node 3 Active Passive Active Active Hardware Crash!!! 18 Virtuozzo Linux Advanced Q&A: deployment and optimization

  19. Q&A: Deployment and Troubleshooting-3 Q: What is the easiest way to locate a VE by IP? A: Assign the VEID based on two last octets of the VE’s IP. For example : 83.241.134.71 → VEID 134071 Q: What is the major differences between UBC and SLM in the memory accounting? A: UBC accounts address space, while SLM accounts physical memory usage 256 KB used 1 MB allocated Address space • UBC accounts 1 MB • SLM accounts 256 KB 19 Virtuozzo Linux Advanced Q&A: deployment and optimization

  20. Q&A: Deployment and Troubleshooting-4 Q: How do I stop all the VEs real fast? A1: # for i in `vzlist -1`; do vzctl stop $i --fast; done A2: # killall -9 init Q: It looks very brutal… can it corrupt the VE file system? A: No, the file system will be cleanly dismounted by the host operating system. Though, applications data may be potentially corrupted because of unclean application shutdown. 20 Virtuozzo Linux Advanced Q&A: deployment and optimization

  21. Q&A: Deployment and Troubleshooting-5 Q: Do I need to apply updates to my EZ templates? A: Generally you do not – EZ template is just a set of configuration files. Included applications are updated automatically when you update your VEs Q: Do you recommend to maintain our own EZ templates repository, or we can use one on the Internet? A: Maintain your own. There is a number of reasons: Saving on network bandwidth and better availability Replicated repository inconsistency may be fixed locally Internet repositories may remove older versions of applications, that has negative impact on VEs using them (preventing a VE migration or restoring from a backup) 21 Virtuozzo Linux Advanced Q&A: deployment and optimization

  22. Q&A: Deployment and Troubleshooting-6 Q: What is the advantage of EZ templates over Standard templates? A: They are a way better. A VE update is just a single command instead of hours of manual work Q: Since centos.org removed CentOS 4.2 from their repository, I cannot restore CentOS EZ template based VE from a backup. What should I do? A: Add http://vault.centos.org/4.2/os/i386/ to the list of repositories on your Virtuozzo server. The most reliable way is still using your own (mirrored) repository Q: Can I apply Virtuozzo updates by a script? A: Yes. Use “vzup2date -m batch install” 22 Virtuozzo Linux Advanced Q&A: deployment and optimization

  23. Q&A: Deployment and Troubleshooting-7 Q: Can I assign an IP to a VE by DHCP? A: Yes, but only in switched networking mode. Routed networking does not support DHCP based IP assignment. Q: How do I mount /tmp with noexec, nosuid, nodev options in a VE? A: # vzctl set VEID --bindmount_add /tmp,nosuid,noexec,nodev --save Q: How can I manage a VE CPU power? A: There are 3 parameters controlling it: A relative CPU weight - CPUUNITS (mandatory) An upper CPU limit - CPULIMIT (optional) A number of CPUs the VE can use - CPUS (optional) 23 Virtuozzo Linux Advanced Q&A: deployment and optimization

  24. Q&A: Deployment and Troubleshooting-8 Q: How do I block particular TCP port for all the VEs on a server? A: Use the “FORWARD” chain on the host in your iptables rules:# iptables -I FORWARD -p tcp --dport 6667 -j DROP Q: I have a VE with huge static content. The content does not need to be backed up. How do I exclude that content from regular VE backup? A: Put the static content on external mount point via mount scripts. Note: directories mounted via “--bindmount” option of vzctl are included into the VE backup 24 Virtuozzo Linux Advanced Q&A: deployment and optimization

  25. Q&A: Deployment and Troubleshooting-9 Q: How do I create a partition greater than 2TB? fdisk tool doesn't seem to be able to do that A: Use GNU “parted” tool to partition your drive and create GPT labeled partition. Note: currently Linux cannot boot from such partitions. Q: I have NAS device which I can mount to the Virtuozzo box over NFS or CIFS. Can I place VE's private area on NAS share? A: No, currently it is not supported. Though, NFS or CIFS partition may be bind-mounted into a VE. 25 Virtuozzo Linux Advanced Q&A: deployment and optimization

  26. Q&A: Deployment and Troubleshooting-10 Q: When I create iptables rule with 'LOG' target inside a VE, packets are still not logged. How do I fix it? A: By default kernel logger is disabled in a VE. To enable it, modify the /etc/rc.d/init.d/syslog file in your VE. Find the lines which read passed klogd skipped #daemon klogd $KLOGD_OPTIONS passed klogd skipped #killproc klogd and change these lines to be daemon klogd $KLOGD_OPTIONS killproc klogd 26 Virtuozzo Linux Advanced Q&A: deployment and optimization

More Related