Velocity Software, Inc. is recognized as a leader in the performance measurement of z/VM and Linux on z. The Velocity Performance Suite consist of a set of tools that enable installations running z/VM to manage Linux and z/VM performance. In addition, many components of server farms can be measured and analyzed. Performance data can be viewed real-time through the use of either 3270 or a browser. The CLOUD Implementation (zPRO) component is designed for full cloud PaaS implementation as well as to extend the capabilities of the z/VM sysprog (system programmer) to the browser world. This feature moves system management to the point-and-click crowd. Archived data and reports can be kept available of long term review and reporting usine zMAP. The zVPS, formally ESALPS, components consist of: zMON (formally ESAMON - real-time display of performance data), zTCP (formally ESATCP - SNMP data collection), zMAP (formally ESAMAP - historical reporting and archiving), zVWS (formally ESAWEB - z/VM based web server), zTUNE (a subscription service), zVIEW (formally SHOWCASE - web based viewing of performance data), zPRO (new to the quality line of Velocity Software Products). Velocity continues to work with other software vendors to ensure smooth interface with or from other products such as VM:Webgateway, CA-Webgateway, EnterpriseWeb, MXG, MICS. Velocity software remains the leader and inovator in the z/VM performance, Linux performance, Managing cloud computing arenas.
About Us | Products | FAQ | Demo | Customer Area | Education | Linux Hints & Tips | Presentations | News | Industry and Events | Employment Opportunities
Home | Contact Us | License Info | Newsletter    

Linux Instrumentation issues under z/VM

The issues are simple, your Linux performance monitor reports bad CPU numbers, requires too much overhead to run, and will become your performance problem. And until you go production, it won't matter....

Linux CPU Reporting under VM is WRONG!

When running in a virtual environment such as z/VM, ALL linux CPU accounting is wrong. This presentation has been given many times at SHARE and IBM technical conferences, but yet people want to depend on products that are just not aware of this problem to provide their CPU measurements. These tools just do NOT report CPU correctly - and has been proven to be incorrect by as much as an order of magnitude. TEST your solution before buying.

ALL agents and performance programs in Linux get their data from the same /proc file system. It is easy to show that this data is wrong by design. Linux measures CPU using a time of day sampling technique that in a dedicated environment works sufficiently to meet the requirements.

When running in a virtual environment, Linux allocates the processor as if it was dedicated. In an idle system, where there is only one linux server running under z/VM, the difference in reporting is not usually noticed. But under load when there is two or more servers active, Linux does not know if there are delays getting cpu or not. This results in larger reporting of CPU utilization than what was actually used.

To test this, run "top" or your agent of choice in one server, then logon multiple servers that are in loops. The more servers on, the more that "top" reports itself using - but it's requirements are unchanged. Or set a maximum share for the linux server to simulate what happens when resources are limited.

Imagine one of your servers goes into a loop. Any server that you logon to and run "top" will show very high CPU utilization. How can you correctly determine the real problem? Or use this data for capacity planning? The VM data is required.

Velocity Software corrects the Linux CPU numbers using a unique data capture method that absolutely requires z/VM performance data for the same interval as the Linux data. This method is unique to ESALPS. No other vendor or product can collect the data concurrently and correct the data. Ask them and ask for proof....

For more information, see the PRORATE PRESENTATION as presented at many conferences in 2001 and 2002.

Does turning off your performance monitor solve your performance problem?

How much CPU should your Idle servers require for instrumentation? The worst performance monitor is the one that becomes the performance problem. Most of the agents that work today in Linux, Unix or NT environments are not efficient, but in most distributed environments, it does not matter. With all the cycles available on those platforms, if an idle server is measuring itself, there is no problem.

In the shared resource environment with Linux running under z/VM, there are two issues to address.
The first issue is the cost of the instrumentation, usually in the form of an agent. If this cost is 5% of a processor, and you expect to run 100 servers, you have just allocated 5 processors to instrumentation. Not Good. NETSNMP is a VERY low cost agent and readily available from sourceforge.
The second issue is the cost of measuring idle servers. Why wake up a server to see what it is doing when it is not doing anything? Waking up an idle server involves CPU, storage and paging, all of which are unnecessary, and all of which take resource away from the other servers.

Velocity Software's ESALPS is the Low Cost Performance Monitor! With ESALPS, the VM data tells us that the server is idle, thus there is no need to request the performance data. As ESALPS uses NETSNMP, the agent is passive and only wakes up when data is requested - unlike other available agents in this environment.

With many servers running, do you see the "system view", or just one linux server? or just z/VM?

The z/VM platform is a shared resource platform that may be configured to support 100's of servers. In performing capacity planning or performance tuning, knowing where the resources are being used is critical. Agents that allow a perspective into one or two linux servers do not provide a system perspective.

ESALPS gathers all z/VM data, Linux data, network data, and data from other distributed servers into one database. From this exclusive integrated database, a full system perspective can be obtained.

Do you monitor idle servers?

Monitoring idle servers on z/VM wastes expensive CPU and storage resources that could be used for performing real work by other servers. ESALPS gives you the ability using the z/VM data to detect an idle server or a server with very low utilization and can then discontinue monitoring until the server becomes active. This can save significant resource and when running hundreds of servers still allows full monitoring of all of them.

Data Example

The following published in the Domino 6.5 Redbook is one example of the kinds of reporting possible with ESALPS. This example was to show that for this workload, java was not the CPU problem, when adding up all the processes with the same name, the "server" process with 67 threads became the obvious place to start tuning. But with an average thread requirement of only .9%, it was not obvious until this report was analyzed.

Report: ESAHSTA   LINUX HOST Application Report    Domino Redbook  ESAMAP
Node/    Process/    <-Application Process Counts------> <-----Processor---->
Date     Application                                     <---Utilization---->
Time     name        Total active Running ResWait Loaded Percent seconds Avg
-------- ----------- ----- ------ ------- ------- ------ ------- ------- ----
LINUXA   java         15.0   15.0     2.0    13.0      0    10.3    92.6  0.7
         kswapd        1.0    1.0       0     1.0      0     9.1    82.2  9.1
         router       11.0   11.0       0    11.0      0    10.6    95.4  1.0
         server       67.0   67.0     1.0    63.0    3.0    63.2   568.5  0.9
         update        3.0    3.0     1.0     2.0      0    10.2    91.7  3.4

Don't miss Velocity Software's Performance Seminars