Velocity Software, Inc. is recognized as a leader in the performance measurement of z/VM and Linux on z. The Velocity Performance Suite consist of a set of tools that enable installations running z/VM to manage Linux and z/VM performance. In addition, many components of server farms can be measured and analyzed. Performance data can be viewed real-time through the use of either 3270 or a browser. The CLOUD Implementation (zPRO) component is designed for full cloud PaaS implementation as well as to extend the capabilities of the z/VM sysprog (system programmer) to the browser world. This feature moves system management to the point-and-click crowd. Archived data and reports can be kept available of long term review and reporting usine zMAP. The zVPS, formally ESALPS, components consist of: zMON (formally ESAMON - real-time display of performance data), zTCP (formally ESATCP - SNMP data collection), zMAP (formally ESAMAP - historical reporting and archiving), zVWS (formally ESAWEB - z/VM based web server), zTUNE (a subscription service), zVIEW (formally SHOWCASE - web based viewing of performance data), zPRO (new to the quality line of Velocity Software Products). Velocity continues to work with other software vendors to ensure smooth interface with or from other products such as VM:Webgateway, CA-Webgateway, EnterpriseWeb, MXG, MICS. Velocity software remains the leader and inovator in the z/VM performance, Linux performance, Managing cloud computing arenas.
About Us | Products | FAQ | Demo | Customer Area | Education | Linux Hints & Tips | Presentations | News | Industry and Events | Employment Opportunities
Home | Contact Us | License Info | Newsletter    

Docker Analysis - ZVPS 4.3

This page explains docker from a performance perspective. The mysterious containers are not so mysterious after looking at the numbers.

Docker containers reside inside Linux servers. What they perceive is "virtualized". But from the Linux server, everything running in the container is very visible.

The first step in understanding how it works is to measure it. A Docker container can be assigned an IP address - or a port from the host Linux server. To utilize zVPS to monitor and manage docker containers, snmp would be installed with the Velocity Software mib in each container, and then the container is monitored as any IP addressable server.

For the following anaysis, there is one docker container with node name "rksctnr1". This container lives in the "sles12" server. The sles12 server is a single virtual cpu server that also provides our OSA Data Capture. These two node names are the reporting entities in this analysis.

Inside the container runs snmp, and a looper process to drive load. From an overall system perspective, resources available to the server also appear to be available to the container. As load goes up on the container, we see exactly the same load on the server. In this analysis, the container was stopped for 45 minutes during the experiment to ensure the with and without pictures are clear

Report: ESALNXS      LINUX VSI System Analysis Report              Velocity Software Corporate   ZMAP 4.3.0
--------------------------------------------------------------------------------------------------------------------------------
                                                                                <-------------Rates per second-------------->
Node/    <---Load Numbers--> CPU >Processor Pct Util> NICE >CPU Overhead%> IO   <-Blocks> <-Swaps-> >PageFlts>
Time     Users Procs MaxProc NBR Total Syst User Idle Time Krnl IRQ  Steal Wait   IN  OUT   IN  OUT  MIN  MAJ Intrpts CSwitch
-------- ----- ----- ------- --- ----- ---- ---- ---- ---- ---- ---- ----- ---- ----- ---- ---- --- ---- ---- ------- -------
11:00:00
rksctnr1     0     0       0 Tot   0.1  0.0  0.0  6.5    0  0.0  0.0   0.1  0.0   6.8  2.5    0    0    0    0     2.0     1.9
sles12       2   102       0 Tot   0.8  0.4  0.4 98.1    0  0.0  0.0   1.1  0.0  42.5 65.0    0    0 15.9    0    31.1    34.5
--------------------------------------------------------------------------------------------------------------------------------
11:15:00
rksctnr1     0     2       0 Tot  23.3  0.5 22.7 51.6    0  0.0  0.0   5.1  0.0   1.3 53.6    0    0  0.1    0    48.4    26.4
sles12       2   103       0 Tot  22.8  0.6 22.1 72.0    0  0.0  0.0   5.1  0.0   1.3 76.4    0    0 16.3    0    54.8    33.1
--------------------------------------------------------------------------------------------------------------------------------
11:30:00
sles12       2    98       0 Tot   1.0  0.6  0.4 97.7    0  0.0  0.0   1.3  0.0     0 34.4    0    0 16.1    0    29.2    27.1
--------------------------------------------------------------------------------------------------------------------------------
11:45:00
sles12       2    97       0 Tot   1.0  0.6  0.4 97.6    0  0.0  0.0   1.4  0.0     0 35.1    0    0 16.1    0    29.0    26.1
--------------------------------------------------------------------------------------------------------------------------------
12:00:00
sles12       2   100       0 Tot   7.4  1.2  6.1 90.3    0  0.0  0.1   2.2  0.2 151.1  656    0    0 16.1    0    56.5    80.8
--------------------------------------------------------------------------------------------------------------------------------
12:15:00
rksctnr1     0     3       0 Tot  13.2  0.5 12.7 56.7    0  0.0  0.0   3.4  0.0   5.8 47.7    0    0  1.7    0    36.7    23.7
sles12       2   102       0 Tot  13.2  0.7 12.4 83.0    0  0.0  0.0   3.7  0.1  36.2  116    0    0 16.0    0    46.4    34.4
--------------------------------------------------------------------------------------------------------------------------------
12:30:00
rksctnr1     0     5       0 Tot  21.4  0.7 20.7 72.9    0  0.0  0.0   5.7  0.0     0 40.5    0    0  2.5    0    53.0    30.1
sles12       2   104       0 Tot  21.8  0.7 21.1 72.5    0  0.0  0.0   5.7  0.0     0 40.4    0    0 16.1    0    53.0    30.0
--------------------------------------------------------------------------------------------------------------------------------
12:45:00
rksctnr1     0     4       0 Tot  20.5  0.6 19.8 67.8    0  0.0  0.0   5.1  0.0  11.0 74.4    0    0  2.9    0    51.5    33.7
sles12       2   105       0 Tot  19.5  0.7 18.8 75.4    0  0.0  0.0   5.0  0.0  10.7 89.5    0    0 16.1    0    53.5    37.4
--------------------------------------------------------------------------------------------------------------------------------
 

From a process perspective, it is interesting to see how processes are reported back. On the ESALNXP report, processes using less than .1% of the 15 minute period are not reported.

In looking at the processes at the 11:15 interval, the stresser process was running to drive a load. From the container perspective, this process is process ID 22. From the host perspective, this process is process ID 37173. After the break, the container calls the process ID 22 again, but from a host perspective, this is a new container and a new set of processes, and now the host sees the process as 38218.

What this proves is that each container can have it's own process ID, and several containers could run and all have the same process ID, but the host will assign each one a unique process ID to be managed at the host level.

The snmpd proces is running that the host level -

Report: ESALNXP      LINUX HOST Process Statistics Report          Velocity Software Corporate   ZMAP 4.3.0
--------------------------------------------------------------------------------------------------------------------------------
node/     <-Process Ident-> Nice PRTY <------CPU Percents----> <---------------Storage Metrics (MB)----------> <-Faults/Second->
 Name     ID    PPID   GRP  Valu Valu  Tot  sys user syst usrt  Size RSS Peak Swap Data Stk  EXEC Lib Lck PTbl min maj mint majt
--------- ----- ----- ----- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- --- --- ---- --- --- ---- ----
11:00:00
 
sles12        0     0     0    0    0 1.05 0.49 0.39 0.04 0.14 3398  203  45K    0  36K 62.6 1154  2K 0.9 18.8 70K   0  14K    2
 snmpd     1485     1  1482  -10   10 0.55 0.26 0.29    0    0   21   10  373    0 83.9  1.5  0.3 138   0 0.49 184   0    0    0
 
11:15:00
 
rksctnr1      0     0     0    0    0 19.0 0.03 0.02 0.03 18.9   31   11  396    0  154  4.4 13.4 192   0 0.80  12   0   21    0
 stresser    22     1     1    0   20 18.9 0.00    0 0.03 18.9    4    2 32.3    0 2.11  1.2  5.6  22   0 0.12   1   0   1
 
sles12        0     0     0    0    0 20.4 0.72 0.99 0.04 18.6 3486  209    0    0    0    0    0   0   0    0 310   0   72    0
 snmpd     1485     1  1482  -10   10 0.94 0.55 0.38    0    0   21   10    0    0    0    0    0   0   0    0 255   0    0    0
 stresser 37173 37096     0    0   20 18.6 0.00    0 0.03 18.6    4    2    0    0    0    0    0   0   0    0   2   0   2
 
11:30:00
 
sles12        0     0     0    0    0 1.23 0.72 0.48 0.01 0.03 2916  181    0    0    0    0    0   0   0    0 302   0   53    0
 snmpd     1485     1  1482  -10   10 1.07 0.62 0.44    0    0   21   10    0    0    0    0    0   0   0    0 299   0    0    0
 
11:45:00
 
sles12        0     0     0    0    0 1.21 0.73 0.48 0.00 0.00 2919  181    0    0    0    0    0   0   0    0 300   0   18    0
 snmpd     1485     1  1482  -10   10 1.07 0.63 0.44    0    0   21   10    0    0    0    0    0   0   0    0 295   0    0    0
 
12:00:00
 
sles12        0     0     0    0    0 7.61 0.97 0.76 0.27 5.61 2923  193    0    0    0    0    0   0   0    0 347   0  461    0
 systemd      1     1     1    0   20 1.84    0 0.00 0.00 1.83    1    1    0    0    0    0    0   0   0    0   0   0    0    0
 containe  1289     1  1289    0   20 1.58 0.00 0.00 0.22 1.36  668   10    0    0    0    0    0   0   0    0   1   0  334    0
 snmpd     1485     1  1482  -10   10 1.04 0.61 0.43    0    0   21   10    0    0    0    0    0   0   0    0 299   0    0    0
 bash     32497 32494 32497    0   20 2.38 0.00 0.00 0.01 2.37    7    4    0    0    0    0    0   0   0    0   1   0   20    0
 
12:15:00
 
rksctnr1      0     0     0    0    0 10.8 0.03 0.02 0.02 10.7   34   12  310    0  111  4.6 16.4 169   0 0.69  12   0    9    0
 stresser    22    17     1    0   20 10.7    0    0 0.02 10.7    4    2 28.7    0 1.91  1.1  5.0  19   0 0.09   0   0
 
sles12        0     0     0    0    0 14.2 1.93 1.25 0.03 11.0 3477  217    0    0    0    0    0   0   0    0 371   0   78    0
 snmpd     1485     1  1482  -10   10 2.60 1.52 1.08    0    0   21   10    0    0    0    0    0   0   0    0 305   0    0    0
 stresser 38218 38213     0    0   20 10.9    0    0 0.01 10.9    4    2    0    0    0    0    0   0   0    0   1   0
 
12:30:00
 
rksctnr1      0     0     0    0    0 20.6 0.04 2.57 0.03 18.0   37   12  560    0  187 10.1 28.8 317   0 1.34  23   0   15    0
 stresser    22    17     1    0   20 18.0 0.00    0 0.03 18.0    4    2 53.8    0 3.57  2.0  9.3  36   0 0.18   1   0   1
 stress      90    89     1    0   20 0.57    0 0.57    0    0    2    0  2.5    0 0.05  0.1  0.0 2.2   0 0.01   0   0    0    0
 stress      99    98     1    0   20 0.54    0 0.54    0    0    2    0  2.5    0 0.05  0.1  0.0 2.2   0 0.01   0   0    0    0
 stress     126   125     1    0   20 0.51    0 0.51    0    0    2    0  2.5    0 0.05  0.1  0.0 2.2   0 0.01   0   0    0    0
 
sles12        0     0     0    0    0 21.2 0.77 0.48 0.03 20.0 3758  224    0    0    0    0    0   0   0    0 321   0   30    0
 snmpd     1485     1  1482  -10   10 1.01 0.60 0.42    0    0   21   10    0    0    0    0    0   0   0    0 295   0    0    0
 stresser 38218 38213     0    0   20 20.0 0.00    0 0.03 20.0    4    2    0    0    0    0    0   0   0    0   1   0   1
 
12:45:00
 
rksctnr1      0     0     0    0    0 20.4 0.05 3.53 0.03 16.8   39   13  540    0  175 10.2 28.2 310   0 1.32  26   0   13    0
 stresser    22    17     1    0   20 16.8 0.00 0.00 0.03 16.8    4    2 50.2    0 3.34  1.9  8.7  34   0 0.16   1   0   1
 stress     135   134     1    0   20 0.50    0 0.50    0    0    2    0  2.5    0 0.05  0.1  0.0 2.2   0 0.01   0   0    0    0
 stress     144   143     1    0   20 0.50    0 0.50    0    0    2    0  2.5    0 0.05  0.1  0.0 2.2   0 0.01   0   0    0    0
 stress     172   171     1    0   20 0.58    0 0.58    0    0    2    0  2.5    0 0.05  0.1  0.0 2.2   0 0.01   0   0    0    0
 stress     181   180     1    0   20 0.51    0 0.51    0    0    2    0  2.5    0 0.05  0.1  0.0 2.2   0 0.01   0   0    0    0
 stress     204   203     1    0   20 0.58    0 0.58    0    0    2    0  2.5    0 0.05  0.1  0.0 2.2   0 0.01   0   0    0    0
 
sles12        0     0     0    0    0 19.5 0.76 0.52 0.04 18.2 3681  228    0    0    0    0    0   0   0    0 353   0  130    0
 snmpd     1485     1  1482  -10   10 1.01 0.60 0.41    0    0   21   10    0    0    0    0    0   0   0    0 287   0    0    0
 stresser 38218 38213     0    0   20 18.2 0.00 0.00 0.02 18.1    4    2    0    0    0    0    0   0   0    0   1   0   1

Process Table Mapping

The process table inside the container is indeed very simple. This is the real time process analysis of the container, and how the host has mapped those processes. Note a one for one mapping of processes. This real time analysis is a different time period than the previous one and proces ID's have changed.

Screen: ESALNXC  Velocity Software - VSIVM4     ESAMON 4.242 11/10 17:42-
1 of 3  Linux Process Configuration             NODE RKSSCTNR1
 
                                   <------ Process ------> Appl     >Stor
Node     Process Name                 ID  PPID Group  Appl Name      Size
-------- ------------------------- ----- ----- ----- ----- -------- -----
rksctnr1 bash                          1     1     1     1 bash         .
          snmpd                     4560     1  4559  4560 snmpd    25419
          stresser.sh               4574     1     1  4563 stresser  3663
           sleep                    6626  4574     1  6626 sleep        .
 
 
 
====>
Screen: ESALNXC  Velocity Software - VSIVM4     ESAMON 4.242 11/10 17:42-
1 of 3  Linux Process Configuration             NODE SLES12
 
                                   <------ Process ------> Appl     >Stor
Node     Process Name                 ID  PPID Group  Appl Name      Size
-------- ------------------------- ----- ----- ----- ----- -------- -----
sles12     sshd                    47050  1259     0 52635 sshd         .
            bash                   47053 47050     0 52466 bash         .
           sshd                    52635  1259     0 52635 sshd     12131
            bash                   52638 52635     0 52466 bash         .
             docker                59375 52638     0 59375 docker    293K
          containerd                1289     1  1289 52452 containe  684K
           containerd-shim         52830  1289     0 52452 containe     .
            bash                   52844 52830     0 52466 bash         .
             snmpd                 57662 52844     0 57662 snmpd    25415
             stresser.sh           57683 52844     0 57669 stresser  3663
              stress               61039 57683 52844 61040 stress
               stress              61040 61039 52844 61040 stress    254
          master                    1489     1  1489  1489 master    529
           qmgr                     1524  1489  1489  1524 qmgr
           pickup                  52904  1489  1489 52326 pickup
          docker                    7195     1  7195 59375 docker    930
           exe                     52825  7195  7195 52447 exe
          systemd                  32388     1 32388 32388 systemd
           (sd-pam)                32389 32388 32388 32389 (sd-pam)
          snmpd                    52318     1     0 57662 snmpd    1939
          osasnmpd                 52322     1     0 52094 osasnmpd
           osasnmpd                52323 52322     0 52094 osasnmpd 1055

Docker Storage Analysis

From a storage perspective, the metrics coming from both perspectives show little virtualization, the numbers appear to be identical. In reality, the container has access to the host resources in the same way any other process would, and there is no virtualization of storage. Containers "fill up" the same way the Linux server does, it "fills up" when the server itself is out of memory.

Report: ESAUCD2      LINUX UCD Memory Analysis Report              Velocity Software Corporate   ZMAP 4.3.0
--------------------------------------------------------------------------------------------------------------------------------
Node/    <-------------------------Storage Sizes (in MegaBytes)-------------------------->
Time/    <--Real Storage--> <-----SWAP Storage----> Total <-------Storage in Use---------> Error
Date     Total  Avail Used  Total Avail Used  MIN   Avail CMM    Buffer Cache Ovrhd Shared Message
-------- ------ ----- ----- ----- ----- ----- ----- ----- ------ ------ ----- ----- ------ ---------------
11:00:00
rksctnr1 1897.6 139.4  1758 124.0 123.9   0.1  15.6 263.3      0    0.0  1501 257.1      0
sles12   1897.6 139.6  1758 124.0 123.9   0.1  15.6 263.5      0    0.0  1498 259.8      0
11:15:00
rksctnr1 1897.6 137.8  1760 124.0 123.9   0.1  15.6 261.6      0    0.0  1502 257.8      0
sles12   1897.6 145.8  1752 124.0 123.9   0.1  15.6 269.6      0    0.0  1498 253.3      0
11:30:00
sles12   1897.6 174.7  1723 124.0 123.9   0.1  15.6 298.5      0    0.0  1484 238.7      0
11:45:00
sles12   1897.6 176.4  1721 124.0 123.9   0.1  15.6 300.2      0    0.0  1484 237.0      0
12:00:00
sles12   1897.6 324.5  1573 124.0 123.9   0.1  15.6 448.4      0    0.0  1334 239.1      0
12:15:00
rksctnr1 1897.6 502.0  1396 124.0 123.9   0.1  15.6 625.9      0    0.0  1153 242.5      0
sles12   1897.6 511.6  1386 124.0 123.9   0.1  15.6 635.5      0    0.0  1149 237.3      0
12:30:00
rksctnr1 1897.6 507.6  1390 124.0 123.9   0.1  15.6 631.5      0    0.0  1153 236.9      0
sles12   1897.6 507.7  1390 124.0 123.9   0.1  15.6 631.5      0    0.0  1153 236.9      0
12:45:00
rksctnr1 1897.6 498.2  1399 124.0 123.9   0.1  15.6 622.1      0    0.0  1156 243.3      0
sles12   1897.6 501.7  1396 124.0 123.9   0.1  15.6 625.6      0    0.0  1154 241.9      0

Docker File System Analysis

From a file system perspective, all of the filesystems that the docker container sees as read/write are "virtualized", and have a different name at the host level.

Screen: ESAHST2  Velocity Software - VSIVM4     ESAMON 4.242 11/10 18:09-18:10
1 of 2  LINUX HOST Storage Analysis Report      CLASS * NODE RKSCTN 2828  414C
 
                   <-Utilization->
         Node/       Pct Alloc      Storage
Time     Group     Size  Used Full Units  R/W Description
-------- -------- ----- ----- ---- ----- ---- --------------------------------
18:10:00 rksctnr1   124     0  0.1    1K      Swap space
                     21    21  100    1K      Shared memory
                   1376  1376  100    1K      Cached memory
                     64     0    0    4K  r/w /dev/shm
                    949     0    0    4K  r/w /dev
                   1898     1  0.1    1K      Memory buffers
                   2022  1615 79.9    1K      Virtual memory
                  20276 11160 55.0    4K  r/w /
                   1898  1615 85.1    1K      Physical memory
 
 
F1=Help     PF2=Zoom     PF3=Quit     PF4=Select    PF5=Plot      PF6=Reset
                         PF9=Sort     PF10=Parms    PF11=More     PF12=Cancel
====>
Screen: ESAHST2  Velocity Software - VSIVM4     ESAMON 4.242 11/10 18:09-18:10
1 of 2  LINUX HOST Storage Analysis Report      CLASS * NODE SLES12 2828  414C
 
                   <-Utilization->
         Node/       Pct Alloc      Storage
Time     Group     Size  Used Full Units  R/W Description
-------- -------- ----- ----- ---- ----- ---- --------------------------------
18:10:00 sles12     949     0  0.0    4K  r/w /var/lib/docker/containers/7cfb1
                     64     0    0    4K  r/w /var/lib/docker/containers/7cfb1
                  20276 11160 55.0    4K  r/w /var/lib/pgsql
                  20276 11160 55.0    4K  r/w /opt
                  20276 11160 55.0    4K  r/w /var/opt
                  20276 11160 55.0    4K  r/w /srv
                  20276 11160 55.0    4K  r/w /tmp
                  20276 11160 55.0    4K  r/w /var/lib/named
                  20276 11160 55.0    4K  r/w /usr/local
 
 
 
 

Multiple Container Analysis

When looking at multiple containers, the numbers add up. There are three containers, each have a different version of a stress program to create load. The sles12 node is the host.

Screen: ESALNXP  Velocity Software - VSIVM4     ESAMON 4.242 11/15 20:33-20:34
1 of 3  VSI Linux Percent Usage by Process      NODE RKS*           2828  414C7
 
                           <-Process Ident-> nice prty <-----CPU Percents----->
Time     Node     Name        ID  PPID   GRP valu valu  Tot sys  user syst usrt
-------- -------- -------- ----- ----- ----- ---- ---- ---- ---- ---- ---- ----
20:34:00 rksctnr1 *Totals*     0     0     0    0    0  8.2  0.1  0.1  0.0  7.9
                  snmpd       26     1    25    0   20  0.2  0.1  0.1    0    0
                  stresser    28     1    28    0   20  7.9    0    0  0.0  7.9
                  stress    4495  4494    28    0   20  0.1    0  0.1    0    0
         rksctnr2 *Totals*     0     0     0    0    0  8.1  0.0  0.0  4.7  3.3
                  snmpd       26     1    25    0   20  0.1  0.0  0.0    0    0
                  stresser    28     1    28    0   20  8.1    0    0  4.7  3.3
         rksctnr3 *Totals*     0     0     0    0    0  9.7  0.9  0.1  8.7    0
                  snmpd       26     1    25    0   20  0.1  0.1  0.1    0    0
                  stresser    28     1    28    0   20  8.7    0    0  8.7    0
                  stress    4273  4272    28    0   20  0.8  0.8    0    0    0
 
====>
Screen: ESALNXP  Velocity Software - VSIVM4     ESAMON 4.242 11/15 20:32-20:33
1 of 3  VSI Linux Percent Usage by Process      NODE SLES12         2828  414C7
 
                           <-Process Ident-> nice prty <-----CPU Percents----->
Time     Node     Name        ID  PPID   GRP valu valu  Tot sys  user syst usrt
-------- -------- -------- ----- ----- ----- ---- ---- ---- ---- ---- ---- ----
20:33:00 sles12   *Totals*     0     0     0    0    0 32.3  0.8  0.6 16.3 14.6
                  bash         1     1     1    0   20    0    0    0    0    0
                  snmpd     2774  1289  2773    0   20  0.3  0.2  0.1    0    0
                  stresser  2795  2781  2795    0   20 11.0    0    0  0.0 11.0
                  snmpd     3109  1289  3108    0   20  0.1  0.1  0.0    0    0
                  stresser  3168  3154  3168    0   20  8.8  0.0    0  5.2  3.6
                  snmpd     3475  1289  3474    0   20  0.1  0.1  0.1    0    0
                  stresser  3502  3488  3502    0   20 11.1  0.0  0.0 11.0  0.0
                  docker    7195     1  7195    0   20  0.0    0  0.0    0    0
                  kworker/ 16184     2     0    0   20  0.0  0.0    0    0    0
                  snmpd    63481     1     0  -10   10  0.7  0.3  0.4    0    0
                  osasnmpd 63485     1     0    0   20  0.1  0.1  0.0    0    0
 

Docker Review

Docker might seem mysterious at first, but it's really not rocket science, and it is easily understood when looking at how the host sees the resources, and how the container sees them. I'm sure there is more work to understand this, but no different than understanding linux performance and capacity planning in general.




Don't miss Velocity Software's Performance Seminars