Vmstat : Shows high value in w column (kthr:w)

vmstat show a large number of jobs in the W column:

Server1-bash-3.2# vmstat 3

 kthr      memory            page            disk          faults      cpu

 r b w   swap  free  re  mf pi po fr de sr sd sd sd sd   in   sy   cs us sy id

 0 0 68 76448832 36343640 166 310 23 43 44 0 257 6 6 6 8 7067 21823 8140 2 1 98

 0 0 63 68541064 19006448 191 175 0 3 3 0 0 0  0  0  2 4782 28848 5270 0  0 99

 0 0 63 68540680 19006352 234 373 0 21 21 0 0 0 0 0  2 5088 10180 5829 0  0 99

 0 0 63 68535824 19001768 191 304 0 0 0 0 0 0  0  0  2 4572 6873 4896  2  0 98

This happens when your server ran out of RAM previously, due to which the swapper had to swap out entire threads out of memory.

You can issue sar -q  1 to find out how long the threads have been swapped out and you can issue iostat -xnPz 1 on the swap device to see how busy it has been.

You can run the following command to view the process that have been swapped out :

• echo "::walk thread thr |::print kthread_t t_schedflag|::grep .'==0 '|::eval p_pidp->pid_id" | mdb -k

or

• echo "::walk thread thr |::print kthread_t t_schedflag|::grep ==0 |::eval <thr=K|::print kthread_t t_procp->p_user.u_comm" | mdb -k

A non-zero number in kthr:w field indicates that certain threads have been swapped out to the (swap) disk as there was a shortage of physical memory. Even when the system frees the memory, those processes continue to swap out till you make them do something (like just truss the process ....)

If the number of processes that are swapped out are not increasing , you are ok. However, if you want to find out which processes have been swapped out and remove them from swap, then you can issue the aboce mentioned command  to get the list of swapped processes and issue truss on them or kill them ..

If the processes are application related, merely restarting the app or rebooting the server will do it.

LDOM : Links

• To remove a domain
• ldm stop-domain -a
•  ldm unbind-domain ldom1
•  ldm remove-domain -a
• To remove all domains and set to factory-default
• http://docs.oracle.com/cd/E19608-01/html/821-1485/factorydefaultconfiganddisablingldoms.html 

• Reference documentation for ldm command
• http://docs.oracle.com/cd/E23120_01/html/821-2855/ldm-1m.html#scrolltoc

• Check ldm version
• ldm -V 

•  LDOM installation whitepaper:
• http://www.oracle.com/technetwork/articles/systems-hardware-architecture/vmsrvrsparc-reliability-163931.pdf

LDOMS :Getting started Part 2

• This post describes the steps to create 7 ldoms in a T4-2 core such that each LDOM uses a core from the first socket.
•  
• Start the required services on the primary domain

# ldm ls
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -n-c--  UART    128   130560M  0.0%  9m
# ldm list-services
# ldm add-vds primary-vds0 primary
# ldm add-vcc port-range=5000-51000 primary-vcc0 primary
# ldm add-vsw net-dev=net0 primary-vsw0 primary
# ldm list-services
VCC
    NAME             LDOM             PORT-RANGE
    primary-vcc0     primary          5000-5100
VSW
    NAME             LDOM             MAC               NET-DEV   ID   DEVICE     LINKPROP   DEFAULT-VLAN-ID PVID VID   MTU   MODE   INTER-VNET-LINK
    primary-vsw0     primary          00:14:4f:f9:59:1c net0      0    switch@0              1               1   1500         on        

VDS
    NAME             LDOM             VOLUME         OPTIONS          MPGROUP        DEVICe
    primary-vds0     primary

• STart vntsd service

# svcadm enable vntsd
# svcs vntsd
STATE          STIME    FMRI
online         14:31:06 svc:/ldoms/vntsd:default
#

Add 7 disks for OS install for 7 ldoms

ldm add-vdsdev /dev/rdsk/c1t0d4s2  c1t0d4s2@primary-vds0
ldm add-vdsdev /dev/rdsk/c1t0d6s2  c1t0d6s2@primary-vds0
ldm add-vdsdev /dev/rdsk/c1t0d8s2  c1t0d8s2@primary-vds0
ldm add-vdsdev /dev/rdsk/c1t0d10s2  c1t0d10s2@primary-vds0
ldm add-vdsdev /dev/rdsk/c2t4d5s2  c2t4d5s2@primary-vds0
ldm add-vdsdev /dev/rdsk/c2t4d7s2  c2t4d7s2@primary-vds0
ldm add-vdsdev /dev/rdsk/c2t4d9s2  c2t4d9s2@primary-vds0

Or export ZFS volume for OS install
#ldm rm-vdisk disk1 ldom1
#zpool destroy ldom1P
zpool create ldom1P c1t0d4s2
zfs create ldom1P/ldom1
zfs create -V 20gb ldom1P/ldom1/disk0
ldm add-vdsdev /dev/zvol/dsk/ldom1P/ldom1/disk0 ldom1_disk@primary-vds0


zpool create ldom2P c1t0d6s2
zfs create ldom2P/ldom
zfs create -V 20gb ldom2P/ldom/disk0
ldm add-vdsdev /dev/zvol/dsk/ldom2P/ldom/disk0 ldom2_disk@primary-vds0


zpool create ldom3P c1t0d8s2
zfs create ldom3P/ldom
zfs create -V 20gb ldom3P/ldom/disk0
ldm add-vdsdev /dev/zvol/dsk/ldom3P/ldom/disk0 ldom3_disk@primary-vds0


zpool create ldom4P c1t0d10s2
zfs create ldom4P/ldom
zfs create -V 20gb ldom4P/ldom/disk0
zpool create -f ldom5P c2t4d5s2
zfs create ldom5P/ldom
zfs create -V 20gb ldom5P/ldom/disk0
ldm add-vdsdev /dev/zvol/dsk/ldom5P/ldom/disk0 ldom5_disk@primary-vds0



zpool create -f ldom6P c2t4d7s2
zfs create ldom6P/ldom
zfs create -V 20gb ldom6P/ldom/disk0
ldm add-vdsdev /dev/zvol/dsk/ldom6P/ldom/disk0 ldom6_disk@primary-vds0


zpool create -f ldom7P c2t4d9s2
zfs create ldom7P/ldom
zfs create -V 20gb ldom7P/ldom/disk0
ldm add-vdsdev /dev/zvol/dsk/ldom7P/ldom/disk0 ldom7_disk@primary-vds0

Set up the primary domain with 1 core and 7680M of memory

ldm start-reconf primary
ldm set-memory 7680M  primary
ldm set-core cid=0 primary
ldm add-config initial


Power cycle the server
 reset /SYS

Create the 7 ldoms as under:

ldm add-domain ldom1
ldm set-core cid=1 ldom1
ldm add-vdisk  disk1 ldom1_disk@primary-vds0 ldom1
ldm add-vnet vnet1 primary-vsw0 ldom1
ldm set-variable auto-boot\?=true ldom1
ldm set-variable  boot-device=disk1 ldom1
ldm add-domain ldom2
ldm set-core cid=2 ldom2
ldm add-vdisk  disk2 ldom2_disk@primary-vds0 ldom2
ldm add-vnet vnet2 primary-vsw0 ldom2
ldm set-variable auto-boot\?=true ldom2
ldm set-variable  boot-device=disk2 ldom2
ldm add-domain ldom3
ldm set-core cid=3 ldom3
ldm add-vdisk  disk3 ldom3_disk@primary-vds0 ldom3
ldm add-vnet vnet3 primary-vsw0 ldom3
ldm set-variable auto-boot\?=true ldom3
ldm set-variable  boot-device=disk3 ldom3
ldm add-domain ldom4
ldm set-core cid=4 ldom4
ldm add-vdisk  disk4 ldom4_disk@primary-vds0 ldom4
ldm add-vnet vnet4 primary-vsw0 ldom4
ldm set-variable auto-boot\?=true ldom4
ldm set-variable  boot-device=disk4 ldom4
ldm add-domain ldom5
ldm set-core cid=5 ldom5
ldm add-vdisk  disk5 ldom5_disk@primary-vds0 ldom5
ldm add-vnet vnet5 primary-vsw0 ldom5
ldm set-variable auto-boot\?=true ldom5
ldm set-variable  boot-device=disk5 ldom5
ldm add-domain ldom6
ldm set-core cid=6 ldom6
ldm add-vdisk  disk6 ldom6_disk@primary-vds0 ldom6
ldm add-vnet vnet6 primary-vsw0 ldom6
ldm set-variable auto-boot\?=true ldom6
ldm set-variable  boot-device=disk6 ldom6
ldm add-domain ldom7
ldm set-core cid=7 ldom7
ldm add-vdisk  disk7 ldom7_disk@primary-vds0 ldom7
ldm add-vnet vnet7 primary-vsw0 ldom7
ldm set-variable auto-boot\?=true ldom7
ldm set-variable  boot-device=disk7 ldom7
ldm add-domain park
ldm set-vcpu -c 8 park




Assign them memory

ldm add-memory 8G park
ldm add-memory 8G ldom1
ldm add-memory 8G park
ldm add-memory 8G ldom2
ldm add-memory 8G park
ldm add-memory 8G ldom3
ldm add-memory 8G park
ldm add-memory 8G ldom4
ldm add-memory 8G park
ldm add-memory 8G ldom5
ldm add-memory 8G park
ldm add-memory 8G ldom6
ldm add-memory 8G park
ldm add-memory 8G ldom7
ldm add-memory 8G park


Bind the domains:

ldm bind-domain ldom1
ldm bind-domain ldom2
ldm bind-domain ldom3
ldm bind-domain ldom4
ldm bind-domain ldom5
ldm bind-domain ldom6
ldm bind-domain ldom7
ldm bind-domain park

• 
  

LDOMS: Getting started

• Verify if LDOM is already installed
• svcs ldmd
• if not, install LDOM using the following command
• pkg install ldomsmanager
• Verify that primary domain exist
• ldm list primary

• Check if the primary domain have the required services
• ldm ldm list-services primary

• If services are not available in primary domain, add these services. Add a virtual disk server, virutal console concentrator and virutal switch to the primary domain. These virtual services must be created to use the control domain as a service domain and to create virtual devices for other domains.
• Create a virtual disk server to allow importing virtual disks into a logical domain
  
• ldm add-vds primary-vds0 primary
• Export physical disks as virtual disks . Later these disks can be assigned to guest domain
• ldm add-vdsdev /dev/rdsk/c3t0d1s2  c3t0d1s2@primary-vds0

• Verify that virtual disks has been added to virtual disk server by checking the VDS section of the following output
• ldm list-bindings

• Add a virtual console concentrator
• ldm add-vcc port-range=5000-5100 primary-vcc0 primary

• Find the primary network interface (dladm show-phys) and add a virtual switch associated with primary network card. Virtual switch service enables networking between virtual network (vnet) devices in logical domains. The following command creates vritual switch named primary-vsw0 on net0 interface in the primary domain
• ldm add-vsw net-dev=net0 primary-vsw0 primary

• Verify that vsw0 device is created by issue dlam show-phys command 
• # dladm show-phys
  LINK              MEDIA                STATE      SPEED  DUPLEX    DEVICE
  net1              Ethernet             unknown    0      unknown   igb1
  net4              Ethernet             up         10     full      usbecm2
  net3              Ethernet             unknown    0      unknown   igb3
  net0              Ethernet             up         1000   full      igb0
  net2              Ethernet             unknown    0      unknown   igb2
  net6              Ethernet             up         1000   full      vsw0
• Also , verify that switch is created over on net0 interface
• VSW
      NAME             MAC               NET-DEV   ID   DEVICE     LINKPROP   DEFAULT-VLAN-ID PVID VID                  MTU   MODE   INTER-VNET-LINK
      primary-vsw0     00:14:4f:fb:3f:ea net0      0    switch@0              1               1                         1500         on    
  

• Verify the services have been created by using the list-services command
• ldm list-services primary

• Remove some resources from primary domain so it can be allocated to guest domains. Configure the control domain with 32 virutal CPUs and 16GB of memory
• # ldm start-reconf primary
• # ldm set-vcpu 32 primary
• # ldm set-memory 16G primary
• # ldm set-crypto 4 primary

• Save the configuration and reboot the system
• ldm add-config initial
• ldm list-config
  
• init 6

• Enable the virtual network terminal server daemon to provide access to the virtual console to each logical domain
• svcadm enable vntsd
• svcs vntsd
  

• Create the guest domain ldom1 from the control domain. Assign 48 VCPUs and 24G of memory to it.
• # ldm create ldom1
• # ldm set-vcpu 48 ldom1
• # ldm set-memory 24G ldom1
• # ldm set-crypto 6 ldom1
• Assign the disk to your domain
• ldm add-vdisk pdisk c3t0d1s2@primary-vds0  ldom1 

•  Set auto-boot? and boot-device variables for the guest domain 
• ldm set-variable auto-boot\?=true ldom1
• ldm set-variable  boot-device=pdisk ldom1

• Assign vnet to the domain
• ldm add-vnet vnet1 primary-vsw0 ldom1