Vmstat : Shows high value in w column (kthr:w)

vmstat show a large number of jobs in the W column:

Server1-bash-3.2# vmstat 3

 kthr      memory            page            disk          faults      cpu

 r b w   swap  free  re  mf pi po fr de sr sd sd sd sd   in   sy   cs us sy id

 0 0 68 76448832 36343640 166 310 23 43 44 0 257 6 6 6 8 7067 21823 8140 2 1 98

 0 0 63 68541064 19006448 191 175 0 3 3 0 0 0  0  0  2 4782 28848 5270 0  0 99

 0 0 63 68540680 19006352 234 373 0 21 21 0 0 0 0 0  2 5088 10180 5829 0  0 99

 0 0 63 68535824 19001768 191 304 0 0 0 0 0 0  0  0  2 4572 6873 4896  2  0 98

This happens when your server ran out of RAM previously, due to which the swapper had to swap out entire threads out of memory.

You can issue sar -q  1 to find out how long the threads have been swapped out and you can issue iostat -xnPz 1 on the swap device to see how busy it has been.

You can run the following command to view the process that have been swapped out :

• echo "::walk thread thr |::print kthread_t t_schedflag|::grep .'==0 '|::eval p_pidp->pid_id" | mdb -k

or

• echo "::walk thread thr |::print kthread_t t_schedflag|::grep ==0 |::eval <thr=K|::print kthread_t t_procp->p_user.u_comm" | mdb -k

A non-zero number in kthr:w field indicates that certain threads have been swapped out to the (swap) disk as there was a shortage of physical memory. Even when the system frees the memory, those processes continue to swap out till you make them do something (like just truss the process ....)

If the number of processes that are swapped out are not increasing , you are ok. However, if you want to find out which processes have been swapped out and remove them from swap, then you can issue the aboce mentioned command  to get the list of swapped processes and issue truss on them or kill them ..

If the processes are application related, merely restarting the app or rebooting the server will do it.