How to determine which process is highly memory consuming over time

A little while back i had a problem with a customers server which were consuming all memory within a short time. I was wondering how i could track which process or processes this were due to, and decided to make a small script to run over night where i knew the memory usage would blow up.

mem_logger.bash

 #!/bin/bash

 while true
 do
   exec >> ./mem_logger.log
   printf "\n\n"
   echo "--------------------------------------"
   date
   echo "--------------------------------------"
   printf "\n"
   ps -eo pmem,pcpu,vsize,pid,user,command | sort -k 1 -nr | head -n 5
   sleep 30;
 done

Run this in the background with:

$ ./mem_logger.bash &

Every 30 second it will run the following command with a timestamp in the output, and append it to a file (we use the exec command to global redirect all stdout to mem_logger.log):

$ ps -eo pmem,pcpu,vsize,pid,user,command | sort -k 1 -nr | head -n 5

Explained:

ps is the process command

-e is the same as -A and means to show other users processes aswell

-o is a flag to filter what options and informations to show (in this case pmem, pcpu, vsize, pid, user, and command)

Then we pipe to sort from key 1 with numeric sorting (the first row which is pmem - we chose this with -o).

Then we pipe to head which, in this case, will show us the 5 first lines. The top 5 memory consuming processes. You can also change this if you want more than the top 5 processes in your file.

The output will be something like the following:

Now you will have a file (mem_logger.log) with the ps commands output over a time span. This gives you a lot of great options to determine what process or program to debug. A good idea is to use other UNIX utils to analyse mem_logger.log.

Feel free to comment. I am very interested to hear how you have determined memory problems, or if you know some great utils to track this.

cpu_logger.bash

If you want to monitor CPU instead, just change around on the -o arguments:

#/bin/bash

 while true
 do
   exec >> ./cpu_logger.log
   printf "\n\n"
   echo "--------------------------------------"
   date
   echo "--------------------------------------"
   printf "\n"
   ps -eo pcpu,pmem,vsize,pid,user,command | sort -k 1 -nr | head -n 30
   sleep 30;
 done