Process stopped unexpectedly
Suddenly killed tasks are often result of the system running out of memory, which is when so called Out-of-memory (OOM) killer steps in. If a task gets killed to save memory, it gets logged into various log files stored at /var/log/
You can search the logs for messages of out of memory alerts with
sudo grep -i -r 'out of memory' /var/log/
Grep goes through all logs under the directory and therefor will show at least the just ran command itself from the /var/log/auth.log. Actual log marks of OOM killed processes would look something like this
kernel: Out of memory: Kill process 9163 (mysqld) score 511 or sacrifice child
The log note here shows the process killed was mysqld with pid 9163 and OOM score of 511 at the time it was killed. Your log messages may vary depending on Linux distribution and system configuration.
If for example a process crucial to your web application was killed as a result of out of memory situation, you have a couple of options, reduce the amount of memory asked by the process, disallow processes to overcommit memory, or simply add more memory to your server configuration.
Current resource usage
Linux comes with a few handy process tracking tools that can help with identifying possible resource outages. You can track memory usage with
The command prints out current memory statistics, for example in 1 GB system the output is something along the lines of
total used free shared buffers cached Mem: 993M 738M 255M 5.7M 64M 439M -/+ buffers/cache: 234M 759M Swap: 0B 0B 0B
Here it is important to make the distinction between application used memory, buffers and caches. On the Mem –line of the output it would appear nearly 75% of our RAM is in use, but then again over half of the used memory is occupied by cached data. The difference is that while applications reserve memory for their own use, the cache is simply commonly used hard drive data that the kernel stores temporarily in RAM space for faster access, which on the application level is considered free memory. Keeping that in mind, it’s easier to understand why used and free memory are listed twice, on the second line is conveniently calculated the actual memory usage when taken into account the amount of memory occupied by buffers and cache.
In this example the system is using merely 234MB of the total available 993MB, and no process is immediate danger of being killed to save resources.
Another useful tool for memory monitoring is ‘top’, which displays useful continuously updated information about processes’ memory and CPU usage, run time and other statistics. This is particularly useful for identifying resource exhaustive tasks.
You can scroll the list using Page Up and Page Down buttons on your keyboard. The program runs on the foreground until cancelled by pressing ‘q’ to quit. The resource usage is shown in percentages and gives an easy overview of your systems workload.
top - 17:33:10 up 6 days, 1:22, 2 users, load average: 0.00, 0.01, 0.05 Tasks: 72 total, 2 running, 70 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.3 us, 0.0 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 1017800 total, 722776 used, 295024 free, 66264 buffers KiB Swap: 0 total, 0 used, 0 free. 484748 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1 root 20 0 33448 2784 1448 S 0.0 0.3 0:02.91 init 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd 3 root 20 0 0 0 0 S 0.0 0.0 0:00.02 ksoftirqd/0 5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H 6 root 20 0 0 0 0 S 0.0 0.0 0:01.92 kworker/u2:0 7 root 20 0 0 0 0 S 0.0 0.0 0:05.48 rcu_sched
In the example output shown above, the system is idle and the memory usage is nominal.
Check if your process is at risk
If your server’s memory gets used up to the extent at which it can threaten system stability, the Out-of-memory killer will choose which process to eliminate based on many variables such as the amount of work done that would be lost and total memory freed. Linux keeps a score for each running process, which represents the likelihood at which the process would be killed in OOM situation.
This score is stored on file in /proc/<pid>/oom_score, where pid is the identification number for the process you are looking into. The pid can be easily found with
ps aux | grep <process name>
The output of the command when searching for mysql for example would be similar to
mysql 5872 0.0 5.0 623912 51236 ? Ssl Jul16 2:42 /usr/sbin/mysqld
Here the process ID is the first number on the row, 5872 in this case, which then can be used to get further information on this particular task.
The read out of this gives us a single numerical value for the chance of the process getting axed by the OOM killer. The higher the number the more likely the task is to be chosen if an out of memory situation should arise.
If your important process has a very high OOM score, it is possible the process is wasting memory and should be looked into. However just high OOM score, if the memory usage otherwise remains nominal, is not reason for concern. OOM killer can be disabled, but this is not recommended as it might cause unhandled exceptions in out of memory situations, possibly leading to a kernel panic or even a system halt.
Disable over commit
In major Linux distributions the kernel allows by default for processes to request more memory than is currently free in the system to improve the memory utilization. This is based on heuristics that the processes never truly use all the memory they request, however if your system is at risk of running out of memory, and you wish to prevent loosing tasks to OOM killer, it is possible to disallow memory overcommit.
To change how the system handles overcommit calls Linux has an application called ‘sysctl’ that is used to modify kernel parameters at runtime. You can list all sysctl controlled parameters with
sudo sysctl -a
The particular parameters that controls memory are very imaginatively named vm.overcommit_memory and vm.overcommit_ratio. To change the overcommit mode, use the following command
sudo sysctl -w vm.overcommit_memory=2
This parameter has 3 different values:
- 0 means to “Estimate if we have enough RAM”
- 1 equals to “Always allow”
- 2 that is used here tells kernel to “Say no if the system doesn’t have the memory”
The important part in changing the overcommit mode is to remember to also change the overcommit_ratio. When overcommit_memory is set to 2, the committed address space is not permitted to exceed swap space plus this percentage of physical RAM. To be able to use all of the systems memory use command
sudo sysctl -w vm.overcommit_ratio=100
These changes are applied immediately but will only persist until the next system reboot. To have the changes remain permanent, the same parameter values need to be added to sysctl.conf –file. Open the configuration file for edit with
sudo nano /etc/sysctl.conf
Add the same lines to the end of the file.
Save the changes (ctrl + O) and exit (ctrl + X) the editor. Your server will read the configurations every time at boot up, and prevent applications from over committing memory.