[tor-bugs] #22255 [Core Tor/Tor]: Frequent OOM kills of tor process
Tor Bug Tracker & Wiki
blackhole at torproject.org
Sat May 27 06:46:24 UTC 2017
#22255: Frequent OOM kills of tor process
--------------------------+------------------------------------
Reporter: DeS | Owner:
Type: defect | Status: new
Priority: High | Milestone: Tor: 0.3.0.x-final
Component: Core Tor/Tor | Version:
Severity: Major | Resolution:
Keywords: | Actual Points:
Parent ID: | Points:
Reviewer: | Sponsor:
--------------------------+------------------------------------
Comment (by DeS):
Obviously a process will die if it can´t get sufficient memory.
But as you saw in the memory logs. The normal usage is way below 400 MB.
And our Exits are limited in regards to bandwith. So we never really
exceed 4xx Mbit/s.
But sometimes as you saw in the memlog memory consumtion starts rising
fast - ending with the kill or an completly unresponsive machine.
What we are looking for is the cause for the rising memory consumpion.
I repeat this machines have been running 1-2 years without such problems
and the problem starte occuring some weeks ago. As we seen on the mailing
list it is also not just this servers - otherwise I would just set them
freshly up.
<code>
dirk at tor2:~$ ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 33432 1528 ? Ss Mai22 0:00
/sbin/init
root 2 0.0 0.0 0 0 ? S Mai22 0:00
[kthreadd]
root 3 0.1 0.0 0 0 ? S Mai22 9:18
[ksoftirqd/0]
root 5 0.0 0.0 0 0 ? S< Mai22 0:00
[kworker/0:0H]
root 7 0.2 0.0 0 0 ? S Mai22 16:50
[rcu_sched]
root 8 0.1 0.0 0 0 ? S Mai22 7:02 [rcuos/0]
root 9 0.1 0.0 0 0 ? S Mai22 7:16 [rcuos/1]
root 10 0.1 0.0 0 0 ? S Mai22 6:50 [rcuos/2]
root 11 0.1 0.0 0 0 ? S Mai22 7:25 [rcuos/3]
root 12 0.0 0.0 0 0 ? S Mai22 0:00 [rcuos/4]
root 13 0.0 0.0 0 0 ? S Mai22 0:00 [rcuos/5]
root 14 0.0 0.0 0 0 ? S Mai22 0:00 [rcuos/6]
root 15 0.0 0.0 0 0 ? S Mai22 0:00 [rcuos/7]
root 16 0.0 0.0 0 0 ? S Mai22 0:00 [rcuos/8]
root 17 0.0 0.0 0 0 ? S Mai22 0:00 [rcuos/9]
root 18 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/10]
root 19 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/11]
root 20 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/12]
root 21 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/13]
root 22 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/14]
root 23 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/15]
root 24 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/16]
root 25 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/17]
root 26 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/18]
root 27 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/19]
root 28 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/20]
root 29 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/21]
root 30 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/22]
root 31 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/23]
root 32 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/24]
root 33 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/25]
root 34 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/26]
root 35 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/27]
root 36 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/28]
root 37 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/29]
root 38 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/30]
root 39 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/31]
root 40 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/32]
root 41 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/33]
root 42 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/34]
root 43 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/35]
root 44 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/36]
root 45 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/37]
root 46 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/38]
root 47 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/39]
root 48 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/40]
root 49 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/41]
root 50 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/42]
root 51 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/43]
root 52 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/44]
root 53 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/45]
root 54 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/46]
root 55 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/47]
root 56 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/48]
root 57 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/49]
root 58 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/50]
root 59 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/51]
root 60 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/52]
root 61 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/53]
root 62 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/54]
root 63 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/55]
root 64 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/56]
root 65 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/57]
root 66 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/58]
root 67 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/59]
root 68 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/60]
root 69 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/61]
root 70 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/62]
root 71 0.0 0.0 0 0 ? S Mai22 0:00
[rcuos/63]
root 72 0.0 0.0 0 0 ? S Mai22 0:00 [rcu_bh]
root 73 0.0 0.0 0 0 ? S Mai22 0:00 [rcuob/0]
root 74 0.0 0.0 0 0 ? S Mai22 0:00 [rcuob/1]
root 75 0.0 0.0 0 0 ? S Mai22 0:00 [rcuob/2]
root 76 0.0 0.0 0 0 ? S Mai22 0:00 [rcuob/3]
root 77 0.0 0.0 0 0 ? S Mai22 0:00 [rcuob/4]
root 78 0.0 0.0 0 0 ? S Mai22 0:00 [rcuob/5]
root 79 0.0 0.0 0 0 ? S Mai22 0:00 [rcuob/6]
root 80 0.0 0.0 0 0 ? S Mai22 0:00 [rcuob/7]
root 81 0.0 0.0 0 0 ? S Mai22 0:00 [rcuob/8]
root 82 0.0 0.0 0 0 ? S Mai22 0:00 [rcuob/9]
root 83 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/10]
root 84 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/11]
root 85 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/12]
root 86 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/13]
root 87 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/14]
root 88 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/15]
root 89 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/16]
root 90 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/17]
root 91 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/18]
root 92 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/19]
root 93 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/20]
root 94 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/21]
root 95 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/22]
root 96 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/23]
root 97 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/24]
root 98 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/25]
root 99 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/26]
root 100 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/27]
root 101 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/28]
root 102 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/29]
root 103 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/30]
root 104 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/31]
root 105 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/32]
root 106 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/33]
root 107 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/34]
root 108 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/35]
root 109 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/36]
root 110 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/37]
root 111 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/38]
root 112 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/39]
root 113 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/40]
root 114 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/41]
root 115 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/42]
root 116 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/43]
root 117 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/44]
root 118 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/45]
root 119 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/46]
root 120 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/47]
root 121 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/48]
root 122 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/49]
root 123 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/50]
root 124 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/51]
root 125 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/52]
root 126 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/53]
root 127 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/54]
root 128 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/55]
root 129 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/56]
root 130 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/57]
root 131 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/58]
root 132 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/59]
root 133 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/60]
root 134 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/61]
root 135 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/62]
root 136 0.0 0.0 0 0 ? S Mai22 0:00
[rcuob/63]
root 137 0.0 0.0 0 0 ? S Mai22 0:00
[migration/0]
root 138 0.0 0.0 0 0 ? S Mai22 0:01
[watchdog/0]
root 139 0.0 0.0 0 0 ? S Mai22 0:00
[watchdog/1]
root 140 0.0 0.0 0 0 ? S Mai22 0:00
[migration/1]
root 141 0.0 0.0 0 0 ? S Mai22 4:14
[ksoftirqd/1]
root 143 0.0 0.0 0 0 ? S< Mai22 0:00
[kworker/1:0H]
root 144 0.0 0.0 0 0 ? S Mai22 0:01
[watchdog/2]
root 145 0.0 0.0 0 0 ? S Mai22 0:00
[migration/2]
root 146 0.1 0.0 0 0 ? S Mai22 8:21
[ksoftirqd/2]
root 148 0.0 0.0 0 0 ? S< Mai22 0:00
[kworker/2:0H]
root 149 0.0 0.0 0 0 ? S Mai22 0:01
[watchdog/3]
root 150 0.0 0.0 0 0 ? S Mai22 0:00
[migration/3]
root 151 0.0 0.0 0 0 ? S Mai22 3:46
[ksoftirqd/3]
root 153 0.0 0.0 0 0 ? S< Mai22 0:00
[kworker/3:0H]
root 154 0.0 0.0 0 0 ? S< Mai22 0:00 [khelper]
root 155 0.0 0.0 0 0 ? S Mai22 0:00
[kdevtmpfs]
root 156 0.0 0.0 0 0 ? S< Mai22 0:00 [netns]
root 157 0.0 0.0 0 0 ? S< Mai22 0:00
[writeback]
root 158 0.0 0.0 0 0 ? S< Mai22 0:00
[kintegrityd]
root 159 0.0 0.0 0 0 ? S< Mai22 0:00 [bioset]
root 160 0.0 0.0 0 0 ? S< Mai22 0:00
[kworker/u129:0]
root 161 0.0 0.0 0 0 ? S< Mai22 0:00 [kblockd]
root 162 0.0 0.0 0 0 ? S< Mai22 0:00 [ata_sff]
root 163 0.0 0.0 0 0 ? S Mai22 0:00 [khubd]
root 164 0.0 0.0 0 0 ? S< Mai22 0:00 [md]
root 165 0.0 0.0 0 0 ? S< Mai22 0:00
[devfreq_wq]
root 166 0.3 0.0 0 0 ? R Mai22 20:23
[kworker/0:1]
root 168 0.0 0.0 0 0 ? S Mai22 0:00
[khungtaskd]
root 169 2.3 0.0 0 0 ? S Mai22 154:44 [kswapd0]
root 170 0.0 0.0 0 0 ? S< Mai22 0:00 [vmstat]
root 171 0.0 0.0 0 0 ? SN Mai22 0:00 [ksmd]
root 172 0.0 0.0 0 0 ? SN Mai22 0:02
[khugepaged]
root 173 0.0 0.0 0 0 ? S Mai22 0:00
[fsnotify_mark]
root 174 0.0 0.0 0 0 ? S Mai22 0:00
[ecryptfs-kthrea]
root 175 0.0 0.0 0 0 ? S< Mai22 0:00 [crypto]
root 187 0.0 0.0 0 0 ? S< Mai22 0:00
[kthrotld]
root 189 0.2 0.0 0 0 ? S Mai22 13:59
[kworker/1:1]
root 208 0.0 0.0 0 0 ? S< Mai22 0:00 [deferwq]
root 209 0.0 0.0 0 0 ? S< Mai22 0:00
[charger_manager]
root 264 0.0 0.0 0 0 ? S< Mai22 0:00
[kpsmoused]
root 266 0.0 0.0 0 0 ? S Mai22 0:00
[scsi_eh_0]
root 267 0.0 0.0 0 0 ? S Mai22 0:00 [hpsa]
root 270 0.0 0.0 0 0 ? S Mai22 0:00
[scsi_eh_1]
root 271 0.0 0.0 0 0 ? S Mai22 0:00
[scsi_eh_2]
root 272 0.0 0.0 0 0 ? S Mai22 0:00
[scsi_eh_3]
root 273 0.0 0.0 0 0 ? S Mai22 0:00
[scsi_eh_4]
root 274 0.0 0.0 0 0 ? S Mai22 0:00
[scsi_eh_5]
root 275 0.0 0.0 0 0 ? S Mai22 0:00
[scsi_eh_6]
root 281 0.2 0.0 0 0 ? S Mai22 18:31
[kworker/2:1]
root 284 0.4 0.0 0 0 ? S Mai22 26:21
[kworker/3:1]
root 293 0.0 0.0 0 0 ? S< Mai22 0:00
[kdmflush]
root 294 0.0 0.0 0 0 ? S< Mai22 0:00 [bioset]
root 296 0.0 0.0 0 0 ? S< Mai22 0:00
[kdmflush]
root 297 0.0 0.0 0 0 ? S< Mai22 0:00 [bioset]
root 314 0.0 0.0 0 0 ? S Mai22 0:01
[jbd2/dm-0-8]
root 315 0.0 0.0 0 0 ? S< Mai22 0:00 [ext4
-rsv-conver]
root 440 0.0 0.0 0 0 ? S< Mai22 0:00 [ext4
-rsv-conver]
root 479 0.0 0.0 19484 196 ? S Mai22 0:00 upstart-
udev-bridge --daemon
root 506 0.0 0.0 51540 4 ? Ss Mai22 0:00
/lib/systemd/systemd-udevd --daemon
root 525 0.0 0.0 15380 208 ? S Mai22 0:00 upstart-
file-bridge --daemon
root 557 0.0 0.0 0 0 ? SN Mai22 0:21 [kipmi0]
message+ 617 0.0 0.0 39252 640 ? Ss Mai22 0:00 dbus-
daemon --system --fork
syslog 639 0.0 0.0 255848 1688 ? Ssl Mai22 0:00 rsyslogd
root 644 0.0 0.0 0 0 ? S< Mai22 0:00 [kvm-
irqfd-clean]
root 704 0.0 0.0 43456 924 ? Ss Mai22 0:00
/lib/systemd/systemd-logind
root 738 0.0 0.0 15268 216 ? S Mai22 0:00 upstart-
socket-bridge --daemon
root 1034 0.0 0.0 17312 8 tty4 Ss+ Mai22 0:00
/sbin/getty -8 38400 tty4
root 1037 0.0 0.0 17312 8 tty5 Ss+ Mai22 0:00
/sbin/getty -8 38400 tty5
root 1042 0.0 0.0 17312 8 tty2 Ss+ Mai22 0:00
/sbin/getty -8 38400 tty2
root 1043 0.0 0.0 17312 8 tty3 Ss+ Mai22 0:00
/sbin/getty -8 38400 tty3
root 1045 0.0 0.0 17312 8 tty6 Ss+ Mai22 0:00
/sbin/getty -8 38400 tty6
root 1060 0.0 0.0 61388 320 ? Ss Mai22 0:00
/usr/sbin/sshd -D
root 1069 0.0 0.0 4376 4 ? Ss Mai22 0:00 acpid -c
/etc/acpi/events -s /var/run/acpid.socket
root 1095 0.0 0.0 23660 192 ? Ss Mai22 0:00 cron
daemon 1096 0.0 0.0 19144 0 ? Ss Mai22 0:00 atd
root 1116 0.0 0.0 19200 400 ? Ss Mai22 0:29
/usr/sbin/irqbalance
unbound 1265 2.9 1.3 96720 51748 ? Ss Mai22 193:16
/usr/sbin/unbound
vnstat 1317 0.0 0.0 7484 360 ? Ss Mai22 0:16
/usr/sbin/vnstatd -d --pidfile /run/vnstat/vnstat.pid
root 1396 0.0 0.0 17312 8 tty1 Ss+ Mai22 0:00
/sbin/getty -8 38400 tty1
root 1410 0.0 0.0 0 0 ? S Mai22 0:00 [kauditd]
dirk 1610 0.0 0.0 14100 840 ? S Mai22 0:10 /bin/bash
-e ./memlog.sh
root 1836 0.0 0.0 0 0 ? S Mai26 0:00
[kworker/1:0]
root 13028 0.0 0.0 0 0 ? S Mai23 0:00
[kworker/0:0]
root 14819 0.0 0.0 0 0 ? S< Mai23 0:00
[kworker/u129:1]
root 15495 0.0 0.0 0 0 ? S 02:26 0:00
[kworker/u128:1]
root 20329 0.0 0.0 0 0 ? S 08:06 0:00
[kworker/3:2]
debian-+ 20802 28.6 3.3 309628 128476 ? Sl 08:11 7:49
/usr/bin/tor --defaults-torrc /usr/share/tor/tor-service-defaults-torrc -f
/etc/tor/torrc1 --hush
debian-+ 21033 18.4 3.2 305140 124116 ? Rl 08:12 4:49
/usr/bin/tor --defaults-torrc /usr/share/tor/tor-service-defaults-torrc -f
/etc/tor/torrc2 --hush
root 21243 0.0 0.0 0 0 ? S 08:23 0:00
[kworker/u128:2]
root 21386 0.0 0.1 105656 4340 ? Ss 08:34 0:00 sshd:
dirk [priv]
dirk 21451 0.0 0.0 105656 1888 ? S 08:34 0:00 sshd:
dirk at pts/0
dirk 21452 0.0 0.0 24080 3592 pts/0 Ss 08:34 0:00 -bash
dirk 21521 0.0 0.0 4352 360 ? S 08:37 0:00 sleep 60
dirk 21523 0.0 0.0 20000 1316 pts/0 R+ 08:38 0:00 ps aux
root 47576 0.0 0.0 0 0 ? S Mai25 0:00
[kworker/2:2]
</code>
I will attach a screenshot
https://trac.torproject.org/projects/tor/attachment/ticket/22255/Screenshot_2017-05-26_09-56-09.png
of top when the problem was akut and I was logged in yesterday. I could
not investigate more this the machine was very unresponsive (I was kicked
out of shell / local console)
--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/22255#comment:29>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online
More information about the tor-bugs
mailing list