[tor-bugs] #29841 [Internal Services/Tor Sysadmin Team]: ipsec VPN generates gigantic logs
Tor Bug Tracker & Wiki
blackhole at torproject.org
Thu Mar 21 01:28:52 UTC 2019
#29841: ipsec VPN generates gigantic logs
-----------------------------------------------------+-----------------
Reporter: anarcat | Owner: tpa
Type: defect | Status: new
Priority: Low | Milestone:
Component: Internal Services/Tor Sysadmin Team | Version:
Severity: Normal | Keywords:
Actual Points: | Parent ID:
Points: | Reviewer:
Sponsor: |
-----------------------------------------------------+-----------------
Serious yak shaving night...
To try to silence this seemingly innocuous warning:
{{{
/etc/cron.daily/logrotate:
error: Compressing program wrote following message to stderr when
compressing log /var/log/syslog.1:
gzip: stdin: file size changed while zipping
}}}
... I have looked at the logrotate configuration deployed through Puppet,
and it seems slightly out of date compared to the one available in
stretch. This is the configuration left over from the stretch upgrade on
eugeni, for example:
{{{
/var/log/syslog
{
rotate 7
daily
missingok
notifempty
delaycompress
compress
postrotate
invoke-rc.d syslog-ng reload > /dev/null
endscript
}
/var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
/var/log/messages
/var/log/error
{
rotate 4
weekly
missingok
notifempty
compress
delaycompress
sharedscripts
postrotate
invoke-rc.d syslog-ng reload > /dev/null
endscript
}
}}}
Out of those, we're not doing the `syslog-ng reload`, the `delaycompress`,
`notifempty` and each logfile is in a separate block which makes it harder
to read. So I looked at doing the postrotate action, but then I realized
it was happening on the syslog logfile which *is* correctly reloaded. so
then i figured the `delaycompress` might be the bit missing.
but before enabling that blindly, I figured I would check if this would
blow up the disk space on a server. how to do that you ask? well with our
shiny new Cumin tool of course:
{{{
anarcat at curie:~(master)$ cumin -p 0 '*' 'for log in /var/log/*.log ; do if
[ `du -b "$log" | cut -f1` -gt 1000000000 ] ; then echo "logfile $log
larger than 1GB"; exit 1 ; fi; done'
74 hosts will be targeted:
alberti.torproject.org,arlgirdense.torproject.org,bracteata.torproject.org,brulloi.torproject.org
,build-
arm-[01-03].torproject.org,build-x86-[05-09].torproject.org,bungei.torproject.org,carinatum.torproject.org
,cdn-backend-
sunet-01.torproject.org,chamaemoly.torproject.org,chiwui.torproject.org,colchicifolium.torproject.org,corsicum.torproject.org,crispum.torproject.org
,crm-ext-01.torproject.org,crm-
int-01.torproject.org,cupani.torproject.org,dictyotum.torproject.org,eugeni.torproject.org,fallax.torproject.org,forrestii.torproject.org,gayi.torproject.org,getulum.torproject.org,gitlab-01.torproject.org,henryi.torproject.org
,hetzner-hel1-[01-03].torproject.org,hetzner-
nbg1-01.torproject.org,hyalinum.torproject.org,iranicum.torproject.org,kvm[4-5].torproject.org,listera.torproject.org,macrum.torproject.org,majus.torproject.org,materculae.torproject.org,meronense.torproject.org,moly.torproject.org,neriniflorum.torproject.org,nevii.torproject.org,nova.torproject.org,nutans.torproject.org,omeiense.torproject.org
,oo-
hetzner-03.torproject.org,opacum.torproject.org,orestis.torproject.org,oschaninii.torproject.org,palmeri.torproject.org,pauli.torproject.org,peninsulare.torproject.org,perdulce.torproject.org,polyanthum.torproject.org,rouyi.torproject.org,rude.torproject.org,savii.torproject.org,saxatile.torproject.org
,scw-arm-ams-01.torproject.org,scw-arm-
par-01.torproject.org,staticiforme.torproject.org,subnotabile.torproject.org,textile.torproject.org,togashii.torproject.org,troodi.torproject.org,unifolium.torproject.org,vineale.torproject.org
,web-cymru-01.torproject.org,web-hetzner-01.torproject.org
Confirm to continue [y/n]? y
|██████████████▌
| 12% (9/74) [00:47<08:25, 7.78s/hosts]
===== NODE GROUP =====
|██████████████▌
| 12% (9/74) [00:52<08:25, 7.78s/hosts]
(3) build-arm-[01-03].torproject.org
|██████████████▌
| 12% (9/74) [00:52<08:25, 7.78s/hosts]
----- OUTPUT of 'for log in /var/...xit 1 ; fi; done' -----
|██████████████▌
| 12% (9/74) [00:52<08:25, 7.78s/hosts]
Connection timed out during banner exchange
|██████████████▌
| 12% (9/74) [00:52<08:25, 7.78s/hosts]
===== NODE GROUP =====
|██████████████▌
| 12% (9/74) [00:52<08:25, 7.78s/hosts]
(5) hetzner-
hel1-01.torproject.org,kvm4.torproject.org,macrum.torproject.org,textile.torproject.org,unifolium.torproject.org
|██████████████▌
| 12% (9/74) [00:52<08:25, 7.78s/hosts]
----- OUTPUT of 'for log in /var/...xit 1 ; fi; done' -----
|██████████████▌
| 12% (9/74) [00:52<08:25, 7.78s/hosts]
logfile /var/log/daemon.log larger than 1GB
|██████████████▌
| 12% (9/74) [00:52<08:25, 7.78s/hosts]
===== NODE GROUP =====
|██████████████▌
| 12% (9/74) [00:52<08:25, 7.78s/hosts]
(1) hyalinum.torproject.org
|██████████████▌
| 12% (9/74) [00:52<08:25, 7.78s/hosts]
----- OUTPUT of 'for log in /var/...xit 1 ; fi; done' -----
|██████████████▌
| 12% (9/74) [00:52<08:25, 7.78s/hosts]
ssh: Could not resolve hostname hyalinum.torproject.org: No address
associated with hostname
|██████████████▌
| 12% (9/74) [00:52<08:25, 7.78s/hosts]
================
PASS
|████████████████████████████████████████████████████████████████████████████████████████████████████████▌
| 88% (65/74) [00:52<00:07, 1.23hosts/s]
FAIL |██████████████▌
| 12% (9/74) [00:52<08:25, 7.78s/hosts]
12.2% (9/74) of nodes failed to execute command 'for log in /var/...xit 1
; fi; done': build-arm-[01-03].torproject.org,hetzner-
hel1-01.torproject.org,hyalinum.torproject.org,kvm4.torproject.org,macrum.torproject.org,textile.torproject.org,unifolium.torproject.org
87.8% (65/74) success ratio (>= 0.0% threshold) for command: 'for log in
/var/...xit 1 ; fi; done'.:
alberti.torproject.org,arlgirdense.torproject.org,bracteata.torproject.org,brulloi.torproject.org,build-x86-[05-09].torproject.org,bungei.torproject.org,carinatum.torproject.org
,cdn-backend-
sunet-01.torproject.org,chamaemoly.torproject.org,chiwui.torproject.org,colchicifolium.torproject.org,corsicum.torproject.org,crispum.torproject.org
,crm-ext-01.torproject.org,crm-
int-01.torproject.org,cupani.torproject.org,dictyotum.torproject.org,eugeni.torproject.org,fallax.torproject.org,forrestii.torproject.org,gayi.torproject.org,getulum.torproject.org,gitlab-01.torproject.org,henryi.torproject.org
,hetzner-hel1-[02-03].torproject.org,hetzner-
nbg1-01.torproject.org,iranicum.torproject.org,kvm5.torproject.org,listera.torproject.org,majus.torproject.org,materculae.torproject.org,meronense.torproject.org,moly.torproject.org,neriniflorum.torproject.org,nevii.torproject.org,nova.torproject.org,nutans.torproject.org,omeiense.torproject.org
,oo-
hetzner-03.torproject.org,opacum.torproject.org,orestis.torproject.org,oschaninii.torproject.org,palmeri.torproject.org,pauli.torproject.org,peninsulare.torproject.org,perdulce.torproject.org,polyanthum.torproject.org,rouyi.torproject.org,rude.torproject.org,savii.torproject.org,saxatile.torproject.org
,scw-arm-ams-01.torproject.org,scw-arm-
par-01.torproject.org,staticiforme.torproject.org,subnotabile.torproject.org,togashii.torproject.org,troodi.torproject.org,vineale.torproject.org
,web-cymru-01.torproject.org,web-hetzner-01.torproject.org
87.8% (65/74) success ratio (>= 0.0% threshold) of nodes successfully
executed all commands.:
alberti.torproject.org,arlgirdense.torproject.org,bracteata.torproject.org,brulloi.torproject.org,build-x86-[05-09].torproject.org,bungei.torproject.org,carinatum.torproject.org
,cdn-backend-
sunet-01.torproject.org,chamaemoly.torproject.org,chiwui.torproject.org,colchicifolium.torproject.org,corsicum.torproject.org,crispum.torproject.org
,crm-ext-01.torproject.org,crm-
int-01.torproject.org,cupani.torproject.org,dictyotum.torproject.org,eugeni.torproject.org,fallax.torproject.org,forrestii.torproject.org,gayi.torproject.org,getulum.torproject.org,gitlab-01.torproject.org,henryi.torproject.org
,hetzner-hel1-[02-03].torproject.org,hetzner-
nbg1-01.torproject.org,iranicum.torproject.org,kvm5.torproject.org,listera.torproject.org,majus.torproject.org,materculae.torproject.org,meronense.torproject.org,moly.torproject.org,neriniflorum.torproject.org,nevii.torproject.org,nova.torproject.org,nutans.torproject.org,omeiense.torproject.org
,oo-
hetzner-03.torproject.org,opacum.torproject.org,orestis.torproject.org,oschaninii.torproject.org,palmeri.torproject.org,pauli.torproject.org,peninsulare.torproject.org,perdulce.torproject.org,polyanthum.torproject.org,rouyi.torproject.org,rude.torproject.org,savii.torproject.org,saxatile.torproject.org
,scw-arm-ams-01.torproject.org,scw-arm-
par-01.torproject.org,staticiforme.torproject.org,subnotabile.torproject.org,togashii.torproject.org,troodi.torproject.org,vineale.torproject.org
,web-cymru-01.torproject.org,web-hetzner-01.torproject.org
}}}
This might not be very easy to read, but the important bit is this:
{{{
(5) hetzner-
hel1-01.torproject.org,kvm4.torproject.org,macrum.torproject.org,textile.torproject.org,unifolium.torproject.org
----- OUTPUT of 'for log in /var/...xit 1 ; fi; done' -----
|logfile /var/log/daemon.log larger than 1GB
}}}
So I looked at the first one of those (hetzner-hel1-01) and lo and behold,
the `daemon.log` is gigantic:
{{{
1,4G /var/log/daemon.log
}}}
I looked into the file briefly and it looks like a *lot* of information
from ipsec. But before I start shaving another yak, I figured I would just
file this as a ticket to document how far I went and let this one rest for
a while.
(I did end up setting delaycompress after doing more investigations in
Prometheus about free disk space, but that's documented in the tor-puppet
commit 44f86c7d and previous.)
--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/29841>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online
More information about the tor-bugs
mailing list