[tor-bugs] #34098 [Internal Services/Tor Sysadmin Team]: crm-int-01 running out of disk space

[Tor Bug Tracker & Wiki]

#34098: crm-int-01 running out of disk space
-----------------------------------------------------+-----------------
     Reporter:  anarcat                              |      Owner:  tpa
         Type:  defect                               |     Status:  new
     Priority:  Very High                            |  Milestone:
    Component:  Internal Services/Tor Sysadmin Team  |    Version:
     Severity:  Major                                |   Keywords:
Actual Points:                                       |  Parent ID:
       Points:                                       |   Reviewer:
      Sponsor:                                       |
-----------------------------------------------------+-----------------
 We're at 92% disk use on crm-int-01 suddenly... It seems that disk usage
 grew suddenly three days ago:

 https://grafana.torproject.org/d/ER3U2cqmk/node-exporter-server-
 metrics?panelId=31&fullscreen&orgId=1&var-node=crm-
 ext-01.torproject.org:9100&var-node=crm-
 int-01.torproject.org:9100&from=1585834161059&to=1588426161060

 The mariadb server was stopped this morning as well. It seems innodb
 crashes with the following assertion:

 {{{
 2020-05-02  9:08:31 41 [ERROR] InnoDB: preallocating 65536 bytes for file
 ./torcrm_prod/civicrm_acl_contact_cache.ibd failed with error 28
 2020-05-02  9:08:31 41 [Warning] InnoDB: Cannot create table
 `torcrm_prod`.`civicrm_acl_contact_cache` because tablespace full
 2020-05-02 09:08:31 0x7f6c50252700  InnoDB: Assertion failure in file
 /build/mariadb-10.3-qB78gy/mariadb-10.3-10.3.22/storage/innobase/dict/dict0dict.cc
 line 491
 }}}

 There are also warnings on startup:

 {{{
 2020-05-02 13:23:17 0 [Note] InnoDB: Ignoring data file './torcrm_prod
 /#sql-ib1158381.ibd' with space ID 1137566. Another data file called
 ./torcrm_prod/civicrm_acl_contact_cache.ibd exists with the same space ID.
 2020-05-02 13:23:17 0 [Note] InnoDB: Ignoring data file
 './torcrm_prod/civicrm_acl_contact_cache.ibd' with space ID 1137566.
 Another data file called ./torcrm_prod/#sql-ib1158381.ibd exists with the
 same space ID.
 }}}

 We have ~1.5GB left on the server:

 {{{
 Filesystem      Size  Used Avail Use% Mounted on
 /dev/sda1        20G   18G  1.6G  92% /
 }}}

 Most of that 18G is in /var with 5GB split between the two databases:

 {{{
     2.4 GiB [##########] /torcrm_prod
     2.3 GiB [######### ] /torcrm_staging
 }}}

 Here's the top 10 tables in disk usage:

 {{{
   528.0 MiB [##########]  civicrm_mailing_event_queue.ibd
   432.0 MiB [########  ]  civicrm_mailing_recipients.ibd
   340.0 MiB [######    ]  civicrm_activity_contact.ibd
   172.0 MiB [###       ]  civicrm_contact.ibd
   168.0 MiB [###       ]  civicrm_log.ibd
   148.0 MiB [##        ]  civicrm_mailing_event_delivered.ibd
    80.0 MiB [#         ]  civicrm_activity.ibd
    60.0 MiB [#         ]  civicrm_group_contact.ibd
    52.0 MiB [          ]  civicrm_subscription_history.ibd
    52.0 MiB [          ]  civicrm_email.ibd
 }}}

 Backups take up the most space, however, at about 10GB. I am not familiar
 with how the backup system works on that host, but there are about 7.5GB
 of SHA256-* files in there:

 {{{
 root at crm-int-01:/var/backups/local/mysql# du -sch SHA256-* | tail -1
 7.6G    total
 }}}

 Some of those are fairly old too:

 {{{
 root at crm-int-01:/var/backups/local/mysql# ls -alt SHA256-*  | tail -2
 -rw-r----- 2 root root 16130992 Feb  8  2019
 SHA256-f6810ff0245807455347d88a1a0d7eaf29368e64188e7c1766b64c0cc143570e
 -rw-r----- 2 root root   130308 Feb  8  2019
 SHA256-c30b262677ed796d892b888f7f417690ca55dd83bf17fa421fdd32438ca2203a
 }}}

 It also seems that the database grew quite a bit -- doubled in size -- in
 the last few months, according to the backup sizes:

 {{{
   800.9 MiB [##########]  20200415-190301-torcrm_prod
   670.4 MiB [########  ]  20200109-190301-torcrm_prod
   398.9 MiB [####      ]  20191002-190301-torcrm_prod
 }}}

 Obviously, a database server running out of disk space is an...
 undesirable condition, to say the least. :) Should we expand the disk
 usage for that server (which I would rather avoid doing during the
 weekend) or is there something you can do on your end to clean stuff up?

 Alternatively, maybe we should improve the backup system here so it
 doesn't take up twice as much disk space as the production server. Or,
 even better, not be on the same partition as the prod...

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/34098>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online