Page MenuHomePhabricator

Public | Fix Website Swap File
Closed, ResolvedPublic

Description

The website ground to a halt yesterday afternoon while sending newsletters. The cause of exhaustion of virtual memory due to dirty MySQL InnoDB buffers, I think. The swap space was exhausted, anyway, and the largest consumer was MySQL and there errors in the MariaDB.log:

2018-11-16 16:18:23 140278728701696 [Warning] Aborted connection 121028 to db: 'unconnected' user: 'phabricator' host: 'localhost' (Got an e
rror writing communication packets)
2018-11-16 16:18:27 140279611279104 [Note] InnoDB: page_cleaner: 1000ms intended loop took 104399ms. The settings might not be optimal. (flu
shed=0 and evicted=0, during the time.)
2018-11-16 16:18:43 140279611279104 [Note] InnoDB: page_cleaner: 1000ms intended loop took 13645ms. The settings might not be optimal. (flus
hed=0 and evicted=0, during the time.)
2018-11-16 16:18:54 140278716106496 [ERROR] Invalid (old?) table or database name '.rocksdb'
2018-11-16 16:19:03 140279611279104 [Note] InnoDB: page_cleaner: 1000ms intended loop took 14635ms. The settings might not be optimal. (flus
hed=0 and evicted=0, during the time.)
2018-11-16 16:19:46 140278726244096 [Warning] Aborted connection 121196 to db: 'phabricator_search' user: 'phabricator' host: 'localhost' (G
ot an error writing communication packets)
2018-11-16 16:20:03 140278715184896 [Warning] Aborted connection 121197 to db: 'phabricator_cache' user: 'phabricator' host: 'localhost' (Go
t an error writing communication packets)
2018-11-16 16:20:12 140278718871296 [Warning] Aborted connection 52883 to db: 'c0karenware' user: 'c0karenware' host: 'localhost' (Got an er
ror reading communication packets)
2018-11-16 16:20:16 140279611279104 [Note] InnoDB: page_cleaner: 1000ms intended loop took 71859ms. The settings might not be optimal. (flus
hed=0 and evicted=0, during the time.)
2018-11-16 16:20:34 140279206901504 [Warning] Aborted connection 121324 to db: 'c5aws' user: 'c5aws' host: 'localhost' (Got an error reading
 communication packets)
2018-11-16 16:20:34 140278724093696 [Warning] Aborted connection 120977 to db: 'c5aws' user: 'c5aws' host: 'localhost' (Got an error reading
 communication packets)
2018-11-16 16:20:35 140279611279104 [Note] InnoDB: page_cleaner: 1000ms intended loop took 11663ms. The settings might not be optimal. (flus
hed=0 and evicted=0, during the time.)

Event Timeline

Joe created this task.Nov 17 2018, 8:14 AM
Joe created this object in space S5 Public.
Joe triaged this task as Unbreak Now! priority.
Joe created this object with visibility "Public (No Login Required)".
Joe closed this task as Resolved.Nov 17 2018, 8:25 AM

Created 3 GB swap file to replace the 1 GB one, and put it on the web volume because it's comparatively idle while the database file are being hit on the system volume.