# Mass hosting
Notes on mass hosting, servers in excess of 500+ sites.
# Enabling data center mode
Data center mode opens up a few additional features in ApisCP intended for mass hosting including per-file InnoDB tables, remote database access, Postfix SMTP banner identification, and centralized postscreen usage.
# Enable data center mode cpcmd scope:set cp.bootstrapper data_center_mode true upcp -b
# Raising inotify watchers
Each site may run its own crond process when crontab,permit=1 via Dev > Task Scheduler. Each crond service attaches an inotify watcher to each spool file in /var/spool/cron to detect changes and reload crond accordingly.
CentOS sets a default of 128, which if hit will generate spurious "Too many open files" messages (syserr EMFILE). This can be confirmed by trying to start any service:
systemctl restart atd.
ApisCP raises this limit to 256, but may need to be higher depending on needs.
sysctl -w fs.inotify.max_user_instances=512
Settings may be saved by creating a file in /etc/sysctl.d, e.g.
echo "fs.inotify.max_user_instances=512" > /etc/sysctl.d/Zinotify.conf
Any file lexicographically greater than "apnscp.conf" will override these settings.
# Table definition cache/prepared statements
Prepared statements may fail with errno 1615: "Prepared statement needs to be re-prepared". This occurs when a significant number of tables exist in the data dictionary.
From the command-line, run
mysql SET GLOBAL table_definition_cache=16384;
If the error resolves, this is due to the table definition (opens new window) limit of 4096 being reached. Changes may be made permanent by adding
table_definition_cache=16384 under the [mysqld] section in
/etc/my.cnf.d/server.cnf or any file lexicographically higher than "apnscp.conf".
# fail2ban extended startup
Hosting more sites creates more opportunity for indiscriminate attacks. Rampart continuously blocks threats, logging to fail2ban to replay its activity if the server were to reboot. "recidive" is a long-term ban list for repeat offenders that accumulates significant entries over its 10 day duration, sometimes up to 10,000 while other short-term ban lists may have a few dozen entries at a time. Whenever fail2ban starts (or restarts), this ban list is recreated from entries in
/var/log/fail2ban.log that can create additional resource contention between CPU/disk that amplifies a thundering herd problem among other services, including per-site PHP-FPM pools that can number in the several hundreds.
fail2ban can be configured to tail
/var/log/fail2ban.log on startup, that is to say instead of replaying the entire log just monitor for new entries. Instead of parsing 100k+ lines only new lines are parsed that can reduce the cumulative burden on startup.
Existing bans in the database will still be applied on startup, but these bans are replayed with less rigor that may create omissions in the "recidive" ban list.
cpcmd scope:set cp.bootstrapper f2b_recidive_tail true upcp -sb fail2ban/configure-jails