Category Archives: Sysadmin

Fixing php5-fpm and Apache hanging with WordPress

I had issues with Apache periodically hanging (failing to deliver a response body to any requests) on all my vhosts. This turned out to be solved by restarting php5-fpm. I enabled the slowlog in php5-fpm to try and find out which scripts were stalling:

sudo mkdir -p /var/log/php5-fpm
sudo vim /etc/php5/fpm/pool.d/www.conf


; The log file for slow requests
; Default Value: not set
; Note: slowlog is mandatory if request_slowlog_timeout is set
slowlog = /var/log/php5-fpm/$pool.log.slow

; The timeout for serving a single request after which a PHP backtrace will be
; dumped to the 'slowlog' file. A value of '0s' means 'off'.
; Available units: s(econds)(default), m(inutes), h(ours), or d(ays)
; Default Value: 0
request_slowlog_timeout = 5s

After a day or so I read the logs and found lots of slow requests to xmlrpc.php for WordPress vhosts.

A crude but effective solution is to block requests to the XML-RPC and Trackback APIs. These features are sometimes targeted by bots for brute force login attempts. I do not use them so I don’t mind disabling them entirely.

Edit your Apache vhost configuration (or .htaccess if you don’t have access to this):

<FilesMatch "^(xmlrpc\.php|wp-trackback\.php)">
Order Deny,Allow
Deny from all
#Allow from x.x.x.x
</FilesMatch>

I noticed considerably lower latency when serving requests to PHP pages after this change.

Apache 2.2 Websocket Proxying on Ubuntu with mod_proxy_wstunnel

Until recently, I was running a Node.js web application using Websockets with socket.io alongside an Apache web server. I wanted Apache to run on port 80 and serve both sites using name-based virtual hosting. However, Apache httpd mod_proxy couldn’t proxy Websockets correctly.

I discovered that there is a new mod_proxy_wstunnel module in the Apache httpd source trunk in an article describing how to backport mod_proxy_wstunnel to Apache 2.4 or 2.2. I figured out the specific steps for doing this on Ubuntu (tested on 11.10 with Apache httpd 2.2.20). I wanted to add them as a comment, but they were disabled on that blog.

Here are the steps, see the original blog post of the patch author for more information:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
# Check apache version (should be 2.2.20 as of writing, if not adjust the next step)
dpkg -s apache2
 
# Checkout apache source
svn checkout http://svn.apache.org/repos/asf/httpd/httpd/tags/2.2.20/ httpd-2.2.20
 
# Get patch and apply it
wget http://cafarelli.fr/gentoo/apache-2.2.24-wstunnel.patch
cd httpd-2.2.20
patch -p1 < ../apache-2.2.24-wstunnel.patch
 
# Build Apache 
svn co http://svn.apache.org/repos/asf/apr/apr/branches/1.4.x srclib/apr
svn co http://svn.apache.org/repos/asf/apr/apr-util/branches/1.3.x srclib/apr-util
./buildconf # EDIT: Some commenters noted that buildconf should be run before the configure
./configure --enable-proxy=shared --enable-proxy_wstunnel=shared
make
 
# Copy the module and recompiled mod_proxy (for new symbols) to the ubuntu apache installation and update the permissions to match the other modules
sudo cp modules/proxy/.libs/mod_proxy{_wstunnel,}.so /usr/lib/apache2/modules/
sudo chmod 644 /usr/lib/apache2/modules/mod_proxy{_wstunnel,}.so
echo -e "# Depends: proxy\nLoadModule proxy_wstunnel_module /usr/lib/apache2/modules/mod_proxy_wstunnel.so" | sudo tee -a /etc/apache2/mods-available/proxy_wstunnel.load
 
# Enable the module (also make any configuration changes you need)
sudo a2enmod proxy_wstunnel
sudo service apache2 restart

Ubuntu: Running a command in GNU screen on boot as another user

Add to /etc/rc.local before the ‘exit 0’ line:

mkdir /var/run/screen
chmod 775 /var/run/screen
chgrp utmp /var/run/screen
su USER -c 'screen -m -d -S SCREENNAME bash -c "COMMAND"'

Replace USER with the user you want the command to run as. SCREENNAME with any name you like, and COMMAND with the actual command to run in the screen.

The first three lines were necessary to work around a bug in screen on Ubuntu.

Accessing Backups of Truecrypted Windows System Drives Under Linux

Unfortunately I lost this article when I lost the database for my old blog. I noticed the link is still generating quite a bit of interest from the superuser question and I have been emailed about the page. I am currently away so don’t have time to try and completely rewrite the article, but here is what I said to the person who emailed me as a rough guide:

Roughly, I ended up creating an image of the disk with dd in linux and then created a VMWare Virtual Machine. I then modified the configuration file (<something>.vmdk IIRC) for the virtual machine to make the virtual disk descriptor map to the dd image I’d created (information about how to do this is on google, you just want to ‘redirect’ the disk descriptor to go to your image rather than the empty container vmware creates). Sometimes you have to fiddle with the SCSI/IDE controller type to get the virtual machine to boot without blue-screening.

Hope this helps.

Run task on drive connection or disconnection in Windows

Description

This script watches for Windows drive letter connection or disconnection events and allows scheduling of tasks to run accordingly. For example, this is useful if you want to run a program to synchronise the contents of a USB pen drive automatically when it is inserted.

Tasks can be configured in a human-readable JSON format configuration file. It installs itself as a service and is very, very alpha.

Requirements

  • The script should work in Python >= 2.6 and requires pywin32.
  • It has been tested on Windows 7 x64 only.

Source code repository: https://github.com/inversion/drive-letter-watcher

Auto-update Ubuntu server with email notifications

It’s useful to have a script to keep the packages on your server up to date. On the desktop there is a GUI auto-update tool which is active by default. The caveat is that you don’t want updates to break your server without you realising.

This script is adapted slightly from the Ubuntu wiki to do a ‘safe-upgrade‘ and to mail directly rather than using an external SMTP server. It also tails the output of the update since it’s pretty long. Use at your own risk etc., script is below. Copy to /etc/cron.weekly and chmod +x.

#!/bin/bash
#
# use aptitude to automatically install updates. log and email any
# changes.
#
 
#
# variables to change
#
 
# address to send results to
MAILTO=you@example.com
 
#
# script is below here (do not change)
#
 
tmpfile=$(mktemp)
 
#
# actually run aptitude to do the updates, logging its output
#
 
echo -e "aptitude update\r\n..." >> ${tmpfile}
aptitude update | tail >> ${tmpfile} 2>&1
echo "" >> ${tmpfile}
echo "aptitude safe-upgrade" >> ${tmpfile}
aptitude -y safe-upgrade >> ${tmpfile} 2>&1
echo "" >> ${tmpfile}
echo "aptitude clean" >> ${tmpfile}
aptitude clean >> ${tmpfile} 2>&1
 
#
# i get a lot of escaped new lines in my output. so the following
# removes them. this could be greatly improved
 
tmpfile2=$(mktemp)
cat ${tmpfile} | sed 's/\r\r/\n/g'|sed 's/\r//g' > ${tmpfile2}
mv ${tmpfile2} ${tmpfile}
 
#
# now send the email (and ignore output)
#
mail -s "Aptitude Upgrade $(date)" ${MAILTO} < ${tmpfile} &> /dev/null
 
#
# and remove temp files
#
 
rm -f ${tmpfile}