Daniel Lockyer

Server monitoring

2018/01/25

This post is about how we monitor Nomad List, Remote OK and others. Over the years, Pieter and I have built up a set of tools to run and inform us when the sites may be experiencing an issue.

Cron Dashboard

We use cron jobs for almost every task, from updating data sources to processing images. Right now we have 81 jobs, with some running multiple times per hour. The cron dashboard I made is powered by a simple bash wrapper script that records metrics and sends the data to a PHP script for processing. This has saved our backs so many times. It also allows us to see which scripts are taking a long time, so I can then go and optimize them.

ngxtop

ngxtop allows us to monitor Nginx logs and perform queries against the data. Using this tool, we’ve easily been able to find people scraping our site, or links that are regularly hit but tend to 404 or 500.

PHP Access Logs

One of my favourite tricks is to use the PHP access logs to monitor the script time of server-side page loads. The following bash command looks at the last 50 entries and sorts them by runtime cost.

sudo tail -n 50 /var/log/php7.1-fpm.access.log | sort -n | cut -c1-$(tput cols)

A typical entry may be something like:

50.886 2048 78.61% remoteok.io /srv/http/remoteok.io/app/feed.php?type=rss

The first column is script time in milliseconds. The second is memory usage. The third is CPU usage.

This is useful because it allows you to see which requests you may want to turn into a static HTML page and avoid PHP for each load. During the launch of Hoodmaps we saw all the page embeds turning up in the output so we switched to static pages and reduced our server load.

PHP-FPM Status

PHP-FPM actually comes with a status page built in. Once enabled, it allows you to see basic information about the process. If you add ‘?full’ to the URL, it lists all the child processes and statistics about them. If you append ‘&json’ to the URL, it outputs it in JSON format. I’m currently working on a dashboard to graph this data.

UptimeRobot

UptimeRobot is a service we use to check for availability and keywords on site pages. It can then inform us via Twitter DM, email or SMS if there is an issue.

Others

Even things like htop and tcpdump can be useful in monitoring and diagnosing issues.

In the end, it’s important to use what works for you. We like to try and stick to tools and services we control so we don’t rely on third-party services.

If you’re looking to set up monitoring for your server, email me at [email protected]!

Share on Twitter