sysadmin

NCDU-fu

Published by Anonymous (not verified) on Thu, 19/10/2017 - 8:55pm in

Tags 

sysadmin

Sometimes it feels good to have some meager sysadmin competencies, such as knowing how to quickly identity where large files are in a particular hosting account. This issue comes up from time-to-time when someone discovers all their storage space has been eaten up, but they are not sure where and why.  Often this is a symptom of a larger problem, such as an error_log run out of control which suggests bad process for a particular application, etc. That was the case on a ticket this morning, and luckily I knew the NCDU command. What is NCDU, you ask?

Ncdu is a disk usage analyzer with an ncurses interface. It is designed to find space hogs on a remote server where you don’t have an entire graphical setup available, but it is a useful tool even on regular desktop systems. Ncdu aims to be fast, simple and easy to use, and should be able to run in any minimal POSIX-like environment with ncurses installed.

In other words, a script for a remote server that finds big files. You can install it on your server, and then run it by navigating in command line to the offending account, which on our cPanel servers would live at /home/offendingaccount and running the command NCDU. After that, it will list all the directories and their sizes, followed by files.  You can then locate the directory with the largest file usage, and then change to that directory and run NCDU again until you find the offending file. In the example this morning, it was a 6 GB error_log in a directory running WordPress, easy fix to clean out space, and a good heads up things in that account need to be checked for a bad plugins, theme, etc.

The life of a Reclaimer is always intense

Users and groups in Debian: getting it right

Published by Matthew Davidson on Thu, 19/10/2017 - 1:15pm in

So ideally when I set up a new computer, I want all the users I trust — including, by necessity and regrettably, myself — to be in the staff group, and all the files they create to be by default writable by anyone in that group. This ought to be easy, and in fact now is, but has changed repeatedly over the decades I've been using Debian GNU/Linux, so I can never remember how it's done, hence this note.

You will need to do all this as root, and to be on the safe side, make sure any user(s) you want to put into the staff group are not currently logged in, as files and directories in the affected home directories will be reassigned to the group, which (I guess) won't work for any currently opened by a running process.

If you enable the pam_umask PAM module, you will only need to configure group-writability once, and it will work regardless of whether you're logging in locally, SSHing, or whatever. As root, edit /etc/pam.d/common-session to include this line:

session optional        pam_umask.so

Then edit the umask line in /etc/login.defs like so:

UMASK 002

If yours isn't a fresh Debian install, the umask setting may already have been overridden in one or more of:

  • /etc/profile
  • /etc/bash.bashrc
  • ~/.profile
  • ~/.bashrc

If so, delete or comment out where necessary. (Source)

Adding a user to the staff group is:

usermod -a -G staff myusername

Making staff the user's primary group — the one which by default newly created files and directories are owned by — is just:

usermod -g staff myusername

Too easy.

Saturday, 7 October 2017 - 6:42pm

Published by Matthew Davidson on Sat, 07/10/2017 - 6:42pm in

I should never reboot my computer.

I am so out of touch that I didn't realise that a new version of Debian came out in June. "Splendid!", I thought. So:

# apt-get update
# apt-get dist-upgrade

… then off for a walk while two gigabytes downloaded (really must get rid of all those first-person shooters that are anyway far too violent for a gentleman of my advanced years).

Get through the upgrade, reboot the computer, and my USB WiFi dongle doesn't work. Here's how to diagnose/fix:

# lsusb
Bus 002 Device 002: ID 8087:8000 Intel Corp. 
[…]
Bus 003 Device 003: ID 045e:00cb Microsoft Corp. Basic Optical Mouse v2.0
Bus 003 Device 002: ID 413c:2003 Dell Computer Corp. Keyboard
Bus 003 Device 007: ID 0bda:8178 Realtek Semiconductor Corp. RTL8192CU 802.11n WLAN Adapter
[…]

Yes, I use a Microsoft mouse. Microsoft branded peripherals have generally been pretty darn good. I think this mouse is at least ten years old, and it's as good as the day I bought it. So now I know the WiFi chipset. I go to the Debian Wiki WiFi page, and find that I need the rtl8192cu driver, which is in the (non-free) firmware-realtek package, which is of course already installed because the blasted thing used to work. So now it's just a matter of:

# modprobe rtl8192cu

…and we're back in business. For good measure, I added rtl8192cu to the /etc/modules file, so that maybe I'll survive the next reboot unscathed. Not that I will be rebooting any time soon.

Set UMW Blogs Right to Ludicrous Speed

Published by Anonymous (not verified) on Sun, 09/07/2017 - 1:34am in

Tags 

sysadmin

UMW Blogs was feeling its age this Spring (it officially turned ten this month—crazy!) and we got a few reports from the folks at DTLT that performance was increasingly becoming an issue. Since 2014 the site had been hosted on AWS (Tim wrote-up the details of that move here) with a distributed setup storing uploads on S3, the databases on RDS, and running core files through an EC2 instance. That was a huge jump for us back then into the Cloud, and the performance jump was significant. AWS has proven quite stable over the last two years, but it’s never been cheap—running UMW Blogs back then cost $500+ a month, and I’m sure the prices haven’t dropped significantly since.


While not as active as back in the day, the consistent traffic on UMW Blogs remains fascinating to me.

One theory Tim floated around the issues we were running into this Spring was that having all the pieces separate, namely the RDS database and Ec2 instance, may be contributing to the slowdown. We couldn’t test that theory during the semester though, so UMW Blogs limped through the Spring. But this week we were able to test that theory by trying a new setup that has the site running about as fast as I can ever remember. Rather than the distributed server setup we had on AWS, this time around we brought all the pieces back together under the same roof on Digital Ocean. We setup an Ubuntu 16.04 instance and then installed a LEMP stack (Apache is replaced with Nginx). After that, we installed WordPress on the LEMP server and we had basic shell of a site to start importing the files and databases from AWS. We made sure we were running PHP 7.1 given its speed, and also installed the http accelerator Varnish. This is a very similar to the setup to what we ran for VCU’s Rampages, and given the success we’ve had there, we had a good feeling about going this route. We were right. 

We spent the earlier part of the week syncing files over to the new server and making sure we had everything on the new server before shutting UMW Blogs down on Thursday and Friday to pull over the databases (UMW Blogs is sharded into 16 databases). Turns out we only needed until 5 Pm on Thursday before the site was back online. 

It was a fairly smooth migration, the only real issue we ran into during the migration was figuring out why we were getting a database connection error Thursday morning after importing the WPMU_global and VIP databases to the new server.  We got the same error when pointing the database from RDS at the new server, so we were stumped. Tim, wondering if it might be a PHP error, updated PHP 7 to 7.1 and that fixed it—there are few better troubleshooters I have ever met when it comes to this stuff! 

The only issue we’ve had since the site has gone back online were reports of uploaded images not resolving. Turns out the plugin sending all files to S3 after the 2014 move was re-writing the URLs,  what’s more they were being pulled from a directory other than blogs.dir on S3 (they were being pulled from s3://umwblogs.org/sites to be exact). Once realizing this, Tim turned the S3 plugin back on and all uploaded files worked, which was a huge relief. So, at least for right now, the only piece not on Digital Ocean are the uploaded files that remain in Amazon’s S3.

I’m sure Tim can and will fill in on any details I may have missed or misrepresented, he managed this migration like a champ—as usual. I did the early setup of the server and the LEMP environment, as well as syncing files from S3 (which turned out to be unnecessary) using S3cmd sync. Tim synced the core files, exported and imported the databases, got Varnish running smoothly (I failed on that one), and more generally cleaned up any messes I made during the initial setup.

It was a fun project, and part of that was setting UMW Blogs to ludicrous speed after a decade of service to the UMW community. Fact is, there is still a lot of amazing work on that site, and the fact that it had almost 3 million pageviews served up to 1.7 million users over the last year alone suggests it’s not dead yet. But costs and sustainability for keeping a site like UMW Blogs around is always a question, so being able to make the site significantly faster for less than half the price of the previous server infrastructure is something to smile about. Also, I get to blog about UMW Blogs again, which never sucks.

Note to self

Published by Matthew Davidson on Mon, 08/02/2016 - 3:05pm in

I forget this every time I go to upgrade Drupal, because it's so simple, and spend an hour trying to make absolutely sure I have it right and haven't missed anything. So assuming you're deploying Drupal with git (with the contents of sites/ untracked, presumably), all you have to do is:

git fetch
git rebase origin/7.x
drush @sites updb

Optionally, you can do a git hard reset to the latest tagged release.