Serving Mercurial on OpenBSD with Gunicorn

One and other thing lead me to hosting my mercurial repositories on an OpenBSD VPS.

Here’s a bit of memo on how I did it.

A dedicated user needs to be created. I call it hg which is as generic as it can be. Then I create home directory of /home/hg, and set its $HOME to /home/hg/repos. Wait.

It’s so I can just push to ssh://hg@hg.myconan.net/reponame and not having to specify additional namespace. The /home/hg itself needs to contain some other files so that’s just how it ended up. I can probably put the extra files somewhere else but it seems simpler to have them all in single directory tree. Now I write it maybe I should’ve made it at /var/hg/root or something like that.

Well, it’s done deal.

I also made ~hg/.ssh/authorized_keys file and fill it with my key. Again, so I can push to it.

With that done, next is installing the required packages:

  • py3-gunicorn
  • supervisor
  • mercurial
  • nginx
  • certbot

Refer to this post on configuring the certbot. It worked so well and requires barely any maintenance so far.

As for gunicorn, I made /home/hg/hgweb directory which contains following files:

  • gunicorn.conf.py
  • hgweb.config
  • hgweb.py

Gunicorn config is pretty simple:

bind = 'unix:/home/hg/gunicorn.sock'
workers = 4
accesslog = '-'
errorlog = '-'
timeout = 30

Nothing fancy, and there’s no worker_class because none of the supported workers (apart of sync) seem to be supported under OpenBSD. Should be fine as it’s just for my personal use.

As for hgweb.py, it’s copied from /usr/local/share/mercurial/hgweb.cgi with config path adjusted to local hgweb.config and removed references to wsgicgi (import and .launch) as I’m using Gunicorn, not CGI.

hgweb.config itself on the other hand, it’s also pretty basic:

[paths]
/ = /home/hg/repos/*

[web]
baseurl = https://hg.myconan.net/
contact = nanaya
staticurl = /static

All those done, last part to start serving with Gunicorn is updating /etc/supervisord.conf. There’s an example in their official docs and I made some adjustments:

[program:hg]
command=/usr/local/bin/gunicorn --config=/home/hg/hgweb/gunicorn.conf.py hgweb:application
user=hg
directory=/home/hg/hgweb
stopsignal=INT
environment=PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin"
stdout_logfile=/var/log/supervisord/%(program_name)s-%(process_num)s.stdout.log
stderr_logfile=/var/log/supervisord/%(program_name)s-%(process_num)s.stderr.log

Mainly for having non-random log file path.

Create log directory with mkdir -p /var/log/supervisord, enable the service with rcctl enable supervisord, and hope it works.

Oh and chown hg:www /home/hg && chmod 710 /home/hg for basic file permissions. Oh and hg:hg owner and 700 permission for repos directory itself.

And lastly nginx:

server {
    listen 443;
    listen [::]:443;
    server_name hg.myconan.net;

    access_log /var/log/nginx/hg.myconan.net-access.log;
    error_log /var/log/nginx/hg.myconan.net-error.log;

    ssl_certificate certs/hg.myconan.net/fullchain.pem;
    ssl_certificate_key certs/hg.myconan.net/privkey.pem;
    ssl_trusted_certificate certs/hg.myconan.net/chain.pem;

    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;

    root /nonexistent;

    location = /favicon.ico {
        return 204;
    }

    location = /robots.txt {
        return 204;
    }

    location / {
        proxy_pass http://unix:/home/hg/tmp/gunicorn.sock;
        proxy_set_header Client-Ip $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header X-Forwarded-Port $remote_port;
        proxy_set_header X-Forwarded-Proto $scheme;

        limit_except GET HEAD {
            deny all;
        }
    }

    location /static/ {
        root /usr/local/lib/python3.8/site-packages/mercurial/templates;
    }
}

Nothing fancy either, just a basic https proxying setup with no write permission as I don’t want to setup http auth and only push using ssh.

/static/ directory is served directly to the installation’s templates directory. Subdirectory name already matches so no alias or symlink is needed.

rcctl enable nginx and don’t forget to rotate the log files by adding the two specified files to /etc/newsyslog.conf.

…that’s kinda long.

Letsencrypt, cavemen edition

Just had to do some letsencrypt setup in some servers so I figured I should write down what I did so I can just check this page again instead of digging how I did it previously.

Requirements:

  • nginx
  • certbot

This assumes the server only serves https and redirects all http traffic. Adjust as needed otherwise.

Full nginx SSL/TLS config not included.

First add this config to nginx to handle verification:

# part of default port 80 config block
location /.well-known/acme-challenge/ {
    root /var/www/certbot;
}

And then create the directory (I’m not actually sure if needed):

# mkdir -p /var/www/certbot

Make the first cert because I’m too lazy to ensure the config directory is setup correctly:

# certbot certonly --webroot -w /var/www/certbot -d DOMAIN_NAME_GOES_HERE --keep --agree-tos --email SOME_KIND_OF@EMAIL_ADDRESS --no-eff-email

At this step, the certificate and all should have been properly generated.

Then use it in nginx configuration, the relevant server block:

ssl_certificate /etc/letsencrypt/live/DOMAIN_NAME_GOES_HERE/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/DOMAIN_NAME_GOES_HERE/privkey.pem;
ssl_trusted_certificate /etc/letsencrypt/live/DOMAIN_NAME_GOES_HERE/chain.pem;

If the full path seems too long, symlink it to nginx config base directory or something.

Update certbot CLI configuration located at /etc/letsencrypt/cli.ini:

rsa-key-size = 4096
text = True
authenticator = webroot
webroot-path = /var/www/certbot

To add more certificates:

# certbot certonly -d ANOTHER_DOMAIN

Don’t forget to update nginx configuration as before.

Since the certificate needs renewal periodically, create this simple script:

#!/bin/sh
# I personally put this in /root/bin/refresh-ssl-certbot

/usr/bin/certbot renew
/path/to/sbin/nginx -s reload

Make executable, etc. Try it to make sure it runs properly.

Then add it crontab. I usually do it weekly.

And done.

There might be smarter way using certbot’s nginx plugin or something but I haven’t bothered reading its documentation and initially this was just a stopgap switching from acme-client which is way simpler but stopped working for me few months ago.

IP address checker

Random idea I came up when reading nginx mailing list. A very simple way to set up external IP address checker using nginx on a remote server.

location = /ip {
    default_type text/plain;
    types { }

    return 200 $remote_addr\n;
}

Accessing /ip will then return the current external IP address. A more fancy output like JSON is possible as well.

dirlist-php

A few years ago I wrote a php script to provide better autoindex within nginx. I used it for quite a long period until I rewrote it in Ruby/Sinatra. But then I figured the setup for it is overly complicated just for simple task. And takes additional memory. I always have php-fpm running anyway so it’s free.

And so I decided to take up the old php script and fix it up. Unfortunately there isn’t other language as easy to setup for web as php which is why I fixed it instead of rewriting in some other languages (or keeping it ruby). The “fixed” version is still pretty much imperative-style but hey, it works.

Only tested with nginx.

PSA: Never trust external X-Forwarded-For

For god knows how long, proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; is one of the line usually included in nginx config snippet for proxying to a unicorn (Rails) backend.

…which is something you should never do unless you have another load balancer in front of the nginx being configured.

That line basically tells nginx to append $remote_addr to whatever X-Forwarded-For value currently set. It is only useful when your nginx is behind other load balancer which set up its own (hopefully correctly) X-Forwarded-For. It should be set explicitly to $remote_addr for any external-facing proxy. Otherwise fun things will happen.

An Attempt to Update moebooru Engine

If you didn’t know, the current moebooru running on oreno.imouto is using ancient version of many things. It also uses a custom lighty module (mod_zipfile) which doesn’t seem to be available anywhere.

I’ve updated it with latest Rails 2.x and made it compatible with nginx. Mostly. You can see it running here.

The plans:

  • Upgrade to Ruby 1.9.
  • Update all plugins.
  • Update anything deprecated.
  • Migrate to Bundler.
  • Use RMagick instead of custom ruby-gd plugin.
  • Use RMagick instead of calling jhead binary.
  • And more!

We’ll see if I can actually finish this one. Grab the source here. Yeah, I’m using Mercurial for a Rails project.

nginx – gzip all the text

During my migration to other server, I recreated some of my configs and enabled gzip compression for most file types. Here’s the relevant config:

gzip on;
gzip_vary on;
gzip_disable "msie6";
gzip_comp_level 6;
gzip_proxied expired no-cache no-store private auth;
gzip_types text/plain text/xml application/xml application/json application/x-javascript text/javascript text/css;

It should cover most text-based content one will ever serve over the web. Probably.

nginx/php single config for SSL and non-SSL connection

This morning I noticed I haven’t upgraded WordPress MU Domain Mapping plugin to the latest version. It supposedly brings better SSL support. And after upgrading I couldn’t log in to my mapped domain blogs (e.g. this blog). Wasn’t it a great way to start my morning?

After some digging, I found out the problem was because I don’t have one PHP(?) parameter – HTTPS – passed properly. It should set to True whenever one is using SSL connection otherwise there’s no way the PHP process can know if the connection is secure or not. Previous version of WPMUDM have a bug in which skips SSL check but in turn enables using HTTPS even without such parameter. Decided it’s my fault (I believe it would completely breaks phpMyAdmin), adding the parameter then I did.

But it’s not that simple: I’m using unified config for both my SSL and non-SSL connection’s PHP include. Splitting the config would make the duplication worse (it’s already relatively bad as it is) so that’s not an option. Using the evil if is also not a solution since it doesn’t support setting fastcgi_param inside it.

Then the solution hit me. The map module – a module specifically made for things like this and to avoid usage of if. I tested it and indeed worked as expected.

Here be the config:

...
http {
  ...
  map $scheme $fastcgi_https {
    https 1;
    default 0;
  }
  ...
  server {
    ...
    location ~ .php$ {
      ...
      fastcgi_param HTTPS $fastcgi_https;
    }
...

And WordPress MU Domain Mapping is now happy.

Update 2012-02-20: nginx version 1.1.11 and up now have $https variable. No need to have that map anymore.

WordPress Multisite with nginx (subdomain/wildcard domain ver.)

WordPress Multisite, previously known as WordPress MU (Multi User), is an application which allows hosting multiple WordPress blogs with just one installation. Instead of creating copies of WordPress for each users’ blogs, one can use one installation of Multisite to be used by multiple users, each with their own blogs. Personal/custom domain is also possible as used for this blog (this blog’s master site is genshiken-itb.org). Too bad, the official documentation only provided guide for installing on Apache. If you haven’t known, I usually avoid Apache – I simply more proficient with nginx. Of course, this blog is also running on nginx therefore it’s perfectly possible to run WordPress Multisite on nginx.

At any rate, reading the official documentation is still a must and this post will only cover the nginx version of Apache-specific parts (namely Apache Virtual Hosts and Mod Rewrite and .htaccess and Mod Rewrite) and only for subdomain install. Subdirectory one will or will not follow some time later.

Assuming you have working nginx and php-cgi (with process manager like php-fpm or supervisord), for starter you’ll want to create a specific file for this WP install. Let’s say this file named app-wordpress.conf. Obviously you have put WordPress installation somewhere in your server. In this example I put the files in /srv/http/genshiken-itb.org/.php/wordpress. Its content:

client_max_body_size 100m;
root /srv/http/genshiken-itb.org;
location /. { return 404; }
location / {
  root /srv/http/genshiken-itb.org/.php/wordpress;
  index index.php;
  try_files $uri $uri/ /index.php?$args;
  rewrite ^/files/(.*) /wp-includes/ms-files.php?file=$1;
  location ~ .php$ {
    try_files $uri =404;
    fastcgi_pass unix:/tmp/php-genshiken.sock;
    fastcgi_read_timeout 600;
    fastcgi_send_timeout 600;
    fastcgi_index index.php;
    fastcgi_param SCRIPT_FILENAME $request_filename;
    include fastcgi_params;
  }
}

Simple enough. Actually, it’s exact same with normal WordPress install except one extra line:

rewrite ^/files/(.*) /wp-includes/ms-files.php?file=$1;

And you’re set. Note that I set fastcgi timeouts higher than default to work around the slow performance of Amazon EC2 Micro Instance. Should only needed on network upgrade and massive blog import.

Anyway, in your main nginx.conf file, put:

server {
  listen 80;
  server_name *.genshiken-itb.org;
  include app-wordpress.conf;
 }

In proper place. The usefulness of separate file for WordPress configuration will become apparent once you want to tweak performance for some blogs. I’ll explain that later if I feel like to.

no-www for nginx

If you happen to be in no-www camp and want to redirect people accessing www.whateverdomain.com to the no-www version but have lots of domain, instead of writing one by one and you’re not keen in using config generator (I’m not), you can use this:

server {
  listen 80;
  listen [::]:80 ipv6only=on;
  server_name ~^www.(?<domain>.+)$;
  rewrite ^ $scheme://$domain$request_uri? permanent;
  access_log /var/log/nginx/access-no_www.log;
}

Remove listen [::]:80 ipv6only=on; if you’re not using IPv6 and adjust the log file path to wherever you want (or just turn off or remove it altogether).

Note that this trick doesn’t work well with HTTPS/SSL domains since you’ll get big fat warning about incorrect domain name in certificate or about self-signed certificate if you’re using wildcard one.

Forcing SSL with nginx and Apache

Not really difficult but I guess it would be useful for some people.

First one is for nginx: create a file called `force-ssl.conf` and put in nginx’s config directory (check by `nginx -V`). And its content is:

if ($scheme = http) {
rewrite ^ https://$host$request_uri? permanent;
}

Include this file (by `include force-ssl.conf;`) in any `location … { }` block you want to force SSL on.

As for Apache, we can do it by using the usual `.htaccess` (and put in corresponding directories):

RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI}

The nginx rule is written by me, Apache rule is written by [someone on internet](http://joseph.randomnetworks.com/2004/07/22/redirect-to-ssl-using-apaches-htaccess/) with minor fix on skipping regex capture (`(.*)` replaced by `^`).

nginx/php 404 proper handler

If you find “No file specified” error message disturbing (as I did), here’s the configuration to remove it!

location ~ .php$ {
try_files $uri =404;
include /etc/nginx/fastcgi_params;
fastcgi_pass 127.0.0.1:55555;
fastcgi_index index.php;
fastcgi_param  SCRIPT_FILENAME    $request_filename;
}

Don’t use `if (-f $request_filename)` – it won’t work and if is evil. Also the `try_files` will ensure that the file actually exists – effectively disabling possible vulnerability with public file upload on certain conditions.

_Last update 2011-07-10 20:16: working version, markdown-fied_

nginx links

Here are the links I find useful:

* [Why nginx](http://hostingfu.com/article/nginx-vs-lighttpd-for-a-small-vps)
* [WordPress with nginx](http://elasticdog.com/2008/02/howto-install-wordpress-on-nginx/)

Note that the latter link should not be followed anymore.

_Last update 2011-07-10 20:24: markdown-fied_