Serving Mercurial on OpenBSD with Gunicorn

One and other thing lead me to hosting my mercurial repositories on an OpenBSD VPS.

Here’s a bit of memo on how I did it.

A dedicated user needs to be created. I call it hg which is as generic as it can be. Then I create home directory of /home/hg, and set its $HOME to /home/hg/repos. Wait.

It’s so I can just push to ssh://hg@hg.myconan.net/reponame and not having to specify additional namespace. The /home/hg itself needs to contain some other files so that’s just how it ended up. I can probably put the extra files somewhere else but it seems simpler to have them all in single directory tree. Now I write it maybe I should’ve made it at /var/hg/root or something like that.

Well, it’s done deal.

I also made ~hg/.ssh/authorized_keys file and fill it with my key. Again, so I can push to it.

With that done, next is installing the required packages:

  • py3-gunicorn
  • supervisor
  • mercurial
  • nginx
  • certbot

Refer to this post on configuring the certbot. It worked so well and requires barely any maintenance so far.

As for gunicorn, I made /home/hg/hgweb directory which contains following files:

  • gunicorn.conf.py
  • hgweb.config
  • hgweb.py

Gunicorn config is pretty simple:

bind = 'unix:/home/hg/gunicorn.sock'
workers = 4
accesslog = '-'
errorlog = '-'
timeout = 30

Nothing fancy, and there’s no worker_class because none of the supported workers (apart of sync) seem to be supported under OpenBSD. Should be fine as it’s just for my personal use.

As for hgweb.py, it’s copied from /usr/local/share/mercurial/hgweb.cgi with config path adjusted to local hgweb.config and removed references to wsgicgi (import and .launch) as I’m using Gunicorn, not CGI.

hgweb.config itself on the other hand, it’s also pretty basic:

[paths]
/ = /home/hg/repos/*

[web]
baseurl = https://hg.myconan.net/
contact = nanaya
staticurl = /static

All those done, last part to start serving with Gunicorn is updating /etc/supervisord.conf. There’s an example in their official docs and I made some adjustments:

[program:hg]
command=/usr/local/bin/gunicorn --config=/home/hg/hgweb/gunicorn.conf.py hgweb:application
user=hg
directory=/home/hg/hgweb
stopsignal=INT
environment=PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin"
stdout_logfile=/var/log/supervisord/%(program_name)s-%(process_num)s.stdout.log
stderr_logfile=/var/log/supervisord/%(program_name)s-%(process_num)s.stderr.log

Mainly for having non-random log file path.

Create log directory with mkdir -p /var/log/supervisord, enable the service with rcctl enable supervisord, and hope it works.

Oh and chown hg:www /home/hg && chmod 710 /home/hg for basic file permissions. Oh and hg:hg owner and 700 permission for repos directory itself.

And lastly nginx:

server {
    listen 443;
    listen [::]:443;
    server_name hg.myconan.net;

    access_log /var/log/nginx/hg.myconan.net-access.log;
    error_log /var/log/nginx/hg.myconan.net-error.log;

    ssl_certificate certs/hg.myconan.net/fullchain.pem;
    ssl_certificate_key certs/hg.myconan.net/privkey.pem;
    ssl_trusted_certificate certs/hg.myconan.net/chain.pem;

    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;

    root /nonexistent;

    location = /favicon.ico {
        return 204;
    }

    location = /robots.txt {
        return 204;
    }

    location / {
        proxy_pass http://unix:/home/hg/tmp/gunicorn.sock;
        proxy_set_header Client-Ip $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header X-Forwarded-Port $remote_port;
        proxy_set_header X-Forwarded-Proto $scheme;

        limit_except GET HEAD {
            deny all;
        }
    }

    location /static/ {
        root /usr/local/lib/python3.8/site-packages/mercurial/templates;
    }
}

Nothing fancy either, just a basic https proxying setup with no write permission as I don’t want to setup http auth and only push using ssh.

/static/ directory is served directly to the installation’s templates directory. Subdirectory name already matches so no alias or symlink is needed.

rcctl enable nginx and don’t forget to rotate the log files by adding the two specified files to /etc/newsyslog.conf.

…that’s kinda long.

Weekly FGO vol. 83

JP

Only epilogue left (and the challenge quest). There are still plenty of time so I’ll try clearing it with lower rarity servants.

On side note, there’s a Fujino banner which I guess kinda makes sense? Inb4 this summer is KnK summer and there’s Fujino summer. Probably not. But one can wish.

Or maybe Tsukihime.

I did 22 rolls and thankfully got her on second 11 rolls. Also got second Penthesilea which is useful as she’s got plenty of party buffs and NP charge (for her own).

Anyway, that’s that and I’m not quite sure what to farm in the meantime until next event. I actually very rarely farm for mats at event free quest but maybe I’ll try it this time around. Can use lots of octuplets and seeds in addition of the usual proofs/dusts/bones.

NA

Hunting Quest is over so now back to slow farming. I don’t think I gathered enough exp cards during the event but oh well. I’ll wait until next super/great success rate up before using them (further) up. I’ve used some as I needed to clear space but I ended up not farming quite as hard as I would like to.

Coming up next is… next Lostbelt? I’ll slowly clear it up and maybe roll a bit for Asclepius.

Weekly FGO vol. 82

JP

Summer has started! The rerun at least. A very early one.

Oh and that Grail Front has ended. The new chest box thing sure was annoying. I guess the developer managed to find a way to force people to actually fight instead of toying around with the AI to finish the map with as few fights as possible.

Back to summer, I did 22 rolls for Illya and managed to get her (second copy) right at the end. That’s good because new summer will be coming soon and there might be someone I actually want. Or not. We’ll find out.

Still slowly progressing through the story although I think I’ve cleared half the missions and a good chunk of the store.

NA

Hunting quest! Yay. I need exactly that. Some essential mats, occasionally lots of qp, and most importantly exp cards. Each on their own is nothing special but getting all of them in single run is quite a deal.

Unfortunately I didn’t manage to get enough lancer exp cards. I did spend quite a lot of apples for that but getting over a thousand of it requires a bit too much effort.

Second day (today) is berserker cards and stakes. I don’t exactly need so much of the latter and I’ve farmed enough berserkers card so I’m taking it slowly until the next one.

I’ll need assassin cards the most but the item, horseshoe, I don’t exactly need too many of them at the moment so I don’t know if I’ll farm a lot there or just whatever. Maybe depends on if I can do it easily without too many taps.

Almost forgot but I did even more rolls on Jeanne Alter and didn’t get her. Welp. I did also get some more Salieris which is nice I guess and then two Tristans which I don’t think I’ll need because I’ve got Chloe for single target archer (although with him being quick may prove useful until Castoria arrives). The best part though, is I got Kaleidoscope. My first one. It’ll surely be useful for farming and stuff. Now I just need to get 5 more of it…

Weekly FGO vol. 81

JP

Dead week next two weeks. At least the free tickets and grail is kinda nice. Also the new strengthening quests are pretty useful. Not so sure about Ozy’s but Sanzo and Bedi are definitely made them way easier to use especially for farming. I still need to level up Sanzo’s skills though. Them caster skill stones are pretty rare.

Nothing else in the horizon although maybe Merlin gacha banner coming up next week? Weird he’s not on banner despite being featured in the event banner.

Oh and as for the grail front itself, it’s… annoying. If I ignore the gold chest box, it’s still pretty easy. Things get very annoying once I try fetching them. Even more annoying the content isn’t all that useful. Low total cost allowed, very high master movement cost, and that chest box combined made the event this time very annoying. I guess I’ll ignore those boxes first time and come back later if I feel like to.

NA

Meanwhile here, the current event is even deader than JP for me because I’m not reading it again.

In the mean time, I did 20 rolls even though I shouldn’t and of course I didn’t get Jalter. I did get Salieri though so that’s nice I guess.

Also Arjuna NP2. I don’t really find him all that useful though. He’s got no survival skill beyond debuff immunity and some heals. His damage buff only lasts for one turn and his NP charge only charges for 25%. Alone he’s not all that bad but I’ve got Napoleon and Jeanne Summer as well. And both provide some party buff better overall buff.

Weekly FGO vol. 80

???

JP

Event is mostly done. I’ve also cleared the challenge quest… kind of.

It certainly wasn’t the smartest way to clear it. I used double Castoria and Musashi Summer to just bulldoze my way through.

May or may not look into clearing it again later.

I also still haven’t cleared the store yet. Maybe later this weekend as I’m currently busy in NA.

NA

I ended up doing 20 rolls for Reines and got Napoleon instead (?). Well, that certainly wasn’t expected. He’s also pretty expensive to level up with loads of mats required for the skills.

Meanwhile I’m almost done with the event itself just need to get a few more event currencies.

And then the raid event. All Barbatos all day. Currently at 86 kills but I still need a lot more QP and mats. I hope I can get at least 200? The team is rather annoying though requiring order change because I don’t have enough firepower otherwise.

IPoE, but static IPv4

Continuing from previous post, at the end I mentioned about using Vultr to avoid paying extra for static IPv4 address through my ISP.

Well, there has been a different problem with IPv4 connection crapping out every now and then so I ended up getting that ISP static IP option hoping it will lessen the problem. No comment on that yet because it’s only been less than 12 hours since I got it set up.

So, the setup itself, because I’m not using one of the supported routers, I had to figure it out myself.

The ISP provides a few needed information for the setup:

  • Static IP Tunnel Endpoint: an IPv6 address to connect for IPv4 connectivity
  • Interface ID: IPv6 address suffix (last 4 group)
    • FreeBSD doesn’t support it (it’s ip-token in Linux) but it really is just for address suffix. Mine’s ::feed so my expected address is 2409:11:1c0:2300::feed. I have it set as external IP address
  • Static IPv4 Address: this is to be set at tunnel interface as source address.
    • There’s no IPv4 target address provided which is required for FreeBSD’s gif interface but apparently any address works. I put in 10.0.0.0
    • This blog says to use source as target as well but apparently it results in packet being forwarded back and forth indicated by 14ms ping to the source IP
  • “Update Server Details”: I have no clue what this actually does
    • It’s a set of URL, username, and password where you’re supposed to make a request to to update… something. The form is simple, just $URL?username=$USERNAME&password=$PASSWORD. The URL uses internal domain so the DNS server from IPv6 autoconfiguration is required to resolve it
    • I just hit it with curl and the move on
    • I suspect it’s to tell the tunnel provider the expected source IPv6 address?

Geared with information above, there are a few changes needed since last post for setup on FreeBSD:

  • IP address on internet port should be suffixed with provided interface ID
  • Tunnel source and target address need to be adjusted
  • Tunnel interface need IPv4 address
  • Default routing for IPv4 is no longer on interface level (-iface gif) but instead the random IPv4 address used as tunnel target address (10.0.0.0 in my example above)
  • NAT is not automatically available anymore so PF is required
  • Also on NAT, MSS will need to be fixed as well
    • I still don’t really understand how this works

Most of the changes should be obvious. And here’s the config for PF:

# This is pf.conf for FreeBSD and won't work on OpenBSD

# variable to not hardcode interface names and stuff
ext_if = gif0
net_local = "192.168.0.0/24"

# I still don't know if this is needed. Or even what the correct value is.
scrub on $ext_if max-mss 1420

# basic nat
nat on $ext_if from $net_local -> ($ext_if)

IPoE, FreeBSD, and Realtek

Update: made it work again with Realtek (see update at the end).

Also: DS-Lite, Japanese ISP, ND proxy, and static IPv6.

After upgrading my server to FreeBSD 13, my ethernet failed to obtain autoconfigured IPv6 address which was weird. It’s been kinda weird before occasionally not receiving address manually after reboot but at least it works if I let it autoconfigure during boot.

Thanks to the fact the IPv6 works pretty much plug and play and usable on multiple systems just with switch, I booted up another FreeBSD 13 system hoping to find out if it’s some broken configuration on FreeBSD 13 on the server or something else.

The result was test system got its IPv6 autoconfigured, even manually with rtsol. Also weird was the main server got its address autoconfigured as well.

While at it, I wondered if I can just use static IP so the overall configuration can be simplified. And the answer was yes: it just works as long I enter the detail manually. I’ve been entering them mostly manually anyway so this was good news.

Good news it was, until I tried it on the main server itself: it worked when the modem and server are bridged by another switch but not when connected directly. It just didn’t work. Swapping back to the switch made it work again.

Back to testing, I tried direct connection to test server, and interestingly enough it worked right away. It also survived reboot, disconnect/reconnect, reconfiguration, etc.

At that point I pointed down it to the possibility of Fast Ethernet mode (100Base-TX) of Realtek just being weird and whipped out my old trusty USB ethernet dongle. And it just worked. Good job, Realtek.

So, yeah, something is broken with Realtek but I don’t care enough to dig deeper so dongle life it is.

As an extra, here’s my configuration, complete with ND proxy so the main server can distribute IPv6 address to other clients at home without having to bridge the modem directly (which gives horrible result of unwanted DNS suffix especially on Windows).

ISP is Interlink and using DS-Lite tunnel (“Multifeed” for this ISP) for IPv4 access.

### BEGIN /etc/rc.conf
# ue0 = internet port (connected to modem)
# em0 = internal port (connected to home switch)

# Basic static IPv6 configuration
ifconfig_ue0=up
# promisc option is probably set by ndproxy and not needed to be explicitly set here but I haven't tested it
# prefixlen 128 so no routing added for this port while keeping the requirement for internet port
# the address prefix and default route can be obtained when using autoconfiguration
ifconfig_ue0_ipv6="inet6 2409:11:1c0:2300:: prefixlen 128 promisc"
ipv6_defaultrouter="fe80::21e:13ff:fec2:e9c5%ue0"

# DS-Lite tunnel
cloned_interfaces=gif0
# target address can be obtained by searching the internet (multifeed) or just ISP documentation
# MTU is from experiment: raise MTU and ping around until it times out (and then add 28 bytes header)
# example: ping -s 1432 -D answers.microsoft.com
# and then try 1434 (with MTU 1500)
ifconfig_gif0="inet6 tunnel 2409:11:1c0:2300:: 2404:8e00::feed:100 prefixlen 128 mtu 1460"
defaultrouter="-iface gif0"

# nd proxy. Don't forget to install the package first: pkg install ndproxy
ndproxy_enable=yes
# interface that connects to the uplink (internet)
ndproxy_uplink_interface=ue0
# mac address of the interface above. Or maybe random address could also work. Not sure
ndproxy_downlink_mac_address="00:22:cf:xx:xx:xx"
# same as defaultrouter above but without interface name
ndproxy_uplink_ipv6_addresses="fe80::21e:13ff:fec2:e9c5"

# internal connection (with local IPv4 for NAT)
ifconfig_em0="10.0.0.1/24"
# same prefix as external interface but prefix 64
ifconfig_em0_ipv6="inet6 2409:11:1c0:2300::1 prefixlen 64 -accept_rtadv"

# for distributing ipv6 addresses. No configuration needed
rtadvd_enable=yes
rtadvd_interfaces=em0

# not sure which of the following are actually needed
ipv6_activate_all_interfaces=yes
# pretty sure at least corresponding forwarding sysctl are needed to be set if those two lines are not enabled
ipv6_gateway_enable=yes
gateway_enable=yes

### END /etc/rc.conf

Interestingly NAT doesn’t need to be manually configured: the DS-Lite tunnel magically handles it. I also keep forgetting about this and confused by the lack of NAT setting in my pf.conf.

Note that the outgoing address 2409:11:1c0:2300:: isn’t reachable from internal network with this configuration. Use 2409:11:1c0:2300::1 instead, including for external access (like this blog).

I should also write up my Wireguard-based external IPv4 one of these days… (because I’m too cheap to pay for Interlink’s static IPv4 – Vultr additional IP for 220yen vs Interlink IPoE static IPv4 for 1100yen).

Update 2021-05-18: I installed Realtek driver (realtek-kmod package) and it works. I previously had to use it as the driver was missing in FreeBSD 12 but switched to the updated built-in driver when upgrading to 13. Tried again with the driver and it works in 13.