Because having a stable setup is too boring, I tried the (not so) new IPoE setup.
Continue readingFlets and IPoE and FreeBSD
Reply
Category Archives: software
Network “upgrade”
After waiting for about a week, the cards finally arrived. It’s a pair of Mellanox ConnectX-3 EN. 10GbE card with SFP+ port. They costed a bit over 10k yen total.
Continue readingNVMe FreeBSD boot, take 2
Continuing from previous post, I looked around a bit more and discovered that rEFInd – which I skipped before – is simpler.
Continue readingNVMe FreeBSD boot with Clover EFI
As mentioned yesterday, I’m upgrading (?) my main server storage to NVMe. The adapters finally arrived today and I set up the thing right away.
Base system is an Intel Xeon E3-1230v2 CPU on an Intel S1200BTL motherboard.
Continue readingDKIM signing with OpenSMTPD and dkimproxy
After a long wait of OpenSMTPD 6.6 and its accompanying rspamd filter, I finally sent my mails DKIM-signed. Along the way I also discovered rspamd does a bit too much for my need and learned that it’s way simpler to use dkimproxy instead.
Outline of the steps required as follow:
- Install OpenSMTPD
- Install dkimproxy
- Create signing keys
- Decide “selector” name
- Add relevant DKIM entry to all relevant domains
- Setup dkimproxy to sign stuff
- Setup OpenSMTPD to relay to dkimproxy before finally sending the message
- Test
- Done
Signing keys are created by:
openssl genrsa -out /etc/mail/dkim/selector1.key 1024
Followed by creating the public key for DNS entry:
openssl rsa -in /etc/mail/dkim/selecto1.key -pubout -out /etc/mail/dkim/selector1.pub
Don’t forget to fix private key permission to 400 owned by whatever user running dkimproxy.
The dkimproxy setting is pretty simple:
listen 127.0.0.1:10027 relay 127.0.0.1:10028 domain domain1.com,domain2.net signature dkim(c=relaxed) signature domainkeys(c=nofws) keyfile /etc/mail/dkim/selector1.key selector selector1
It’s pretty straightforward.
And equally straightforward the settings for OpenSMTPD:
table aliases file:/etc/mail/aliases listen on lo0 listen on lo0 port 10028 tag DKIM action "local" mbox alias <aliases> action "relay_dkim" relay host smtp://127.0.0.1:10027 action "outbound" relay match tag DKIM for any action "outbound" match for local action "local" match for any action "relay_dkim"
First line sets the aliases.
Followed by a listener on localhost because this is just an example for sending-only server.
The listener on port 10028 is to accept the signed mail by DKIM to be finally sent.
local action sending mails to mbox for local user.
relay_dkim action will send mails to dkimproxy which will sign the email…
And relayed to 10028, accepted by smtpd, tagged DKIM and thus will be finally sent to where it should be. I learned the hard way it needs to come first because mails will be acted on first match.
Local for local. Don’t bother doing anything.
And finally the rest will be relayed to dkimproxy. As mentioned above, this must come after the outbound action for DKIM tagged mails.
If you want to run proper mail server with rspamd and stuff, read this instead. That article was also the reason I started looking into all this DKIM stuff.
Letsencrypt, cavemen edition
Just had to do some letsencrypt setup in some servers so I figured I should write down what I did so I can just check this page again instead of digging how I did it previously.
Requirements:
- nginx
- certbot
This assumes the server only serves https and redirects all http traffic. Adjust as needed otherwise.
Full nginx SSL/TLS config not included.
First add this config to nginx to handle verification:
# part of default port 80 config block
location /.well-known/acme-challenge/ {
root /var/www/certbot;
}And then create the directory (I’m not actually sure if needed):
# mkdir -p /var/www/certbotMake the first cert because I’m too lazy to ensure the config directory is setup correctly:
# certbot certonly --webroot -w /var/www/certbot -d DOMAIN_NAME_GOES_HERE --keep --agree-tos --email SOME_KIND_OF@EMAIL_ADDRESS --no-eff-emailAt this step, the certificate and all should have been properly generated.
Then use it in nginx configuration, the relevant server block:
ssl_certificate /etc/letsencrypt/live/DOMAIN_NAME_GOES_HERE/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/DOMAIN_NAME_GOES_HERE/privkey.pem;
ssl_trusted_certificate /etc/letsencrypt/live/DOMAIN_NAME_GOES_HERE/chain.pem;
If the full path seems too long, symlink it to nginx config base directory or something.
Update certbot CLI configuration located at /etc/letsencrypt/cli.ini:
rsa-key-size = 4096
text = True
authenticator = webroot
webroot-path = /var/www/certbotTo add more certificates:
# certbot certonly -d ANOTHER_DOMAINDon’t forget to update nginx configuration as before.
Since the certificate needs renewal periodically, create this simple script:
#!/bin/sh
# I personally put this in /root/bin/refresh-ssl-certbot
/usr/bin/certbot renew
/path/to/sbin/nginx -s reloadMake executable, etc. Try it to make sure it runs properly.
Then add it crontab. I usually do it weekly.
And done.
There might be smarter way using certbot’s nginx plugin or something but I haven’t bothered reading its documentation and initially this was just a stopgap switching from acme-client which is way simpler but stopped working for me few months ago.
IP address checker
Random idea I came up when reading nginx mailing list. A very simple way to set up external IP address checker using nginx on a remote server.
location = /ip {
default_type text/plain;
types { }
return 200 $remote_addr\n;
}
Accessing /ip will then return the current external IP address. A more fancy output like JSON is possible as well.
VirtualBox again
Due to reasons I swapped my web server (was in VM) and VM box role. Or more like web server is now bare metal and functions as VM box. It’s downgraded from real virtualization (Hyper-V) to VirtualBox though.
That said, FreeBSD doesn’t seem to be quite fully functional on Hyper-V thanks to performance hit and slow disk detection on boot causing failure.
Yeah, I’ve had enough of it so I decided to swap the role and run the web server directly on the server instead.
On the bright side, I can use zfs for everything and no more stupid SSD setup (it was unbalanced 500 and 250 GB disks).
VirtualBox seems to be flaky as always though apparently once it’s setup correctly it runs without much trouble. Also thankfully VirtualBox on FreeBSD has complete scripts for running it headless and autostart on boot.
This was also my main setup for quite a long time years ago and I sure hope it’s at least as stable as it was back then! That said, the initial setup seems to be flakier. Oh well.
Rails and Bootstrap
This hit me when I was rewriting front-end part of Zeropaste. Of course later I found out that there’s gem for it.
But anyway, if you don’t want to add another gem and feel like writing crapload of divs, it’s not that difficult. It breaks Rails standard form error handling though.
First, change the default error field handling to do nothing instead of wrapping it in a div:
config.action_view.field_error_proc = proc { |html| html }
(put in application.rb)
And then create this helper:
def error_class(object, attribute) "has-error" if object.errors.include? attribute end
Finally, here’s how to use it (for attribute key of a model):
<div class="form-group <%= error_class f.object, :key %>"> <%= f.label :key %> <%= f.text_field :key, :class => "form-control" %> </div>
…and done.
Network performance checklist: TSO/LSO
Took me good few hours to find out why specific combination of task performs incredibly slowly on my server.
For reference, in my case it’s port-forwarded ssh/https connection over openvpn.
[ Client ] --(Internet)--> [ Gateway ] --(OpenVPN)--> [ Server (SSH) ]
Doing anything which takes up bandwidth (displaying log files, etc) will shoot the cpu load (at interrupt) up by a lot. By a lot I mean over 50% of one core.
This guy have the reasoning why it’s happening but I don’t know how much of it is correct. VMware support page also suggesting disabling it if network performance is slow.
In FreeBSD it’s:
echo 'net.inet.tcp.tso=0' >> /etc/sysctl.conf
In Windows it’s this .reg file:
Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters] "DisableTaskOffload"=dword:00000001
It seems to involve a shell script in Linux so I won’t bother writing it off here since it differs by system.
As usual, YMMV.
…and there goes my time 🙁
Jiku-Chu direct link
If you visit Jiku-Chu regularly, you may have noticed that some of their links use javascript and can’t be opened to new tab. And thus here’s the user script to turn them into normal links. Tested on Chrome/Tampermonkey. Probably works on Greasemonkey as well? Idk.
[ Script link ]
dirlist-php
A few years ago I wrote a php script to provide better autoindex within nginx. I used it for quite a long period until I rewrote it in Ruby/Sinatra. But then I figured the setup for it is overly complicated just for simple task. And takes additional memory. I always have php-fpm running anyway so it’s free.
And so I decided to take up the old php script and fix it up. Unfortunately there isn’t other language as easy to setup for web as php which is why I fixed it instead of rewriting in some other languages (or keeping it ruby). The “fixed” version is still pretty much imperative-style but hey, it works.
Only tested with nginx.
Wasting time / 2048
I’ve been wasting time on this thing called 2048 lately. And finally obtained my then-current goal of 8192. And then ended at 112k score since I wasn’t careful enough.
Next goal is 16384 but I’ll need to be able to reach 8192 without too much effort first…
PSA: Never trust external X-Forwarded-For
For god knows how long, proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; is one of the line usually included in nginx config snippet for proxying to a unicorn (Rails) backend.
…which is something you should never do unless you have another load balancer in front of the nginx being configured.
That line basically tells nginx to append $remote_addr to whatever X-Forwarded-For value currently set. It is only useful when your nginx is behind other load balancer which set up its own (hopefully correctly) X-Forwarded-For. It should be set explicitly to $remote_addr for any external-facing proxy. Otherwise fun things will happen.
FreeBSD pkg (manual) upgrade thingy
For working with locked packages (the ones which must be upgraded through compilation because of using custom options)
#!/bin/sh # filename: pkg-lock-outdated pkg query -e '%k = 1' %o | while read pkgorig; do pkg version -ovL => -O "$pkgorig" done
The script above is to list locked packages which need upgrading. And to upgrade everything at once (and sit in front of PC waiting for whole process)
#!/bin/sh listfile="/tmp/pkgforupgrade.$(date '+%Y%m%d%H%M%S')" pkg-lock-outdated | cut -f 1 -d '<' > "$listfile" while read <&3 outdated; do pkg unlock "$outdated" portmaster "$outdated" pkg lock "$outdated" done 3< "$listfile" rm -f "$listfile"
There’s another alternative of unlocking all packages at once, run batched portmaster, and lock them all again.
Zeropaste update
It now supports paste deletion. Only via web rendered properly for now.
Inb4 feature creep.
WordPress 3.5 Initial Impressions
This blog has been updated to WordPress 3.5. Initial impressions:
- everything is less rounded now
- Twenty Twelve (the new theme) sucks for following reasons:
- uses custom font: causes this blog unreadable until fonts are loaded (at least in chrome)
- image header: positioned below navigation menu (and the header text is above the menu)
- no more search bar in navigation menu
- the new media uploader looks useful
Overall there isn’t much change (what should I expect here) but the new theme sucks. At least to be used on this blog.
0paste Updates
With my move to a better VPS, I changed some things on 0paste:
- MRI Ruby 1.9.3
- Passenger
Worked quite well and much easier to maintain.
Zeropaste update
I decided to mess around with Zeropaste and added some “features”:
- Show paste in plaintext (fixed width) and Markdown
- Allows displaying paste in fixed width instead of full/auto
- Setting mode and width based on url
- Automatically updating url based on selected mode and width
0paste.com has been updated accordingly, including Rubinius 2.0.0rc1.
Now you can read glopping Asuna in convenient fixed width markdown (NSFW).
Yet another pastebin: 0paste.com
I decided to get a domain for it because of :reasons:. Anyway, the old ones from p.myconan.net is still accessible (it’ll redirect to new url at 0paste.com). Donations welcome.
Also because the world needs yet another pastebin. This one is running on Rubinius using Puma, by the way. Four threads with awesomest possible caching for showing pastes.
[ 0paste.com | Source ]
Completely Disable UAC in Windows 8
Windows 8, just like Windows 7, has Control Panel interface to disable UAC. There’s difference though: disabling UAC via Control Panel in Windows 8 doesn’t fully disable UAC. You can check it by launching Command Prompt: in Windows 7, you’ll get administrator command prompt (the signs are “Administrator: Command Prompt” window title and default directory at %WINDIR%System32) while in Windows 8, you’ll get normal command prompt.
Also reported here (complete with “fix”).
Fix by editing registry:
- Key/Path:
HKEY_LOCAL_MACHINESOFTWAREMicrosoftWindowsCurrentVersionPoliciesSystem - Name:
EnableLUA - Value:
0
Note that there isn’t really much reason anymore to completely disable UAC. Well, in my case, this ancient (but useful) tool isn’t quite compatible with UAC.
Windows 8 Initial Impressions
6 hours with Windows 8.
MetroModern UI is shit (the applications)- The Start Menu (or whatever it’s called now) is ok-ish though
- The lock screen is awesome with its customizable background image
- The global one is difficult to change though
- File association was updated and doesn’t work properly in at least IrfanView and 7-Zip
- Compressed folder still exists and can be disabled the same way as Windows 7
- So is Aero Shake. Really though, there is no more Aero but there’s still Aero Shake? Is it some kind of joke?
- The window border is thick-ish but I guess OK for now (quick google also showed some border customizer)
- Hyper-V is in it as promised. Works as expected. Most OS need Legacy Network Adapter (except Ubuntu and SLES) or installation of the integration driver for some OSes. The biggest plus is it works at system level – VMs can be automagically started as Windows starts
- All applications I usually use works without problem
- It’s not move to (top right) corner but move to corner and move down a bit
- Updates (from Microsoft Update) which requires license agreement doesn’t work (at least on my PC)
- There’s no discernible performance difference
- Yes, Everything also works
- Also, flat
- Windows Explorer is a bit better: no more intrusive action buttons above file list (moved to the top)
- Remote desktop work OK. There’s no more classic style but I think the new one (Modern UI) doesn’t use too much bandwidth (being flat and all)
- WRT RDP, the 2X Client I usually use in Android doesn’t work with Windows 8. The good old Remote RDP Lite works though (and with CyanogenMod’s plain keyboard, the physical keyboard input works again)
I think that’s it for now.
Low End Box and Web Applications
…one does not simply run both PHP and Rails applications (and MySQL) in single 128 MB box.
This is what happens if you try doing it. Swapping all the time. Hopefully Debian will fare better but it means I’ll need to reinstall the box again. I’ll do a test run in my local machine first though as not to waste my time doing backup/restore again like today.
(Yeah, Zeropaste is up at p.myconan.net)
Zeropaste – the featureless pastebin
Tonight, when trying to compile Rubinius with Rubinius, I got some errors and wanted to report to relevant party. As usual, the log of what happened is required so I used my usual pastebin – pastie.org to send the logs. But then I noticed that the “Raw” link in it doesn’t provide an actual raw file anymore. What the fuck. It is now a html disguised as txt.
So I decided to whip up a new pastebin (because the world need one more pastebin) which doesn’t have any actual features (like tag highlighting, etc). I also learned the way to create shortest RESTful path possible (read: ‘/’).
There may or may not be more features coming. Developed in Rubinius because I can. Using mysql because of where it’ll be deployed at (see below).
I’ll get around deploying this soon after reinstalling VPS running this blog to Ubuntu or Debian. Running yum in a 128 MB box is suffering.
[ Source Code ]
Ruby 1.9, Rails, and UTF-8
(main purpose of this post is to link this “server error” page of rubygems.org)
The relevant issue in Rails Issue Tracker (3789). AFAICT, there are few ways to “fix” (read: workaround) this:
- Modify the relevant Rack code to handle this crap
- Create additional middleware to intercept (how?) the request (tried, either didn’t work or horribly inefficient)
- Extend rack before it is started
Well, they all sucks. Hopefully someone comes up with actual working solution for this.
Oh, there’re another solutions:
- Use REE 1.8 (really?)
- Use JRuby in 1.8 mode
- Use Rubinius (rbx 2.0 where?) in 1.8 mode
Um, yeah.
Update: I figured out how to “fix” it. Check it out in Moebooru (requires this).
Rails: read_multi and dalli
Be careful when using read_multi with dalli: it may return nil-valued key instead of the correct key.
The issue is tracked here and thanks to this I dropped the read_multi usage in moebooru and used the much simpler (and most likely slower) single fetch (per entry) instead. There’s alternative way to use it – do a read_multi and refetch whatever missing/nil-keyed but apparently I’m too lazy to do it.
Configuring SHA rounds for password in RHEL5/6
Due to NIH syndrome and Drepper being Drepper, the only remotely secure password hashing algorithm in RHEL5/6 is multi-rounds SHA512. The default is just salted SHA512 which sucks.
Also applies to CentOS, ScientificLinux, and other RHEL clones.
Anyway, to update the default setting, these files need updating:
/etc/login.defs: add new lineSHA_CRYPT_MIN_ROUNDS 5000/etc/pam.d/system-auth-ac: find line with `password sufficient pam_unix.so sha512` and appendrounds=5000.
Note that the change to last file may or may not be persistent. I have no idea how to properly set it up.
Finally, run this command: authconfig --updateall.
If you’re using RHEL5, run authconfig --passalgo=sha512 --update first.
Moebooru 3.2
Finally reached the goal: Rails 3.2.6. The isn’t much change between 3.1 and 3.2 which is why I skipped 3.1. As there isn’t much front-end change, it means there isn’t any user-visible changes. Or at least there shouldn’t be.
Plans:
- Move to jQuery
- Use paper_trail for versioning (or something else)
- Add SVG support (and maybe drop SWF support)
- Actual news ticker
- And more!
3.2.0 has been branched and currently running on some servers. The next version, 3.2.1 is on default branch and being tested on moe.myconan.net. As (finally) there are database changes, it will not be tested directly on live servers until it is stable enough.
Changes include better support for JRuby, use mini_magick instead of custom plug-in, less monkey-patches (but more bugs), and various clean-ups.
Rails 3.2 in Subdirectory
Steps to be into Rails in subdirectory:
Update config.ru to understand the subdirectory mapping:
run Moebooru::Application
change to
map (ENV['RAILS_RELATIVE_URL_ROOT'] || '/') do run Moebooru::Application end
And then start Rails with correct environment variable (example if you use Unicorn):
RAILS_RELATIVE_URL_ROOT='/img' bin/unicorn
And that’s it. No need to mess with routes.rb as I previously thought after searching and experimenting for few hours. Links etc are properly generated with correct prefix. Or at least based on my quick testing.
May or may not work with earlier version(s) as I haven’t bothered to test it anywhere else.
Basic Dovecot/Postfix in Ubuntu
Configuring mail system is annoying. There are quite a bit different components which must be configured to work together.
My main choice for mail system is dovecot/postfix. As I don’t really understand how all this thing goes, I may have missed or misunderstood some parts. Or most of them. Feel free to correct this post.
For starter, most of basic configurations for Postfix and Dovecot has already been done by Ubuntu (or Debian) default configuration which includes enabling IMAP and TLS.
LDA
LDA (or MDA) delivers received mails to correct user and location. I let dovecot handle this thing because it’s easier this way. In /etc/postfix/main.cf:
mailbox_command = /usr/lib/dovecot/dovecot-lda -f "$SENDER" -a "$RECIPIENT"
[ Source ]
And that’s about it.
Maildir
The Mailbox format. The alternatives are mbox (ancient, shouldn’t be used anymore, I believe), or dbox (Dovecot only), or some other formats (which I don’t really care about). So basically I go with Maildir.
/etc/dovecot/conf.d/10-mail.conf:
mail_location = maildir:~/Maildir
[ Source ]
SASL
The last one, Postfix authentication. I use Dovecot SASL because it’s easier.
/etc/postfix/main.cf:
smtpd_sasl_type = dovecot smtpd_sasl_path = private/auth smtpd_sasl_auth_enable = yes smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination
/etc/dovecot/conf.d/10-master.conf:
unix_listener /var/spool/postfix/private/auth {
mode = 0666
}
[ Source ]
For Dovecot config, it’s usually already in there, one just uncomment it to enable.
Restart Dovecot and Postfix, and that’s it. As I mentioned before, Ubuntu has preconfigured many things which leaves me only few additional tasks to be done to enable simple mail system (with TLS, IMAP, and whatnot).
Protip: use Google Apps or Live Domain instead of managing your own mail server.