Server upgrade (part 2)

So FreeBSD ran stable on my latest Ryzen setup for at least 18 hours. I guess it’s safe to say it’s stable now.

The x1 graphics card has arrived, and it works without problem. It sure is nice having a tiny graphics card. Maybe I should get more of this.

There were small hitch after I changed back the SATA cabling to connect to the HBA. Some of the drives weren’t detected properly for some reason. Fiddled with the cables a bit and thankfully everything came back up normal.

With this, the upgrade is mostly done. Apart of the ethernet card which I’m still waiting for arrival of the ones with correct bracket and ECC RAM which will be a while until I save enough money for.

There’s also SMR drives situation. I recently learned that manufacturers have started switching to SMR drives which has relatively low random write speed. That explained why the resilver time was so horrible back when I upgraded the pool. Thankfully it can buffer some burst load although it’s not always enough for ZFS operations. The read speed should be mostly fine so I’m thinking of keeping these drives until they die which then I’ll buy normal PMR drives for the replacements. Unfortunately, those will be a bit expensive.

Oh, I almost forgot one more upgrade coming whenever this pandemic situation is over. I’m thinking of taking home a pair of 500GB-ish SSD currently sitting idle in work dev server. I can use them for my home partition as the (NVMe) disk is getting full at 70% and my home directory somehow accounts for 100GB (20%) of it. Slightly slower home directory will be a bit sad but it’s better than having full disk.

Unrelated, but looking again, I considered getting X470 motherboard instead of X570. It’s a bit cheaper and doesn’t need fan. It also has better PCIe configuration at x8/x8 instead of my current X570’s x16/x4. Too bad one of the M.2 NVMe slots is only PCIe Gen 2 x2 (the cheap one). That’s a bit on the slower side for NVMe disk. Still, I probably should’ve gone with that one. I might even be able to fit x16 graphics card on the x1 slot. It’s too late now. RIP me.

There’s also the ASRock Ryzen server motherboard which price quite a lot lower than I thought at 30k. But that one is, well, more expensive. And involves buying directly overseas.

Server upgrade

Not the final form

I’ve been considering this on and off quite a long time ago as I noticed the Intel part post-Ivy Bridge isn’t going to get much cheaper. And then during my 10Gbit upgrade a while back, I learned my server could barely handle half of 10Gbit available. There’s also problem with I need a bit more RAM but I don’t want to buy any more DDR3 sticks as it’s a dead platform by now.

Thankfully Ryzen continued AMD’s tradition (?) of not locking ECC feature on most systems so I upgraded to it three years ago. And my server crashed. A lot. It was unstable. I then tried again two years ago but it’s still crashing. I ended up selling the system and bought a cheap Ivy Bridge server board from ebay last year. It held up pretty well. It even got NVMe upgrade earlier this year.

The thing is, just like for desktop, I sure could use faster CPU. Ryzen 3000 series brought along a lot of per-core performance (IPC) increase. A lot more compared to 1000 series. And it has gotten pretty cheap, at least on 6 cores realm.

I upgraded my desktop end of last year and now it’s turn for the server. Except unlike the desktop one, there’s no good deal this time around. It didn’t help I need more PCIe slots than usually available on cheap motherboard. And I actually wondered if I should wait for B550 and see how it goes especially considering X570 requires fan for its southbridge.

But I ended up getting X570 anyway because I don’t want to wait longer 🙂 I’ve resumed doing some hobby dev work recently and sure could use some upgrade. Especially as my plan for VM on desktop system with NFS-backed storage didn’t go quite well.

Anyway, I upgraded the motherboard (ASRock X570 Pro4) and CPU (AMD Ryzen 5 3500). For RAM I took two sticks from my desktop which currently has a bit too many. Those will need to be upgraded to ECC sometime later when the budget permits.

For graphics card, as this isn’t a server board and there’s no onboard GPU on the CPU, I’m getting a cheap PCIe x1 GT 710 1GiB from Zotac. It costed a bit under 5k on Amazon Outlet. It’s second hand but should be fine. I hope. It’s not arrived yet so I’m currently using another fanless GPU I have but it’s using one x16 slot as even though the PCIe x1 slots are open-ended, there isn’t enough clearance for x16 card.

With one of the only two x16 slots used by the graphics card, I stuck with using my HBA at x1 slot. Thankfully the motherboard has loads of SATA ports (8) so I only need two from the HBA. There’s no cable management though as 1) it’s temporary; and 2) it’s way more annoying with 10 SATA cables in total instead of just two SFF-8087 and two SATA. That should be fixed this weekend.

There’s also network card bracket problem. The brackets I mentioned weeks ago have finally arrived but the size didn’t match. Good job, Fujitsu. I couldn’t find the brackets for those cards either so I’m getting another pair of cards. Assuming they will actually arrive as they’ve been stuck in China for a bit over two weeks now. I just hope they actually arrive. And that they actually work. That would be nice.

The unused board and CPU and RAM will be repurposed for my work dev server. My current one is pretty similar just one generation behind (E3-1235 vs E3-1230v2). I need a new case and PSU though but those shouldn’t be too expensive. Combined, the server will have plenty of RAM (32GiB).

That said, I don’t know if this 3000 series of Ryzen is finally stable enough for FreeBSD. That’s actually the most important thing as otherwise I’ll be forced back to the old system and maybe figure out what to do with this board and CPU. I’ll report back when I got the correct graphics card, I guess. Or earlier if it still crashes.

Bonus photo:

This is definitely not how to install a card. It does work though

Upgrade Log 3

The last one for this batch! Everything arrived, assembled, and finished without much problem.

Windows 10 is even more annoying than ever. Disabling Cortana now must be done using Group Policy. Great. I have to slowly live it up because this is the future of Windows and I don’t see myself using another operating system for desktop for foreseeable future.

Also, don’t disable universal app background process if you want a functional start menu search.

<insert a bunch of other tweaks here>

Up next

Closest upgrade I can think of is getting an extra 6+To drive so I have 6 drives raidz2 instead of current 5 which is quite a waste. I’m not sure how to migrate the data though. That’ll cost about 25k?

And I remembered about my netbook only having 2Gio of RAM. Surely can be upgraded to 8Gio for maximum lulz. Or just more useful. I remember it’s much more usable when it’s running on 4Gio of RAM. I don’t exactly remember when and why it’s only 2 now. It already has SSD so the RAM upgrade would pretty much max out upgrades for this system. Not counting higher capacity/performance SSD because I don’t think it won’t make much difference apart of having more storage – faster SSD won’t help the slow CPU much. 5k for RAM.

After that, I can certainly use more storage for my main desktop. A 1To SSD would certainly be nice. A bit expensive at 33k.

With storage out of the way (and moves the 525Go drive to office desktop), I think my office server can also use some storage upgrade. Just like current home server, it can certainly use two more drives for optimum raidz2. That means a controller, HDD cage, and one extra HDD (because I already have one spare 3To HDD). The total would be about 51k.

There’s VGA card upgrade for main desktop but I’m still not sure about that. I don’t really need it but certainly would be nice! Let’s pretend it’ll cost 40k for whatever card at that budget whenever the upgrade is happening.

Talking about VGA card, there’s also a would-be-nice upgrade for my office desktop VGA. It’s currently running GT730 which is not quite fast. Limited to 45W, current choice is limited to GT1030 at 10k.

At this point there isn’t much left to be upgraded. So let’s upgrade the server RAM to 32Gio from currently pitiful 12Gio. I would like to pretend it’s cheap but it really isn’t even now. I was pretty lucky last time getting two sticks of 8Gio for just 10k but it won’t happen often. So maybe about 25k I’d be willing to spend.

I think there is no more after this. I probably won’t reach this far until at least next year or even later anyway and something may break in the meantime, requiring change of plan.

  1. (5k) RAM: 8Gio PC3-12800S
  2. (25k+) Storage: 6+To HDD
  3. (33k) Storage: 1To SSD
  4. Storage:
    • (4k) Controller: LSI SAS 9212
    • (7k) Misc: HDD Cage 2 5.25″ to 3 3.5″
    • (15k?) Storage: 3+To HDD
  5. (40k) VGA card: ???
  6. (10k) VGA card: GT1030 (or better)
  7. (25k) RAM: 32Gio PC3-12800E

Total: 164k.

…maybe this will happen sooner than expected ( ゚◡゚)

VirtualBox again

Due to reasons I swapped my web server (was in VM) and VM box role. Or more like web server is now bare metal and functions as VM box. It’s downgraded from real virtualization (Hyper-V) to VirtualBox though.

That said, FreeBSD doesn’t seem to be quite fully functional on Hyper-V thanks to performance hit and slow disk detection on boot causing failure.

Yeah, I’ve had enough of it so I decided to swap the role and run the web server directly on the server instead.

On the bright side, I can use zfs for everything and no more stupid SSD setup (it was unbalanced 500 and 250 GB disks).

VirtualBox seems to be flaky as always though apparently once it’s setup correctly it runs without much trouble. Also thankfully VirtualBox on FreeBSD has complete scripts for running it headless and autostart on boot.

This was also my main setup for quite a long time years ago and I sure hope it’s at least as stable as it was back then! That said, the initial setup seems to be flakier. Oh well.

Memo and stuff.

Ruby 2.3 on FreeBSD 11

Compiling Ruby on FreeBSD is not quite simple.

  • make sure to tell it to also find libraries in /usr/local
  • and tell configure script to find OpenSSL in /usr because the later version isn’t quite compatible with latest ruby yet

So here’s the configure line

CFLAGS=-I/usr/local/include \
CPPFLAGS=-I/usr/local/include \
LDFLAGS=-L/usr/local/lib \
./configure --prefix="/opt/ruby23" --disable-install-doc --with-openssl-dir=/usr \
&& make \
&& make install

PSA: Never trust external X-Forwarded-For

For god knows how long, proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; is one of the line usually included in nginx config snippet for proxying to a unicorn (Rails) backend.

…which is something you should never do unless you have another load balancer in front of the nginx being configured.

That line basically tells nginx to append $remote_addr to whatever X-Forwarded-For value currently set. It is only useful when your nginx is behind other load balancer which set up its own (hopefully correctly) X-Forwarded-For. It should be set explicitly to $remote_addr for any external-facing proxy. Otherwise fun things will happen.

FreeBSD pkg (manual) upgrade thingy

For working with locked packages (the ones which must be upgraded through compilation because of using custom options)

#!/bin/sh
# filename: pkg-lock-outdated

pkg query -e '%k = 1' %o | while read pkgorig; do
  pkg version -ovL => -O "$pkgorig"
done

The script above is to list locked packages which need upgrading. And to upgrade everything at once (and sit in front of PC waiting for whole process)

#!/bin/sh

listfile="/tmp/pkgforupgrade.$(date '+%Y%m%d%H%M%S')"
pkg-lock-outdated | cut -f 1 -d '<' > "$listfile"

while read <&3 outdated; do
  pkg unlock "$outdated"
  portmaster "$outdated"
  pkg lock "$outdated"
done 3< "$listfile"

rm -f "$listfile"

There’s another alternative of unlocking all packages at once, run batched portmaster, and lock them all again.

WordPress 3.5 Initial Impressions

This blog has been updated to WordPress 3.5. Initial impressions:

  • everything is less rounded now
  • Twenty Twelve (the new theme) sucks for following reasons:
    • uses custom font: causes this blog unreadable until fonts are loaded (at least in chrome)
    • image header: positioned below navigation menu (and the header text is above the menu)
    • no more search bar in navigation menu
  • the new media uploader looks useful

Overall there isn’t much change (what should I expect here) but the new theme sucks. At least to be used on this blog.

PostgreSQL authentication quick start

Connecting to PostgreSQL from command line can be a bit confusing.

For starter, just like MySQL, psql command defaults to connecting to socket instead of tcp. To make matter confusing, most PostgreSQL installation defaults to ident (also called peer)authentication for socket connection: it basically matches current user’s username (ssh login, etc) with PostgreSQL equivalent.

So, instead of using this to login from root to PostgreSQL superuser (usually named postgres or pgsql):

# psql -U postgres

you do this (assuming sudo installed):

# sudo -u postgres psql

The configuration for this is located in pg_hba.conf of PostgreSQL data (or config in Debian) directory (/etc/postgresql/$version/main in Debian, /usr/local/pgsql/data in FreeBSD, /opt/PostgreSQL/$version/data in EnterpriseDB PostgreSQL).

To switch to password based authentication for all methods just replace ident (or peer) with md5 in respective lines and reload/restart the service. Don’t forget to set password for postgres user first before changing this otherwise you won’t be able to connect. You can then connect using psql to any user using password.

Windows 8 Initial Impressions

6 hours with Windows 8.

  • Metro Modern UI is shit (the applications)
  • The Start Menu (or whatever it’s called now) is ok-ish though
  • The lock screen is awesome with its customizable background image
  • The global one is difficult to change though
  • File association was updated and doesn’t work properly in at least IrfanView and 7-Zip
  • Compressed folder still exists and can be disabled the same way as Windows 7
  • So is Aero Shake. Really though, there is no more Aero but there’s still Aero Shake? Is it some kind of joke?
  • The window border is thick-ish but I guess OK for now (quick google also showed some border customizer)
  • Hyper-V is in it as promised. Works as expected. Most OS need Legacy Network Adapter (except Ubuntu and SLES) or installation of the integration driver for some OSes. The biggest plus is it works at system level – VMs can be automagically started as Windows starts
  • All applications I usually use works without problem
  • It’s not move to (top right) corner but move to corner and move down a bit
  • Updates (from Microsoft Update) which requires license agreement doesn’t work (at least on my PC)
  • There’s no discernible performance difference
  • Yes, Everything also works
  • Also, flat
  • Windows Explorer is a bit better: no more intrusive action buttons above file list (moved to the top)
  • Remote desktop work OK. There’s no more classic style but I think the new one (Modern UI) doesn’t use too much bandwidth (being flat and all)
  • WRT RDP, the 2X Client I usually use in Android doesn’t work with Windows 8. The good old Remote RDP Lite works though (and with CyanogenMod’s plain keyboard, the physical keyboard input works again)

I think that’s it for now.

Zeropaste – the featureless pastebin

Tonight, when trying to compile Rubinius with Rubinius, I got some errors and wanted to report to relevant party. As usual, the log of what happened is required so I used my usual pastebin – pastie.org to send the logs. But then I noticed that the “Raw” link in it doesn’t provide an actual raw file anymore. What the fuck. It is now a html disguised as txt.

So I decided to whip up a new pastebin (because the world need one more pastebin) which doesn’t have any actual features (like tag highlighting, etc). I also learned the way to create shortest RESTful path possible (read: ‘/’).

There may or may not be more features coming. Developed in Rubinius because I can. Using mysql because of where it’ll be deployed at (see below).

I’ll get around deploying this soon after reinstalling VPS running this blog to Ubuntu or Debian. Running yum in a 128 MB box is suffering.

[ Source Code ]

Ruby 1.9, Rails, and UTF-8

(main purpose of this post is to link this “server error” page of rubygems.org)

The relevant issue in Rails Issue Tracker (3789). AFAICT, there are few ways to “fix” (read: workaround) this:

  • Modify the relevant Rack code to handle this crap
  • Create additional middleware to intercept (how?) the request (tried, either didn’t work or horribly inefficient)
  • Extend rack before it is started

Well, they all sucks. Hopefully someone comes up with actual working solution for this.

Oh, there’re another solutions:

  • Use REE 1.8 (really?)
  • Use JRuby in 1.8 mode
  • Use Rubinius (rbx 2.0 where?) in 1.8 mode

Um, yeah.

Update: I figured out how to “fix” it. Check it out in Moebooru (requires this).

Rails: read_multi and dalli

Be careful when using read_multi with dalli: it may return nil-valued key instead of the correct key.

The issue is tracked here and thanks to this I dropped the read_multi usage in moebooru and used the much simpler (and most likely slower) single fetch (per entry) instead. There’s alternative way to use it – do a read_multi and refetch whatever missing/nil-keyed but apparently I’m too lazy to do it.

Disabling Upstart Service in Ubuntu (11.04+)

Took me few weeks to find out that this one-liner does wonder:

echo manual >> /etc/init/mysql.override

(the line above is to disable mysql, obviously. And must be done as root)

The answer is on first hit (as of this post’s writing) of googling “ubuntu disable service” but you need to scroll down a bit and ignore shitload of crappy, outdated explanations to find that small gem.

Unfortunately doesn’t apply to previous LTS. Or does it?

zpool hourly status check

I’m setting up cron job for a storage server using ZFS. There’s zpool status -x but it returns “all pools are healthy” (or “no pools available”) on no error and prints errors to stdout (instead of stderr), rendering it annoying for cron job.

#!/bin/sh

set -e
set -u

zstatus="`zpool status -x 2>&1`"
case "${zstatus}" in
  "all pools are healthy"|"no pools available")
    return 0
  ;;
  *)
    printf "%sn" "${zstatus}" >&2
    return 1
  ;;
esac

Put it in a file (e.g. /root/bin/zpool-status-cron), make it executable, and add it to crontab.

bcrypt in Debian

WARNING: using method below will lock yourself out when using emergency console since whatever crypt it’s using surely doesn’t understand bcrypt (as I experienced myself). Additionally, this solution won’t add bcrypt support to other applications using crypt interface like proftpd unless it’s started by preloading libxcrypt.so first (also from my own experience).

As much as Drepper want to pretend bcrypt is wrong solution, it actually gives one benefit: ease of switch to Linux. Some systems use bcrypt by default or configurable to use it. On other case, there might be time where you need system’s (or applications using system’s) crypt to handle bcrypt passwords from external system (usually web applications).

It’s quite difficult to enable bcrypt support in RHEL based distro as there is no libxcrypt and pam_unix2 packages available. Thankfully it’s available in Debian (and derivatives) in package libpam-unix2.

The README.Debian says to modify files in /etc/pam.d but if I remember it correctly, it confused apt PAM handling system or whatever. Fast forward few weeks, I discovered a better way to use it by creating PAM configuration in /usr/share/pam-configs. Since it’s mostly equivalent to normal pam_unix, I just copy and modify the file using this (long-ass) oneliner sed:

sed -e 's/pam_unix.so/pam_unix2.so/g;s/^Name: Unix authentication$/Name: Unix2 authentication/;s/pam_unix2.so obscure sha512/pam_unix2.so obscure blowfish rounds=8/;s/ nullok_secure//' /usr/share/pam-configs/unix > /usr/share/pam-configs/unix2

Then execute pam-auth-update, select Unix2 authentication and deselect Unix authentication. Don’t forget to update passwords for all other users as well or they won’t be able to login since pam_unix2 doesn’t recognize sha based hashes.

Actually, change all other users password to use md5 first before replacing the PAM with pam_unix2.

Update 2012-04-01: Removed nullok_secure since it isn’t supported.

Update 2012-06-09: Added warning.

FreeBSD is Rolling Release (the ports)

Don’t get tricked by the “release” system. Apart of the base system, FreeBSD perfectly qualifies as rolling release. I guess it’s also why the binary package management sucked so badly. You won’t find how to upgrade certain packages using binary method in their Ports’ UPDATING page.

Here’s the example:

20120225:
  AFFECTS: users of archivers/libarchive
  AUTHOR: glewis@FreeBSD.org

  libarchive has been updated to version 3.0.3, with a shared library bump.
  This requires dependent ports to be rebuilt.

  # portmaster -r libarchive
  or
  # portupgrade -r archivers/libarchive

You would think the dependent packages got version bump to ensure their proper dependency – but they didn’t. Instead you had to recompile everything depending on it.

And then there’s another case:

20120220:
  AFFECTS: users of graphics/libungif
  AUTHOR: dinoex@FreeBSD.org

  libungif is obsolete, please deinstall it and rebuild all ports using
  it with graphics/giflib.

  # portmaster -o graphics/giflib graphics/libungif
  # portmaster -r giflib
  or
  # portupgrade -o graphics/giflib graphics/libungif
  # portupgrade -rf giflib

Of course, ArchLinux kind of managed to do it but that’s a purely binary rolling release Linux distro. The maintainer worked hard to ensure such kind of thing get handled properly by all their users which mostly use binary packages. FreeBSD on other hand tried to claim capable of both but it really isn’t (unless I missed something).

I’m intending to contact pkgng creator to ask his opinion about this but have yet to do it…

Removing Annoying Speaker Static Noise

I’m not sure which sound cards are exhibiting this problem but at least it is in my system (onboard Realtek HD – Intel DH61BE motherboard running Windows 7 x64). It’s been annoying me since like forever and finally tonight I decided to actually solve the problem.

As it turns out, the solution is quite simple: disable PC Beep channel. A quick google showed this hit quite a bit of people and apparently this is the reason (or at least related).

On related note, apparently I’ve did this before and then completely forgotten. This is why I wrote it this time.

nginx – gzip all the text

During my migration to other server, I recreated some of my configs and enabled gzip compression for most file types. Here’s the relevant config:

gzip on;
gzip_vary on;
gzip_disable "msie6";
gzip_comp_level 6;
gzip_proxied expired no-cache no-store private auth;
gzip_types text/plain text/xml application/xml application/json application/x-javascript text/javascript text/css;

It should cover most text-based content one will ever serve over the web. Probably.

On Comment System

I just migrated to Disqus – a fully managed comment system for websites. I saw it first used in Engadget. It didn’t work quite good at the time (or at least I didn’t have good memory on it) but it’s quite wonderful now.

One of the problem I’m having when leaving comment on other blogs is it’s difficult to track which posts and sites I’ve left a comment on. Some sites (like this site used to be) provide “Notify via E-mail on Replies” option but it’s clunky at best and with dozens, hundreds of sites out there it’s quite impossible to track them all. Not to mention you have to visit each blogs to unsubscribe from notification.

Then come Disqus – it’s a centralized comment system which allows any website to use their service and let the users enjoy one-stop interface to manage all their comments on various websites. It uses JavaScript to embed the comment interface on a page – not the best way but I guess it’s acceptable now with emergence of smartphones and tablets which actually capable of rendering JS.

Replacing WordPress’ comment system with Disqus is quite easy. The official WordPress plugin provides everything to migrate comments quickly and easily. It can even keep the comments synced with local database – allowing quick way out in case Disqus goes evil™.

I doubt anyone still read this blog (and blog is so 2009) but well, here it is.

Server changes, etc

Apparently now this blog is blazing fast. I’m not sure why but I sure can feel the difference.

I’ve done lots of changes on this site so I’m not sure which one did the trick most:

  • Not using WordPress Multisite
  • Moved to 64-bit OS
  • gzip-ed everything (er, most things)
  • Used MySQL 5.5

Yeah, those are the changes. Also this site moved to all my four VPS’. Last (this) one is in Hostigation. Hopefully the won’t have the network problem like the one occurred sometime last month anymore. I enabled collectd with ping plugin so I’ll have concrete data on what’s happened next time there’s network problem.

…and as it turns out, I set the expiration overly aggressive resulting in caching even the main page. Epic fail on my side.

nginx/php single config for SSL and non-SSL connection

This morning I noticed I haven’t upgraded WordPress MU Domain Mapping plugin to the latest version. It supposedly brings better SSL support. And after upgrading I couldn’t log in to my mapped domain blogs (e.g. this blog). Wasn’t it a great way to start my morning?

After some digging, I found out the problem was because I don’t have one PHP(?) parameter – HTTPS – passed properly. It should set to True whenever one is using SSL connection otherwise there’s no way the PHP process can know if the connection is secure or not. Previous version of WPMUDM have a bug in which skips SSL check but in turn enables using HTTPS even without such parameter. Decided it’s my fault (I believe it would completely breaks phpMyAdmin), adding the parameter then I did.

But it’s not that simple: I’m using unified config for both my SSL and non-SSL connection’s PHP include. Splitting the config would make the duplication worse (it’s already relatively bad as it is) so that’s not an option. Using the evil if is also not a solution since it doesn’t support setting fastcgi_param inside it.

Then the solution hit me. The map module – a module specifically made for things like this and to avoid usage of if. I tested it and indeed worked as expected.

Here be the config:

...
http {
  ...
  map $scheme $fastcgi_https {
    https 1;
    default 0;
  }
  ...
  server {
    ...
    location ~ .php$ {
      ...
      fastcgi_param HTTPS $fastcgi_https;
    }
...

And WordPress MU Domain Mapping is now happy.

Update 2012-02-20: nginx version 1.1.11 and up now have $https variable. No need to have that map anymore.