STH Forums - Short Downtime Tonight

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Patrick

Administrator
Staff member
Dec 21, 2010
12,511
5,792
113
I just wanted to give folks a bit of notice that later tonight I am going to take the forums down for a few minutes (10-20).

Two items need to get done:
1. Cut over to new server (testing has it much faster)
2. Upgrade/ patches

I am headed to Las Vegas tomorrow morning and want to ensure STH is not running from there while I am fiddling with a few items.
 
  • Like
Reactions: eva2000

Patrick

Administrator
Staff member
Dec 21, 2010
12,511
5,792
113
NO! this place is like crack cocaine, don't take it from me.

Hah, kidding, good luck.
Not too much luck involved at this point. The new VM I am copying over every hour right now (via zfs send/ receive) and doing dry runs.

Probably the bigger issue is going to be DNS propagation. :)
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,511
5,792
113
Well -

1 minute to zfs send/ receive and verify
1 minute to update network config and reboot
1 minute to swap Route 53 DNS entries
2 minutes to update XenForo (it was up after this)
2 minutes to update the server

Not terrible at all.
 

MiniKnight

Well-Known Member
Mar 30, 2012
3,072
973
113
NYC
Am I reading right that you transferred 40GB in 1 minute?

Why did you edit the network config instead of having one ready to go and downloading it?
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,511
5,792
113
No only the last 50MB or so. I had been sending incremental streams to ensure that the downtime was as low as possible.
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,511
5,792
113
Totally doable from an architecture standpoint except the XF patches which you need to have the board go down for.
 

iq100

Member
Jun 5, 2012
68
3
8
Patrick, if you used gigatexal "two identical hosts" couldn't the XF patches be applied without any perceivable downtime? First bring down one server's XF forum sw to peform XF update. Then incremental copy of new forum posts, if any, from on-line server.

Totally doable from an architecture standpoint except the XF patches which you need to have the board go down for.
Have two identical hosts (hyper converged?) and then upgrade one whilst the other serves data to users. Then if all goes well bring the other online. Then take the other offline. And upgrade it. Like A/B testing?
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,511
5,792
113
Patrick, if you used gigatexal "two identical hosts" couldn't the XF patches be applied without any perceivable downtime? First bring down one server's XF forum sw to peform XF update. Then incremental copy of new forum posts, if any, from on-line server.
The XF updates often hit the database. So you end up with an non-updated web front end(s) and updated front end(s) hitting a database that is either in the non-updated or updated state. Meanwhile, people can be posting content. Likely would be OK but I thought a few minutes of closing the board down would be better than having to troubleshoot if something went wrong. From a user's standpoint, the downtime would have seen the board inactive (old server) until the new server was fully patched and then the DNS was switched to an active board on the new server. Alerts never went off during the changeover because the HTTPS service was responding from one of the two servers the entire time.

Unfortunately, XF still does not have a read-only mode which is what I would much rather use. The XF upgrade process on a single server actually brings down forums to an inactive state anyway, just so you do not get a user making a change with a XF (web) and database mismatch.
 

iq100

Member
Jun 5, 2012
68
3
8
gigatexal, no offense to Patrick intended. The concepts of "non downtime", high-availability, are of interest to many here. Is there a hardware/software design, aka Tandem Non-Stop, that achieves this promise? BTW spitball is a programming language :)
SPITBOL - Wikipedia, the free encyclopedia
CAP theorem - Wikipedia, the free encyclopedia

I was merely spitballing. I have no experience with true production updates and have full faith in Patrick and his update strategy for his website that we graciously are able to be a part of.
 
Last edited:
  • Like
Reactions: gigatexal