Category: Administration

  • Apache 2.4.64, SNI, and 421 Misdirected Request: cause and fix

    After upgrading to Apache 2.4.64 (or after applying linux vendors updates) many sites behind a reverse proxy started returning 421 Misdirected Request. The practical cause is simple: when your proxy makes an HTTPS connection to Apache, Apache now expects a correct TLS SNI value that matches the HTTP host. If the proxy omits SNI or sends a different name than the Host header, Apache can reject the request with 421. This change exposed default proxy settings that used to work by accident.

    What the 421 error means

    The HTTP 421 Misdirected Request status means the request reached a server that is not configured to respond for the combination of scheme and authority that you used. Browsers often reuse HTTP/2 connections across hosts if the server allows it, but Apache 2.4.64 tightened the checks so that connections without a matching SNI and Host are rejected. That is why you can see the error only on some requests or only on some subdomains after the update. The error message is the following:

    Misdirected Request

    The client needs a new connection for this request as the requested host name does not match the Server Name Indication (SNI) in use for this connection.

    Who is affected

    You are affected if a reverse proxy connects to Apache over HTTPS and does not pass SNI to the backend. This is common for nginx in front of Apache, HAProxy with HTTPS backends, and some CDNs or load balancers that re-encrypt to the origin. Panels that pair nginx and Apache (for example EA-Nginx on cPanel, Plesk, Hestia, and similar) were among the first to report 421 after the update.

    Fix: send the correct SNI to Apache and keep it consistent with Host

    The fix is to enable SNI on the upstream HTTPS hop and make its value match the HTTP Host that Apache should serve. Keep the connection private (VPC, VLAN, VPN, or same host) if you do not want the name visible outside your network. Do not roll back the Apache security update. Configure your proxy once and the 421 errors stop.

    nginx to Apache over HTTPS

    Enable SNI toward the backend and keep Host consistent.

    location / {
        proxy_pass              https://backend.example.com;
        proxy_set_header        Host $host;
        proxy_ssl_server_name   on;      # send SNI
        proxy_ssl_name          $host;   # SNI value
    [..]
    }

    HAProxy to Apache over HTTPS

    Pass the Host header as the SNI value on the TLS connection to Apache. Add verification settings that match your policy.

    backend app
        http-request set-header X-Forwarded-Proto https
        server app1 127.0.0.1:443 ssl sni req.hdr(Host)

    Apache to Apache with mod_proxy over HTTPS

    Use an https URL whose hostname is the site you want to serve so that Apache sends that name as SNI to the backend.

    SSLProxyEngine on
    ProxyPass        / https://site.example.com/
    ProxyPassReverse / https://site.example.com/

    How to test and confirm

    From the proxy host, curl the backend using the public hostname and the backend IP to confirm that SNI and Host agree. Watch your Apache error log for messages about missing SNI or a vhost mismatch while you test.

    curl -vk https://example.com/ --resolve example.com:443:127.0.0.1

    FAQ

    The 421 you see is produced by the stricter SNI and authority checks added around 2.4.64 and enforced by HTTP/2 handling. The cure is to send a correct SNI on the upstream connection and keep it consistent with Host.

    Do I need to verify the backend certificate for SNI to matter? No. Apache uses SNI during the TLS handshake to pick the correct SSL vhost and certificate before it even reads the HTTP request. Certificate verification by the proxy is a separate operational choice. The important part for avoiding 421 is that the SNI you send and the Host header you send resolve to the same Apache vhost, precisely the ServerName and ServerAlias directives. Matching the SNI with a domain in a certificate that is missing in the apache vhost directives will also fail.

  • Minecraft Linux Server Auto-Update

    Every time Mojang ships a new Bedrock build, your server goes stale. Clients update themselves, refuse to connect to an older server, and you are back to SSH, hunting down a zip, and copying files over your install while trying not to touch your worlds folder.

    This script does that for you. It checks the official download API, does nothing at all if you are already current, and if you are not, it swaps in the new server files while leaving your worlds and your configuration alone. Bedrock Dedicated Server on Linux only, not Java Edition.

    What the script does

    On every run it:

    1. Asks the official Minecraft download API for the current Linux Bedrock build.
    2. Compares that version against a small version file inside your server folder.
    3. Exits immediately if they match. Nothing is downloaded, nothing is touched.
    4. Otherwise downloads the zip to a temp folder, tests it for corruption, and confirms the server binary is in there.
    5. Copies the new files over your install with rsync, skipping your configs and your worlds.
    6. Records the new version number and cleans up.

    What it does not do:

    • It does not restart your server. The files get replaced, but the running process keeps serving the old version until it restarts. There is a section on this below.
    • It does not back up your worlds. It never touches them, but take backups anyway.
    • It does not remove files that disappeared from newer releases.

    Run it as the user that owns the server files. Nothing here needs root.

    Step 1: Install Dependencies

    sudo apt update
    sudo apt install curl jq unzip rsync wget

    Step 2: Create the Update Script

    Save the following as ~/update-bedrock.sh:

    #!/usr/bin/env bash
    set -euo pipefail
    
    HOME_DIR="$HOME"
    INSTALL_DIR="$HOME_DIR/bedrock-server"
    TMP_DIR="$HOME_DIR/bedrock-tmp"
    API_URL="https://net-secondary.web.minecraft-services.net/api/v1.0/download/links"
    ZIP_NAME="bedrock.zip"
    
    command -v jq >/dev/null || { echo "jq is required"; exit 1; }
    
    echo "Fetching version info..."
    DATA=$(curl -fsSL "$API_URL")
    DOWNLOAD_URL=$(echo "$DATA" | jq -r '.result.links[] | select(.downloadType == "serverBedrockLinux") | .downloadUrl')
    VERSION=$(basename "$DOWNLOAD_URL" | grep -o '[0-9.]\+')
    
    [[ -z "$DOWNLOAD_URL" || -z "$VERSION" ]] && { echo "Failed to extract URL/version"; exit 1; }
    
    [[ -f "$INSTALL_DIR/bedrock-server.version" && "$(cat "$INSTALL_DIR/bedrock-server.version")" == "$VERSION" ]] && {
      echo "Already up to date"
      exit 0
    }
    
    mkdir -p "$TMP_DIR"
    wget -q -O "$TMP_DIR/$ZIP_NAME" "$DOWNLOAD_URL"
    unzip -t "$TMP_DIR/$ZIP_NAME" >/dev/null || { echo "ZIP test failed"; exit 1; }
    unzip -oq "$TMP_DIR/$ZIP_NAME" -d "$TMP_DIR/unpacked"
    
    [[ -x "$TMP_DIR/unpacked/bedrock_server" ]] || { echo "Missing server binary"; exit 1; }
    
    rsync -a \
      --exclude='autoupdate.sh' \
      --exclude='server.properties' \
      --exclude='permissions.json' \
      --exclude='whitelist.json' \
      --exclude='ops.json' \
      --exclude='allowlist.json' \
      --exclude='valid_known_packs.json' \
      --exclude='worlds/' \
      "$TMP_DIR/unpacked/" "$INSTALL_DIR/"
    
    echo "$VERSION" > "$INSTALL_DIR/bedrock-server.version"
    rm -rf "$TMP_DIR"
    echo "Updated to version $VERSION"

    How it works

    The API returns a plain JSON list of the current download links, one per platform. The jq filter picks the entry tagged serverBedrockLinux. Swap that string for serverBedrockPreviewLinux if you want to follow preview builds, and keep preview builds away from worlds you care about. This is why the script does not scrape the download page. The old scraping scripts all broke when bot protection went up in front of it, and the API does not care what your user agent is.

    bedrock-server.version is a one line file the script writes into your server folder. It is not part of the official zip and the server never reads it. It is the script’s only memory, and it is what makes a daily cron job cheap: on a server that is already current, the entire run is one HTTPS request and an exit. Delete that file to force a clean reinstall of the current version.

    Everything downloads into a temp folder, never into the live server folder. unzip -t tests the archive before a single file is copied, so a truncated download or a full disk dies there instead of halfway through overwriting your server binary.

    Then rsync copies the new files in, and the excludes are what keep your server yours:

    • server.properties is your port, difficulty, gamemode, view distance, and everything else you tuned. The zip ships a default copy that would flatten it.
    • permissions.json and ops.json are your operator lists.
    • allowlist.json is the allowlist, and whitelist.json is what it used to be called. Both are excluded, so this works on old installs and new ones.
    • valid_known_packs.json lists the packs the server knows about. The server rewrites this file itself. If a release ever makes it complain about packs, delete the file and let it rebuild.
    • worlds/ is your saves. The one that matters.

    There is no --delete, so files that vanish from newer releases just stay behind. That wastes a little disk and it avoids the entire class of accident where one wrong exclude pattern eats a world.

    One quirk worth knowing: the character class in grep -o '[0-9.]\+' also matches the dot in front of zip, so the version comes out as 1.26.33.2 with a trailing dot on the end. Nothing breaks, because the same expression writes the version file and reads it back. If it bothers you in the log, use grep -oE '[0-9]+(\.[0-9]+)+' instead.

    Step 3: Make It Executable

    chmod +x ~/update-bedrock.sh

    Run it once by hand before you trust it to cron. On a clean box it doubles as the installer: the server folder does not exist yet, so rsync creates it and you never have to download the zip yourself.

    Step 4: Automate with Cron (Optional)

    Edit your crontab with crontab -e and add this line to check daily at 3 AM:

    0 3 * * * ~/update-bedrock.sh >> ~/bedrock-update.log 2>&1

    Put that in the crontab of the user that owns the server, not root. The script builds every path from $HOME, and root’s home is /root.

    Restart the server after an update

    This is the part that catches people out. rsync renames each new file into place, so replacing files under a live process does not crash it, but the running server keeps its old binary open. It carries on serving the old version, players still get told the server is out of date, and the script has already cheerfully printed Updated.

    The update is not finished until the server process restarts.

    Assuming your server runs as a systemd service called bedrock, add this just above the rsync block:

    SERVICE="bedrock"
    RESTART=0
    if systemctl is-active --quiet "$SERVICE"; then
      echo "Stopping $SERVICE..."
      sudo systemctl stop "$SERVICE"
      RESTART=1
    fi

    And this at the very end, after the version file is written:

    if [[ "$RESTART" == 1 ]]; then
      echo "Starting $SERVICE..."
      sudo systemctl start "$SERVICE"
    fi

    Because the script already exited earlier when there was nothing to do, this only ever fires on a real update. Your server is not bounced nightly for no reason.

    For cron to run those two commands without a password, add a narrow sudo rule with sudo visudo -f /etc/sudoers.d/bedrock-update, using your own username:

    minecraft ALL=(root) NOPASSWD: /usr/bin/systemctl stop bedrock, /usr/bin/systemctl start bedrock

    The server is stopped at that point anyway, which makes it the right moment for a world backup. Drop this in right after the stop:

    tar -czf "$HOME_DIR/worlds-$(date +%F).tar.gz" -C "$INSTALL_DIR" worlds

    Checking it worked

    What your install thinks it is running:

    cat ~/bedrock-server/bedrock-server.version

    What is actually current:

    curl -fsSL https://net-secondary.web.minecraft-services.net/api/v1.0/download/links \
      | jq -r '.result.links[] | select(.downloadType=="serverBedrockLinux") | .downloadUrl'

    Troubleshooting

    Failed to extract URL/version. The API is unreachable, or its JSON changed shape. Run the curl above by hand and look at what comes back. Your install has not been touched.

    ZIP test failed. Truncated download or a full disk. Check df -h and run it again.

    It says Updated but players still see the old version. The server was never restarted.

    Permission denied during the rsync. You ran the script as the wrong user. If you ran it once as root, fix the ownership with sudo chown -R minecraft:minecraft ~/bedrock-server.

    Cron never runs it. Check that the script is executable and that crontab -l shows the line under the right account. An empty log usually means the crontab belongs to another user.

    Done

    Your server now spots new releases on its own, refuses to touch anything if the download looks wrong, keeps your worlds and settings across every update, and comes back up on the new version without you logging in. Check the log now and then to make sure it is still saying what you expect.

  • Apache PROXYPASS, NEGATIVE PROXYPASS AND AUTH_BASIC

    Today i had to face a weird problem with Apache 2. I wanted to setup a webmail on the SAME virtualhost that i was using to proxy to another host.
    Here’s a summary of my configuration:

    <VirtualHost *:80>

    ServerAdmin sysadmin@localhost

    DocumentRoot /var/www/folder

    ServerName localhost

    Alias /mail /var/lib/roundcube/
    <Directory /var/lib/roundcube/>

    Options Indexes Includes FollowSymLinks

    AllowOverride All

    AuthType Basic

    AuthUserFile /var/lib/roundcube/.htpasswd

    AuthName “Protected Folder”

    require valid-user

    </Directory>
    ProxyRequests Off

    ProxyPreserveHost On
    <Proxy *>

    Order deny,allow

    Allow from all

    </Proxy>
    ProxyPass /mail/ !

    ProxyPass / http://0.0.0.0/ ttl=60 retry=0 status=I keepalive=on timeout=2500 disablereuse=on
    </VirtualHost>

    The problem is that the auth_basic wasn’t working correctly in this setup, Apache was answering with a 200 instead of a 401 message, which prevented the browser from understanding it was actually an authentication..

    But this config was working fine without the auth, the webmail was working. And it was working fine with auth but no proxypass.
    So what was wrong?! Thanks to the guys @freenode i discovered that Apache was proxying the requests to custom errors in /error/ (as i uncommented the custom errors in apache2.conf). The solution was to add:

    ProxyPass /error/ !

    Turn loglevel to debug in case you have a similar issue, in my case i could read this:

    [Fri Mar 04 15:44:36 2011] [debug] mod_proxy_http.c(56): proxy: HTTP: canonicalising URL //10.10.10.10/error/HTTP_UNAUTHORIZED.html.var
    [Fri Mar 04 15:44:36 2011] [debug] proxy_util.c(1506): [client 1.1.1.1] proxy: http: found worker http://10.10.10.10/ for http://10.10.10.10/error/HTTP_UNAUTHORIZED.html.var

    Hope it helps.

  • MySQL: Restore a dropped database

    NOTE BEFORE TRYING ANYTHING DO A BACKUP OF ALL DATABASES. 1 MISTAKE CAN BE A BIG PROBLEM, IMAGINE 2 MISTAKES AT THE SAME TIME!!

    You need bin logs for this tutorial, if you didn’t have them enabled, then you have to look somewhere else..

    Today i accidentally “dropped” the database of a site i host thanks to my lazyness and the use and the brilliant and intuitive software called phpmyadmin, which i usually NEVER use.

    What i did is a simple drop database mysite;

    I had a backup from may 21 at 07:40 (check carefully the filetime of your last backup).

    Then i went to /var/log/mysql/

    Did ls -la to check the time of the logs:

    -rw-rw—-  1 mysql adm   66005373 mai 21 06:25 mysql-bin.001036
    -rw-rw—-  1 mysql adm   38683086 mai 22 06:25 mysql-bin.001037
    -rw-rw—-  1 mysql adm   48038277 mai 23 06:25 mysql-bin.001038
    -rw-rw—-  1 mysql adm   40780613 mai 24 06:25 mysql-bin.001039
    -rw-rw—-  1 mysql adm   42856471 mai 25 06:25 mysql-bin.001040
    -rw-rw—-  1 mysql adm   48990369 mai 26 06:25 mysql-bin.001041
    -rw-rw—-  1 mysql adm   54580611 mai 27 06:25 mysql-bin.001042
    -rw-rw—-  1 mysql adm   49505245 mai 28 06:25 mysql-bin.001043
    -rw-rw—-  1 mysql adm   57696248 mai 29 06:25 mysql-bin.001044
    As i knew i deleted the database on may 29 near 18:10, i ran this command:
    mysqlbinlog –start-datetime=”2010-05-21 07:40:00″ –stop-datetime=”2010-05-29 18:05:00″ -d mysite mysql-bin.0010* > /tmp/replay.sql
    Then i recreated my dropped database:
    create database mysite;
    Put the old backup:
    mysql -umysite -pmysite mysite < mysite.sql
    And applied the replay sql right after:
    mysql -umysite -pmysite mysite < /tmp/replay.sql
    And everything came back magically! Hope it helps!