HILLarious - HackTheBox Challenge Writeup

Challenge Information

Description

During a typical workday, one of Blackink-corp’s servers was attacked. The cause? An executable that encrypted all the files of their new secret project. Amidst stress and confusion, the security team managed to recover the software used and the files retrieved from the compromised server. Your mission: assist the team in regaining control and recovering the project data.

Solution

The challenge archive looked deceptively empty at first — the extracted directory contained only a 0-byte ransom binary. The real payload was still locked inside the password-protected zip (unzip -P hackthebox), which unpacked to the actual 8996-byte ransom executable plus three .enc files that hadn’t been staged into the working directory at all: secret.txt.enc, notes.txt.enc, and document.txt.enc.

ransom turned out to be a UPX-packed, PIE ELF whose CLI only exposed an encrypt subcommand — no decrypt path was shipped, meaning the entire recovery logic had to be reverse-engineered from the encryption routine and re-derived by hand.

Reversing the packed binary

No upx binary or root access was available in the container, so the official static release was fetched directly and unpacked purely in Python (the tar xf route failed because the xz helper binary was missing, so lzma.open() + tarfile were used instead to decompress the release archive in memory).

Once unpacked, disassembly of the .text section around the encryption routine revealed the full scheme:

1. Each encrypted file starts with a 20-byte header: the ASCII marker "SNAR", an 8-byte little-endian timestamp, a 4-byte original length, and a 4-byte version.
2. Critically, the recovered file timestamps matched each plaintext file’s mtime exactly — meaning the key-derivation seed was stored in cleartext inside the encrypted file itself. This was the fatal flaw in the scheme.
3. From that seed, the key material is derived as:
   • h = FNV-1a-64(timestamp bytes) XOR 0xdeadbeefcafebabe
   • xor_key = the 8 little-endian bytes of h
   • a 4-element hill_key array is generated via an LCG (A = 0x5851f42d4c957f2d) seeded from h, with a fixup that increments k0 by 2 whenever the resulting Hill-cipher determinant (k0*k3 - k1*k2) comes out even — guaranteeing the matrix is invertible mod 256.
4. Plaintext is padded to an even length (confirmed by secret.txt being 47 bytes on disk but 48 bytes once encrypted).
5. The padded plaintext is run through a 2×2 Hill cipher mod 256.
6. The Hill cipher output is then XOR’d byte-wise against xor_key[i & 7] to produce the final ciphertext.

The giveaway that pinned down the cipher family was a small leaf function at offset 0x1340 that computed k[0]*k[3] - k[1]*k[2] and then ran an extended-Euclidean-algorithm loop with ecx = 0x100 — textbook code for computing a Hill-cipher determinant and its modular inverse mod 256.

Building the decryptor

With the full scheme recovered, decryption was just the inverse of each step, applied in reverse order:

1. Read the 20-byte SNAR header from each .enc file to recover the timestamp and original length.
2. Re-derive h, xor_key, and hill_key from the timestamp using the exact same FNV-1a-64 + LCG-with-fixup logic.
3. XOR the ciphertext bytes against xor_key[i & 7] to undo step 6 of encryption.
4. Compute the modular inverse of the 2×2 Hill matrix mod 256 (via the determinant’s modular inverse) and apply it to invert the Hill cipher.
5. Trim the result back down to the original length recorded in the header, discarding the padding byte.

Running this against secret.txt.enc (and the other two .enc files for corroboration) recovered the plaintext project data, including the flag.

Key Steps

# 1. The provided archive was password-protected — extract with the known password
unzip -P hackthebox -o challenge.zip -d /tmp/w
cd /tmp/w
file ransom
xxd secret.txt.enc | head

# 2. No upx binary or root in the sandbox — pull the static release and unpack
#    the .tar.xz purely with Python (system `xz` was unavailable)
cd /tmp
curl -sL -o upx.txz https://github.com/upx/upx/releases/download/v5.0.2/upx-5.0.2-amd64_linux.tar.xz
python3 -c "
import lzma, tarfile, io
d = lzma.open('upx.txz').read()
t = tarfile.open(fileobj=io.BytesIO(d))
t.extractall('/tmp/upx_extracted')
"

# 3. Unpack the UPX-packed ransomware binary
/tmp/upx -d -o ransom.unp ransom
file ransom.unp

# 4. Disassemble the encryption routine to recover the cipher scheme
objdump -d -M intel ransom.unp | less
readelf -sW ransom.unp

#!/usr/bin/env python3
# Decryptor derived from the reverse-engineered "SNAR" ransomware scheme.
import struct

FNV_OFFSET = 0xcbf29ce484222325
FNV_PRIME  = 0x100000001b3
XOR_CONST  = 0xdeadbeefcafebabe
LCG_A      = 0x5851f42d4c957f2d
MASK64     = (1 << 64) - 1

def fnv1a64(data: bytes) -> int:
    h = FNV_OFFSET
    for b in data:
        h ^= b
        h = (h * FNV_PRIME) & MASK64
    return h

def derive_keys(timestamp: int):
    ts_bytes = struct.pack('<Q', timestamp)
    h = fnv1a64(ts_bytes) ^ XOR_CONST
    xor_key = struct.pack('<Q', h)

    # LCG chain seeded from h produces the 4 Hill-matrix coefficients
    seed = h
    hill = []
    for _ in range(4):
        seed = (seed * LCG_A + 1) & MASK64
        hill.append(seed & 0xff)
    k0, k1, k2, k3 = hill

    # Fixup: nudge k0 until the determinant is odd (invertible mod 256)
    det = (k0 * k3 - k1 * k2) & 0xff
    while det % 2 == 0:
        k0 = (k0 + 2) & 0xff
        det = (k0 * k3 - k1 * k2) & 0xff

    return xor_key, (k0, k1, k2, k3), det

def modinv256(a: int) -> int:
    # Extended Euclidean algorithm mod 256 (mirrors the 0x1340 leaf function)
    a %= 256
    for x in range(256):
        if (a * x) % 256 == 1:
            return x
    raise ValueError("not invertible mod 256")

def hill_decrypt(block: bytes, key):
    k0, k1, k2, k3 = key
    det = (k0 * k3 - k1 * k2) & 0xff
    inv_det = modinv256(det)
    # Adjugate matrix, scaled by the modular inverse of the determinant
    i0 = ( k3 * inv_det) & 0xff
    i1 = (-k1 * inv_det) & 0xff
    i2 = (-k2 * inv_det) & 0xff
    i3 = ( k0 * inv_det) & 0xff

    out = bytearray()
    for i in range(0, len(block), 2):
        p0, p1 = block[i], block[i + 1]
        c0 = (i0 * p0 + i1 * p1) & 0xff
        c1 = (i2 * p0 + i3 * p1) & 0xff
        out += bytes([c0, c1])
    return bytes(out)

def decrypt_file(path: str) -> bytes:
    with open(path, 'rb') as f:
        data = f.read()

    magic, timestamp, orig_len, version = struct.unpack('<4sQII', data[:20])
    assert magic == b'SNAR'
    ciphertext = data[20:]

    xor_key, hill_key, _ = derive_keys(timestamp)

    # Undo step 1: XOR with the repeating 8-byte key
    xored = bytes(b ^ xor_key[i & 7] for i, b in enumerate(ciphertext))

    # Undo step 2: invert the 2x2 Hill cipher mod 256
    plaintext = hill_decrypt(xored, hill_key)

    return plaintext[:orig_len]

for fname in ['secret.txt.enc', 'notes.txt.enc', 'document.txt.enc']:
    recovered = decrypt_file(fname)
    print(f'--- {fname} ---')
    print(recovered.decode(errors='replace'))

# 5. Run the decryptor against all recovered .enc files
python3 decrypt.py
# secret.txt contains the flag: HTB{REDACTED}

Tools Used

• unzip — extracting the password-protected challenge archive
• upx (static v5.0.2 release, unpacked via Python lzma/tarfile) — unpacking the UPX-compressed PIE ELF
• objdump / readelf — disassembling and analyzing the packed binary’s encryption routine
• python3 — reimplementing FNV-1a-64, the LCG key schedule, and the 2×2 Hill-cipher matrix inversion mod 256 for the decryptor

Key Learnings

• Read the whole delivery, not just the obvious file. The working directory initially looked empty (a 0-byte ransom), but the real binary and all three ciphertexts were sitting inside the password-protected zip and simply hadn’t been staged — always fully extract and inventory an archive before assuming a challenge is broken or incomplete.
• A missing system tool is rarely a dead end. With no upx, no root, and no xz binary available, the static UPX release could still be fetched and decompressed entirely in Python using the standard lzma and tarfile modules — worth remembering as a general pattern for sandboxed reversing environments.
• Storing key-derivation material in cleartext defeats even a “secure-looking” custom cipher. The Hill cipher itself (with a forced-odd, invertible determinant mod 256) is a legitimate block cipher, but embedding the exact seed (a file timestamp) directly in the ciphertext header made the entire key schedule trivially reproducible — the classic ransomware-crypto mistake of conflating obfuscation with confidentiality.
• Determinant + extended-Euclid code is a strong fingerprint. Spotting a small leaf function computing k0*k3 - k1*k2 followed by an extended-Euclidean loop bounded by 0x100 was the single clue that identified the cipher as a 2×2 Hill cipher mod 256, cutting straight to the right decryption approach instead of guessing at the scheme from raw byte patterns.