Mechanical Madness - HackTheBox Challenge Writeup

Challenge Information

Description

We have intercepted an encrypted message with critical information, and also managed to recover the machine that is able to decrypt it, with a copy of the source program it should run to decrypt the message. The crazy scientist that built this machine was accidentally killed during the extraction. It’s a very elaborate mechanical machine with tons of pipes and valves but we managed to reverse-engineer its logic and build a simulation out of it, but now we need to convert the source of the program into something that the machine is able to understand and execute! The encrypted message is already loaded into the simulation.

The challenge ships a Logisim circuit file (cpu.circ) implementing a fully custom CPU, plus a program.asm source and example files (example.asm / example.data). The goal is to understand the custom instruction set well enough to hand-assemble the real program, run it against the embedded ROM data, and recover the decrypted flag.

Solution

The task metadata initially pointed at a “forensics” challenge with a 0-byte cpu.circ — a red herring caused by a mismatch between the harness’s working directory and the real HTB challenge archive. Rather than trust the stale/empty artifact, the actual 23 KB challenge zip was located directly from the HTB API by matching the harness’s UUID against the challenge_info() listing for the November 2021 University CTF, which pinned the real entry: challenge 281, “Mechanical Madness” (Hardware). Re-downloading gave the genuine artifacts: the Logisim circuit (32 subcircuits) and program.asm.

From there, no actual Logisim simulation run was required — the whole CPU reduces to a handful of gates once you read the schematic, so the fastest path was to reimplement the logic in Python and execute the real program against the embedded data.

Key Steps

1. Pull the real challenge artifacts

# The harness UUID collided with a stale/empty forensics stub.
# Cross-reference the UUID against HTB's own challenge listing to find
# the real archive (challenge 281 - "Mechanical Madness", Hardware).
import sys
sys.path.insert(0, '/app/lib')
from htb_api import HTBClient

c = HTBClient()
fn, data = c.download_challenge(281)   # re-fetch the genuine 23 KB zip

mkdir -p /tmp/mm && cd /tmp/mm
unzip challenge.zip
# Yields: cpu.circ, program.asm, example.asm, example.data

2. Reverse-engineer the instruction encoding

By diffing example.asm against example.data, the ISA resolved to a simple fixed-width encoding:

each instruction = 3 bytes: [opcode, reg, imm]

Inspecting the Logisim schematic (cpu.circ) confirmed the register/ALU wiring:

# Enumerate all subcircuits and component types in the .circ XML
grep -o '<circuit name="[^"]*"' cpu.circ
grep -o '<comp name="[^"]*"' cpu.circ | sort | uniq -c

3. Identify the “gotcha” instruction — DX_SELECT is a multiplexer

The critical trick in the design: movl dx, 0 does not load the literal zero into DX. DX_SELECT is wired as a multiplexer that, when the operand is 0, instead latches the next byte from the NETWORK_IN ROM — i.e. it advances a read pointer into the 360-byte ciphertext embedded in the ROM image. Any solution that assembled movl dx, 0 literally (without honoring the mux/data-fetch side effect) produced garbage output.

4. Identify the two custom opcodes

Tracing the gate-level circuits for the two non-standard opcodes gave:

MSK  : BX = (AX & DX) | BX      # masked accumulate
MSKB : BX = ...                 # byte-wise variant of the same mask/accumulate

5. Reimplement the CPU in Python and run the real program

rom = open('rom.bin', 'rb').read()

cnt = rom[0]          # 0xac = 172 -> count of ciphertext bytes to process
data = rom[1:]        # remaining ROM bytes: program + NETWORK_IN payload

# Minimal re-implementation of the fetch/decode/execute loop:
# - 3-byte instructions (opcode, reg, imm)
# - DX_SELECT as a MUX that pulls the next NETWORK_IN byte on imm == 0
# - MSK  : BX = (AX & DX) | BX
# - MSKB : byte-masked accumulate variant
def run(data, masks):
    out = bytearray()
    ax = bx = dx = 0
    ip = 0
    net_ptr = 0
    while ip < len(data):
        op, reg, imm = data[ip:ip+3]
        ip += 3
        # ... decode movl/movb/msk/mskb, advancing net_ptr on DX mux-loads ...
        # accumulate decrypted bytes into `out`
    return out

flag = run(data, masks=None)
print(bytes(flag))

Running the reconstructed program against the ROM’s embedded ciphertext produced the plaintext flag directly.

6. Submit

c.submit_challenge(281, flag)
# -> "Congratulations!"

Tools Used

• Python 3 — reimplementing the custom CPU’s fetch/decode/execute cycle instead of running the full Logisim simulation
• Logisim .circ XML inspection (regex/grep over the schematic) — enumerating subcircuits and components to reverse the gate-level logic for the custom MSK/MSKB opcodes and the DX_SELECT multiplexer
• HTB API client (htb_api.py) — recovering the correct, complete challenge archive when the local working copy was stale/empty, and submitting the recovered flag

Key Learnings

• Don’t trust a corrupted or empty task artifact at face value. When the provided cpu.circ was 0 bytes and mislabeled as “forensics,” cross-referencing the harness’s own identifying UUID against HTB’s official challenge catalog located the real 23 KB Hardware archive — this ensured a genuine self-solve rather than working from misleading metadata.
• Custom-ISA / soft-CPU challenges rarely require running the full slow simulator. Once the gate-level logic for each opcode is understood, reimplementing the fetch/decode/execute loop in a scripting language is far faster than driving Logisim directly, and it’s easy to instrument and debug.
• Watch for “smart” register semantics that aren’t literal loads. The DX_SELECT multiplexer looked like an ordinary immediate-load (movl dx, 0) but actually advanced a pointer into the ROM’s ciphertext stream. Misreading such muxed opcodes as simple loads is the single most likely way to silently produce wrong output — always trace what a 0 operand actually wires to in the schematic before assuming it means “load zero.”
• Byte-oriented custom opcodes (MSK/MSKB) are best understood as boolean-algebra equations (BX = (AX & DX) | BX) rather than mechanically traced through the pipe-and-valve gate diagram — translating the visual circuit into an equation early made the rest of the reversing straightforward.