Virtually Mad - HackTheBox Challenge Writeup

Challenge Information

Description

Your friend loves to make pretty odd programs. This time you are given a special “machine” and you have to crack the correct code.

The challenge ships a single binary, virtually.mad, that hints at a custom virtual machine. No further service or network component is involved — the whole challenge is reversing the binary’s instruction set and feeding it the one input that satisfies its internal win condition.

Solution

The archive extracted a 0-byte virtually.mad at first glance — a red herring from how the artifact was staged. Re-extracting the source zip with the standard HTB archive password recovered the real file: a 14,472-byte, stripped x86-64 PIE ELF.

Disassembling it showed the binary is not a “crackme” with a hidden string comparison — it’s a tiny bytecode virtual machine. It reads a hex string from stdin, and the flag is whatever input satisfies the VM’s win condition; there is no flag string embedded anywhere in the binary to strings out.

The VM:

1. Requires strlen(input) % 8 == 0.
2. Splits the input into 8-hex-char (32-bit) opcodes.
3. Executes each opcode against four registers (a, b, c, d).
4. After execution, compares final register/flag state against a hardcoded win condition and, only if it matches, prints the flag built from the supplied input itself.

Reversing the opcode encoding and the win condition, then working backwards from the win condition to the unique 5-opcode program that produces it, yielded the correct input and the flag.

Key Steps

1. Recover the real binary from the source archive

# The staged artifact was 0 bytes — pull the real one out of the zip
unzip -o -P hackthebox /out/<uuid>.zip
cd rev-virtuallymad
file virtually.mad
# virtually.mad: ELF 64-bit LSB pie executable, x86-64, stripped

2. Static triage

strings -n 4 virtually.mad | head -60
objdump -d -M intel virtually.mad > /tmp/dis.txt
wc -l /tmp/dis.txt

No plaintext flag or obvious flag-comparison logic turned up — a strong signal this is an emulate-and-compare design rather than a straight string check.

3. Reverse the instruction encoding

Reading the dispatch table (1716aa) and the per-slot mask/validation table (jump table at 2118), each 32-bit opcode is 8 hex nibbles, read n7 → n0:

 n7 | n6     | n5 | n4       | n3   | n2..n0
 ---|--------|----|----------|------|-------------------
  0 | opcode | 1  | dest reg | mode | 12-bit value/index

opcode (n6): 1=MOV  2=ADD  3=SUB  4=CMP  5=HALT
dest reg (n4): 0=a  1=b  2=c  3=d
mode (n3): 0=immediate (n2..n0 = value)
           1=register  (n2 = source register index)

Constraints layered on top of this:

• n7 must always be 0.
• n5 must always be 1 — any other value trips perror("Bad instruction!").
• The jump table at 2118 pins the expected opcode and destination register per slot position — e.g. slot 0 must be ADD a, slot 2 must be SUB b, etc.
• Each handler independently re-validates n5 and the mode field.
• Crucially, main only actually executes an instruction if (op & 0xfff) <= 0x100 — otherwise the opcode is silently skipped even though it passed the mask check. This is the trap: a well-formed-looking opcode can do nothing at runtime.

4. Reverse the win condition

At 1a48, the program checks, after exactly 5 executed opcodes:

a == 0x200
b == -1
c == -1
d == 0
flags == 0x10000000

5. Work backwards to the unique 5-opcode solution

Since immediates are capped at 0x100, a == 0x200 can only be reached by two ADD a, #0x100 instructions — there’s no single-instruction path. From there the rest of the register/flag state pins the remaining three opcodes uniquely:

ADD a, #0x100      ; a = 0x100
ADD a, #0x100      ; a = 0x200          <- satisfies a == 0x200
SUB b, #1          ; b = 0 - 1 = -1     <- satisfies b == -1
MOV c, b           ; c = b = -1         <- satisfies c == -1
CMP d, #0          ; d unchanged (0), flags = 0x10000000

Encoded as the 8-nibble opcode stream (0 prefix, opcode, 1, dest reg, mode, value):

02100100  02100100  03110001  01121100  04130000

Concatenated and fed to the binary:

echo "0210010002100100031100010112110004130000" | ./virtually.mad
# HTB{REDACTED}

Tools Used

• unzip (password-protected archive recovery)
• file, strings
• objdump (Intel-syntax disassembly)
• Manual instruction-set reverse engineering / bytecode-VM analysis
• bash (crafting and feeding the winning opcode stream to the binary)

Key Learnings

• When a challenge artifact looks broken (e.g. a 0-byte file), check whether it was mis-staged before assuming a dead end — the real binary was one password-protected unzip away.
• Not every crackme hides the flag as a string; here the binary was a small custom VM and the flag was derived from the attacker’s own input only after it satisfied an internal win condition — strings/static-string hunting was a dead end by design.
• Validation logic can be layered and partially decoupled from execution: an opcode passing every format/mask check can still be silently dropped by an unrelated runtime gate ((op & 0xfff) <= 0x100 here). Always trace what actually executes, not just what validates.
• Working backwards from a small, fully-specified win condition (fixed register values, fixed instruction count) is often faster than brute-forcing forward through the instruction space — the immediate-value cap made the two-ADD path for a == 0x200 the only possibility, which pinned the rest of the program uniquely.