vvm - HackTheBox Challenge Writeup

Challenge Information

Description

A new startup claims to have developed an unbreakable client-side password validation system. VCs have invested millions, but I’m a bit skeptical of their claim. Can you prove them wrong?

The challenge ships a single stripped ELF64 PIE binary, vvm, that reads a password from stdin and prints Correct! or Incorrect. Under the hood it’s not a normal password check at all — it’s a small custom bytecode VM that self-decrypts its own handler table at startup and then interprets a bespoke instruction stream against the input.

Solution

1. Recovering the real artifact

The provided vvm binary initially appeared as a 0-byte file. Since the session/challenge identifier matched HTB’s internal challenge.file_name for this challenge, the real artifact was pulled directly via the HTB challenge API (challenge id 743, “vvm”, Reversing/Hard) and unzipped, yielding the actual ~23 KB stripped ELF64 PIE.

2. Self-decrypting VM bootstrap

Static analysis showed the binary’s .data section holds 24 opcode-handler stubs, each XOR-encrypted with a short repeating key:

# XOR key recovered from the binary's decryption loop
key = bytes([0x2a, 0x0b, 0x21, 0x21, 0x4d, 0x2a])

At startup the binary:

1. XOR-decrypts each of the 24 stubs with this key.
2. mmaps an RWX region and copies the decrypted handlers into it.
3. Builds a malloc(0xe0)-sized context/table of 28 function-pointer slots (opcode handlers).
4. Enters a dispatch loop that reads a dword opcode from the bytecode stream and does:

call [ctx + opcode*8]

Opcode 0x1c is the VM’s RET/halt instruction.

3. Ground-truthing the instruction set in gdb

Statically pairing the compiler-scheduled assembly to VM opcodes was unreliable — the compiler hoisted stores past the following lea, causing naive static pairing to mis-map opcode slots to the wrong handler code. Instead, the ISA was recovered dynamically:

# break at the dispatch loop, dump the live handler table
break *dispatch_loop_addr
run <<< "test"
# read the ctx pointer out of .bss, then dump all 28
# function pointers + their decrypted code bytes

The 28 function pointers and their machine code were extracted from the live process, then disassembled with Capstone. This produced a ground-truthed picture of the ISA: a stack-based VM operating on a qword-wide stack and dword-wide code words, supporting instructions such as PUSH, PICK, DUP, CALL, JNZ, LOADB, SHL, SHR, OR, PACK, plus VM-level getline/ptrace (used for anti-debugging) primitives.

4. Disassembling the bytecode program

With the ISA known, the 808-dword bytecode program embedded in the binary was extracted and a custom disassembler written for it:

import struct

data = open("/tmp/vvm_x/rev_vvm/vvm", "rb").read()
prog_off = 0x4000  # offset of the bytecode blob inside the binary
words = struct.unpack_from("<808I", data, prog_off)

# decode each dword into (opcode, operand) using the
# ground-truthed opcode table from step 3
for w in words:
    opcode = w & 0xff
    operand = w >> 8
    # ... dispatch to per-opcode decoder ...

Decoding the program revealed the validation logic:

• The password length is checked obfuscated as (24L − 8) / 10 == 76 — algebraically just strlen(input) == 32.
• The 32 input bytes are shuffled/permuted according to a fixed schedule baked into the bytecode.
• The shuffled bytes are folded through a mixing function (rotate + XOR/PACK operations) into a small set of 32-bit checksum values.
• Those checksums are compared against constants embedded in the program.

5. Recovering the mixing function and inverting it

The mixing routine used a right-rotate/XOR pattern typical of custom checksum functions:

M = 0xffffffff

def rotr(v, n):
    n &= 31
    return ((v >> n) | (v << (32 - n))) & M

# target checksums extracted from the VM's embedded constants
cs = [706975780, -1972114847, -1170424423, ...]  # (recovered from bytecode)

# with the shuffle order and mixing function known, the
# 32-byte permutation was inverted / brute-forced per block
# to recover the plaintext bytes satisfying each checksum

Since the shuffle was a fixed, known permutation and the mixing function was invertible (or small enough to brute-force per byte block), the original 32-character password could be reconstructed byte-by-byte to satisfy all embedded checksum constants.

6. Validating and submitting

The candidate password was tested directly against the real binary:

echo 'HTB{REDACTED}' | ./vvm
# -> Correct!

The flag was then submitted via the HTB API and confirmed correct.

Key Steps

1. Recovered the real (non-zero-byte) vvm ELF via the HTB challenge API when the provided artifact was empty.
2. Identified the binary as a self-decrypting bytecode VM: .data handler stubs XOR’d with a 6-byte repeating key, unpacked into an RWX region at runtime.
3. Ground-truthed all 28 opcode handlers dynamically in gdb (dumping the live decrypted function-pointer table) rather than trusting static compiler-scheduled asm, which mis-paired stores/opcodes.
4. Wrote a custom disassembler for the 808-dword embedded bytecode program using the recovered ISA (stack VM: PUSH/PICK/DUP/CALL/JNZ/LOADB/SHL/SHR/OR/PACK, plus anti-debug ptrace checks).
5. Decoded the validation logic: obfuscated strlen == 32 check, fixed byte-shuffle, then a rotate/XOR mixing function reduced to checksums compared against embedded constants.
6. Inverted the shuffle + mixing function to reconstruct the 32-byte password satisfying all checksums.
7. Verified the recovered password against the live binary (Correct!) and submitted the flag.

Tools Used

• objdump (Intel-syntax disassembly of the host binary)
• gdb + Python scripting (dynamic dump of decrypted opcode-handler table)
• capstone (disassembling the dumped handler machine code)
• Python (custom bytecode disassembler, checksum inversion/brute-force)
• HTB internal API (HTBClient) to recover the real challenge artifact and submit the flag
• pyelftools (ELF section inspection)

Key Learnings

• Don’t trust static pairing of scheduled assembly. Compilers can hoist stores past nearby instructions that look like their “obvious” pair (e.g., a lea immediately following a store isn’t necessarily related to it); when a VM’s opcode table is built at runtime, dump it live in a debugger rather than inferring the mapping statically.
• Self-decrypting handler tables are just XOR-obfuscation with extra steps. Once the key and RWX-mmap pattern are identified, the “unbreakable” custom VM reduces to a normal reversing problem: recover the ISA, then disassemble the bytecode.
• Obfuscated arithmetic checks are often trivially algebraic. (24L − 8) / 10 == 76 is just strlen == 32 wrapped in noise — always simplify these expressions before treating them as meaningful logic.
• Shuffle + checksum designs are frequently invertible. If the permutation is fixed and the mixing function is a small set of rotate/XOR operations, the “one-way” checksum can often be inverted or brute-forced per block rather than needing to search the full password space.
• Zero-byte or missing artifacts can sometimes be recovered directly from the platform API using identifiers embedded in the task metadata, rather than treating a broken download as a dead end.