Callfuscated - HackTheBox Challenge Writeup

Challenge Information

Description

VM, MBA, OP, JI (Junk Instructions), what are these terms. Well, now you’re going to defeat them.

The prompt names four obfuscation techniques stacked on top of a crackme binary: a custom Virtual Machine, Mixed Boolean-Arithmetic (MBA) expressions, Opaque Predicates, and Junk Instructions. The goal is to recover a flag validated somewhere inside that VM.

Solution

1. Artifact Recovery

The staged crackme binary was a 0-byte placeholder. The real 59,528-byte ELF was located inside the session’s output zip, protected with the standard HTB archive password.

cd /out
unzip -lv a12c732e-e1d2-4f45-9b09-5b62cd822ec8.zip
# Password: hackthebox
unzip -P hackthebox a12c732e-e1d2-4f45-9b09-5b62cd822ec8.zip -d work
cd work/rev_callfuscated
chmod +x crackme

2. First Look — Callfuscation

The binary is non-PIE with a ~46 KB .text section for what should be a trivial “check my password” program — an immediate sign of heavy code-layout obfuscation.

strings crackme | grep -iE "correct|incorrect|flag|htb\{"
echo "AAAAAAAAAAAAAAAA" | timeout 10 ./crackme
objdump -d --start-address=0x401080 --stop-address=0x401400 -M intel crackme | head -120

Reading the disassembly linearly produced garbage. Every real basic block was only 1–3 instructions long and ended in a call X, and the byte immediately following each call was a pop r8 instruction — but that pop r8 belonged to a completely different call chain and was never actually executed as a “next instruction.” This is the “callfuscation” referenced by the challenge name:

• Each call acts as an unconditional goto to the callee.
• The callee’s first instruction (pop r8) discards its own return address off the stack, so the call never truly “returns” in the conventional sense — it’s used purely for control-flow redirection, defeating naive linear/recursive disassembly and confusing tools that assume call implies a paired ret.

Objdump’s linear sweep interleaves bytes from unrelated chains, so a straight top-to-bottom read of .text is meaningless without following the actual call targets.

3. Building a Chain-Follower Disassembler

To recover real control flow, a small Capstone-based “chain follower” was written: start at an entry address, disassemble one real instruction, then jump to the target of its trailing call (skipping the dead pop r8 byte that never executes), and repeat.

# lin.py — re-linearize callfuscated code by following call targets
import capstone
from elftools.elf.elffile import ELFFile

f = open('crackme', 'rb')
elf = ELFFile(f)
# Load .text bytes + base address for capstone disassembly
text = elf.get_section_by_name('.text')
code = text.data()
base = text['sh_addr']

md = capstone.Cs(capstone.CS_ARCH_X86, capstone.CS_MODE_64)
md.detail = True

def disasm_chain(start, limit=4000):
    addr = start
    out = []
    for _ in range(limit):
        off = addr - base
        insns = list(md.disasm(code[off:off+16], addr, count=1))
        if not insns:
            break
        insn = insns[0]
        out.append(insn)
        if insn.mnemonic == 'call':
            # Follow the call target directly as the "next" real instruction —
            # the callee's leading `pop r8` is dead code from another chain.
            target = int(insn.op_str, 16)
            addr = target
        elif insn.mnemonic.startswith('j') or insn.mnemonic == 'ret':
            break
        else:
            addr += insn.size
    return out

Running this against main’s entry point (0x409002 / 0x409fbb) produced a coherent, readable instruction stream instead of noise, revealing that main builds:

• A 586-dword bytecode array (the VM program).
• A 1000-dword operand stack, allocated in main’s own stack frame.

4. Understanding the VM and the Noise Trick

The program calls srand(0x539) early, then repeatedly calls rand(). At first these values looked like meaningless entropy scattered through MBA-obfuscated helper functions — until the pattern became clear: the random values are fed in as “noise operands” that the surrounding MBA arithmetic is engineered to cancel out exactly. They’re a deliberate red herring meant to make static MBA simplification (e.g. via SMT solvers) produce wrong results unless the exact rand() sequence is reproduced.

Rather than hand-simplify each MBA expression, the whole binary was emulated with Unicorn, stubbing out libc calls in Python:

# emu2.py — full-binary emulation with libc stubs, tracing the VM dispatch loop
import struct, json
from unicorn import *
from unicorn.x86_const import *
from elftools.elf.elffile import ELFFile

# ... map ELF segments into Unicorn, set up stack ...

def hook_code(uc, address, size, user_data):
    # Log (pc, opcode, sp, stack-top) every time execution reaches
    # the VM's opcode dispatch head, so opcode semantics can be
    # inferred later purely from stack deltas.
    if address == DISPATCH_HEAD:
        opcode = uc.mem_read(OPCODE_PTR, 4)
        sp = uc.reg_read(UC_X86_REG_RSP)
        trace.append((address, struct.unpack('<I', opcode)[0], sp,
                      list(uc.mem_read(STACK_BASE, 16))))

def hook_puts(uc, ...): ...   # stub puts()
def hook_printf(uc, ...): ... # stub printf()
def hook_scanf(uc, ...): ...  # feed our candidate input
def hook_rand(uc, ...):
    # Re-implement glibc's TYPE_3 additive-feedback generator
    # (seeded via srand(0x539)) so rand() output matches exactly —
    # required for the MBA noise-cancellation to line up.
    ...

Logging (pc, opcode, stack-pointer, stack-contents) at the dispatch head and diffing consecutive states let the VM’s opcode table fall straight out — no manual MBA algebra needed:

5. Recovering the Check and Building the Key

With opcode semantics known, the 586-dword bytecode program was decoded as a sequence of arithmetic/logic operations applied to input bytes, culminating in comparisons against a set of constant “key” dwords baked into the binary. A small solver reconstructed the transform and the expected constant table:

# Recover byte-wise key material from the traced VM operations
def u(x):
    return x & 0xffffffff

# Constants extracted from the decoded VM bytecode / trace
K = [152372026, 1115519090, 808520192, 704982578,
     -805966625, 420741918, -151533907, 1094795585, ...]

# The VM's xor/add/sub chain per input byte was inverted here to
# derive the required plaintext bytes that satisfy every check.

Feeding the derived candidate string back into the real binary confirmed the solve:

echo 'HTB{REDACTED}' | ./crackme
# -> Correct.

6. Flag Submission

O=/out/<run>
echo 'HTB{REDACTED}' > "$O/flag.txt"

Submitted via the HTB API and accepted:

{"message": "Congratulations!"}

Tools Used

• Capstone — disassembly engine used to build the custom chain-follower that defeats the call-based control-flow obfuscation.
• Unicorn Engine — full-binary CPU emulation to sidestep hand-solving MBA expressions and reproduce rand()-based noise cancellation exactly.
• pyelftools — ELF section/segment parsing to locate .text, .rodata, .data and drive both the disassembler and the emulator’s memory map.
• objdump / readelf — initial static triage of sections, PLT, and relocations.
• Python — glue for tracing, opcode-semantics inference from stack diffs, and final key-recovery/solver logic.

Key Learnings

• “Callfuscation” defeats naive disassembly by abusing call/ret semantics: a call is used purely as an unconditional jump, and the callee immediately pops its own return address (pop r8) so it never returns conventionally. Any tool assuming call-implies-return, or reading .text linearly, produces nonsense. The fix is a small custom disassembler that follows actual call targets rather than falling through byte-by-byte.
• rand() can be weaponized as MBA “noise” that only cancels out under the exact seeded sequence. Static/symbolic MBA simplifiers that don’t reproduce the PRNG state exactly will derive incorrect simplified expressions. Full concrete emulation (Unicorn) with a faithful glibc rand() re-implementation sidesteps this entirely.
• Dynamic tracing beats static algebra for custom VM opcodes. Rather than reverse-engineering each MBA-obfuscated VM handler by hand, logging (pc, opcode, stack-pointer, stack-contents) at the dispatch loop and diffing consecutive trace entries revealed the entire opcode table (push/drop/add/sub/mul/or/xor/load) almost for free.
• Challenge category labels can mislead. Despite being tagged as a forensics/misc-style task, this was a pure reversing challenge — worth confirming the actual binary/artifact type early rather than trusting the header.
• Artifact staging quirk: the crackme binary was delivered as a 0-byte placeholder with the real file inside a password-protected session zip — always check the zip archive when a staged binary looks empty or truncated.