Detecting DOSBox via the Callback Opcode

A December 2025 post on datagirl-xyz by Snow, walking through how to reliably detect that a DOS program is running inside DOSBox — without relying on something brittle like the BIOS version string at FE00:0061 or the existence of the Z: drive. The approach exploits a custom x86 instruction that DOSBox invented for itself.

The setup

DOSBox is an MS-DOS emulator that doesn't really contain DOS or a BIOS in any conventional sense — virtual COM files like MOUNT.COM are how DOSBox's host functionality reaches the guest. Disassembling MOUNT.COM shows ordinary x86 (a stack-shrink, an INT 21h) followed by what looks like garbage:

0000000A  FE 38 05 00       ; ndisasm: db 0xfe / cmp [di],al / ...

That sequence isn't garbage — it's a callback into the emulator. In src/cpu/core_normal/prefix_none.h, DOSBox decodes the FE opcode group: /0 is the real INC Eb, /1 is the real DEC Eb, and /7 is the DOSBox-only CallBack, which reads a 16-bit callback ID from the next two bytes and dispatches into the host. So FE 38 00 00 runs callback 0x0005 and falls through as if it were a four-byte NOP.

The detection

On any real x86 from the 80186 onward, an unknown opcode raises #UD (interrupt 06h). So the test is symmetric to DOSBox's behavior:

1. Install a custom INT 06h handler.
2. Execute db 0xfe, 0x38, 0x00, 0x00 (a callback to ID 0).
3. If the handler fires, advance IP past the four-byte sequence and zero AX. Restore the previous handler.
4. After the sequence, inspect AX — if it's still the pre-test value, the host swallowed the opcode silently and you're inside DOSBox.

The post includes the full NASM exception handler. The bookkeeping is moderately fiddly because the handler has to walk the saved IP on the stack, peek at the offending opcode bytes (bx == 0x38fe after the little-endian load), and either advance IP / signal "not DOSBox" or restore the original IVT entry and let the real #UD handler retry.

The 86Box bug

Snow's first hardware-style test was on 86Box (a real-PC emulator separate from DOSBox), and the detector falsely reported "DOSBox." Stepping through with DEBUG.COM showed 86Box was happily executing FE 38 00 00 instead of trapping. Root cause: 86Box had inherited a bug from PCem where any FE ModR/M with a non-zero /n field was decoded as FE /1 (DEC). So FE /2, /4, /7 all silently decremented something instead of trapping.

Fix was straightforward and is merged upstream in 86Box PR #6561. Snow credits Linear (linear.network) for cross-checking on real hardware to rule out an Intel documentation issue.

Why this is the right route

The post is explicit that brittle approaches exist and are easier:

• BIOS version string at FE00:0061 starting with DOSBox (a comment in DOSBox-X already says they'd like to make this user-overridable)
• The Z: drive's existence or serial number
• DOSBox-X's own custom MOUNT and VER quirks

All of those can be spoofed by editing a config or rebuilding. The callback opcode is, in contrast, an inherent part of how DOSBox works — to remove it would be to break MOUNT.COM, IMGMOUNT.COM, and the rest of the virtual command set.

Adjacent emulators

The post closes by noting that detecting other DOS-likes is much easier:

• NTVDM and the Win9x MS-DOS Prompt: a single INT 2Fh call
• DOSEMU: a surprising number of host-callback APIs implemented as COM files (UNIX.COM runs arbitrary host commands), so detection is just probing whether those callables exist

None are as hard to spoof as a custom CPU instruction, but all are also more likely to cause side effects than reading a BIOS string.

The test program is on Snow's git forge at git.2ki.xyz/snow/dostests (NASM source).