Developing an undetected debugger on Windows - Part 4 [Evaluation]

Part 4 – Evaluation

You are reading Part 4: Evaluation and Results.

• 👉 Part 1: Theory
• 👉 Part 2: Detection
• 👉 Part 3: Implementation

🔗 Explore the complete project here: GhostDebug on GitHub

Evaluation and Results

In this chapter the detectability of the newly developed debugger is examined. First the test environment is described. Then the detectability is measured and compared to classic debuggers like x64dbg, including results with plugins such as ScyllaHide.

In addition, potential attack points of the presented approach are discussed. Although the debugger avoids the classical Debugging APIs and therefore bypasses common detection, strategies exist that could still reveal its presence. These aspects are discussed for further hardening and future improvements.

Test Environment

The test target is a program developed for this purpose. At runtime it executes several debugger detection routines. Only after pressing Enter are the checks triggered, so a debugger must already be attached to test effectiveness. The test target uses the following methods:

• IsDebuggerPresent
• CheckRemoteDebuggerPresent
• PEB (Process Environment Block) BeingDebugged flag
• NtQueryInformationProcess (ProcessDebugObjectHandle, ProcessDebugPort)
• CloseHandle trick with invalid handles

After execution the program prints which checks detected a debugger. Three scenarios were tested. In each case the debugger was attached first, then Enter was pressed to trigger the checks:

1. Vanilla x64dbg (without plugins) – standard configuration.
2. x64dbg with ScyllaHide – plugin configured to bypass as many checks as possible.
3. The new VEH debugger – using the developed debugger to test invisibility.

This setup allows comparison of effectiveness of different debugging approaches.

Detectability of the Debugger

Test Results

Vanilla x64dbg In standard configuration all checks succeed in detecting the debugger. IsDebuggerPresent, CheckRemoteDebuggerPresent, the BeingDebugged flag, the CloseHandle trick, and NtQueryInformationProcess all report a debugger. This is expected because x64dbg uses the standard Debug APIs.

x64dbg with ScyllaHide With ScyllaHide most anti-debugging methods are bypassed. API calls like IsDebuggerPresent and CheckRemoteDebuggerPresent are hooked and return false. The BeingDebugged flag and CloseHandle trick are also bypassed. However, NtQueryInformationProcess still detects a debugger, because the test target disables the instrumentation callback used by ScyllaHide. This makes NtQueryInformationProcess a reliable detection method in this test [Check Point 2023a].

VEH-based debugger All checks report that no debugger is present. Neither IsDebuggerPresent, CheckRemoteDebuggerPresent, nor NtQueryInformationProcess detect anything. Because the debugger avoids the Win32 Debugging Interface entirely, no debug flags are set and no hooks are required. This shows the strength of the Vectored Exception Handling approach: it stays invisible to common detection.

Discussion and Possible Detection Methods

Even with these positive results, detection is still possible:

• Timing checks – using the CPU instruction RDTSC (Read Time-Stamp Counter). If an application measures delays between instructions, it could detect stepping or inserted breakpoints. This requires the program to explicitly place timing checks [Ferrie 2011].

• Scanning vectored handler list – an application could inspect LdrpVectorHandlerList in ntdll.dll where vectored exception handlers are registered. This could reveal the debugger’s VEH. However, many legitimate applications (e.g., overlays, streaming tools) also use VEH, so identification is uncertain. A countermeasure is moving the exception hook, e.g., by inserting a trampoline hook in KiUserDispatchException.

• Kernel-mode anti-debugging – advanced systems (e.g., EasyAntiCheat) move exception handling into kernel mode, overwriting pointers in the Hardware Abstraction Layer (HAL). This allows interception of exceptions and protection of registers such as CR3 against inspection [AVX Blog n.d.]. This goes beyond the scope of usermode debugging.

Summary: The VEH-based debugger is not detected by typical checks and avoids hooks unlike ScyllaHide. Advanced detection is possible but requires much more effort.

---

Conclusion and Outlook

In this work a debugger was developed that avoids the classical Win32 Debugging APIs and instead uses Vectored Exception Handling. The implementation shows that both breakpoints (INT 3) and single-stepping can be implemented effectively without triggering common detection.

Summary of Results

Evaluation showed that the VEH-based debugger is almost invisible in typical test environments. By avoiding Win32 interfaces no debug flags are set in the process. Compared to x64dbg in standard configuration, where all detection methods succeeded, the VEH debugger was not detected. Even with ScyllaHide, NtQueryInformationProcess remained a weak point, while the VEH debugger required no hooks and still stayed hidden.

This demonstrates the strength of the debugger: it operates on the exception layer, outside the usual detection mechanisms.

Possible Improvements

Future extensions include:

• Manual-mapping injection Instead of loading the DLL with LoadLibrary, manual mapping could be used to inject without visible module references in the process [Elastic 2017].

• Hooking KiUserDispatchException To evade scans of LdrpVectorHandlerList, a trampoline hook in KiUserDispatchException could hide the VEH handler registration.

• Extended stepping functions Currently only single-stepping is supported. Step-over and step-out functionality could be added to improve usability.

• Hardware and memory-access breakpoints In addition to software breakpoints, hardware breakpoints and memory watchpoints could be implemented to monitor specific addresses.

Final Remark

The VEH-based debugger shows that even without the official debug port and APIs, a powerful analysis tool can be built. It offers an alternative to classical debuggers that are easily detected. Despite its unusual basis in exception handling, the core operation is similar to an API-based debugger. The key difference is when and why the operating system forwards an exception to the debugger.