Part 2 – Detection
You are reading Part 2: Detection.
🔗 Check out the source code here: GhostDebug on GitHub
Reasons for Debugger Detection
Because debuggers allow deep insight into the execution of a program, in many areas there is a desire to prevent debugging. This desire is not limited to malware that wants to make analysis by security researchers harder. Commercial software developers also use anti-debugging techniques, for example to strengthen Digital Rights Management (DRM) or to prevent cheat development in games. As a result, many methods have been created to detect the presence of a debugger and, if possible, to block or manipulate the program behavior.
The motivation to detect a debugger goes beyond malware analysis and includes several areas:
-
Malware and malicious programs Malware authors usually want to prevent security researchers from analyzing their software in detail. Since a debugger allows tracking execution, making communication visible, or extracting encryption routines, malware has an interest in making analysis as hard as possible. For this reason, malicious programs integrate anti-debugging techniques to make analysis more difficult or impossible [Kim et al. 2019].
-
Crack protection Software vendors of commercial applications often use protection mechanisms to make illegal copying or manipulation harder. A debugger could be used to bypass such mechanisms, for example to disable license checks. By checking for a debugger, programs can detect and stop such attempts early, making attacks on software integrity more difficult [VMProtect n.d.].
-
Games and anti-cheat measures Computer games are another field where debugger detection is very important. Cheat developers use debuggers to analyze important data structures and logic, for example by reading memory where player values are stored or by finding functions responsible for game physics. Debugging makes it easier to find and bypass weaknesses in anti-cheat systems. To prevent cheats, game developers use anti-debugging or anti-cheat systems that include debugger detection. If a debugger is detected, the game may end execution or block the player [Game Hacking Academy n.d.].
-
General anti-reversing measures Many developers want to protect their software to defend intellectual property or to prevent that implementation details become public. Debuggers are central tools for reverse engineering and make it easier to extract algorithms for business logic and protection mechanisms. Anti-debugging can therefore act as an initial barrier to increase the effort for reverse engineers.
Regardless of the purpose, the principle of debugger detection is always to look for characteristics caused by a debugger. These can be set flags in the process environment, unusually long execution times between instructions, or calls to operating system functions. All serve the same goal: detect analysis software and take countermeasures.
Common Anti-Debugging Techniques
In Windows there are many strategies to detect a debugger and sabotage program execution once a debugger is found. Most of these methods target classic debuggers such as WinDbg or x64dbg. This section focuses on explicit API-based checks, where the program queries flags or process structures via defined function calls. Behavioral methods such as timing attacks or checking process names are more context dependent and are not discussed here.
IsDebuggerPresent
The Windows API function IsDebuggerPresent is one of the most common ways to check if a process is executed under a debugger. Internally it checks the BeingDebugged flag in the Process Environment Block (PEB) of the running process. If this flag is set, the operating system signals that a debugger is attached. The program can then react, for example by aborting execution or switching mode [MS Docs: IsDebuggerPresent 2025].
1
2
3
4
if (IsDebuggerPresent())
std::cout << "Debugger detected!" << std::endl;
else
std::cout << "Debugger not detected!" << std::endl;
CheckRemoteDebuggerPresent
CheckRemoteDebuggerPresent makes it possible to check whether another process is debugged. Like IsDebuggerPresent, it queries the BeingDebugged flag in the PEB, but for a target process. A common use is to check if a debugger is attached to the current process. The difference is that this function requires a process handle as parameter [MS Docs: CheckRemoteDebuggerPresent 2024].
1
2
3
4
5
6
bool isDebuggerPresent = false;
CheckRemoteDebuggerPresent(GetCurrentProcess(), &isDebuggerPresent);
if (isDebuggerPresent)
std::cout << "Debugger detected!" << std::endl;
else
std::cout << "Debugger not detected!" << std::endl;
PEB BeingDebugged Flag
Behind these functions is the BeingDebugged flag in the PEB of a process. Programs can read this flag directly from the PEB without calling official APIs. This avoids obvious function calls such as IsDebuggerPresent and bypasses hooks or patches targeting API functions [MS Docs: PEB 2022], [Check Point 2023a].
1
2
3
4
5
6
7
8
9
10
#ifndef _WIN64
PPEB peb = (PPEB)__readfsdword(0x30);
#else
PPEB peb = (PPEB)__readgsqword(0x60);
#endif // _WIN64
if (peb->BeingDebugged)
std::cout << "Debugger detected!" << std::endl;
else
std::cout << "Debugger not detected!" << std::endl;
CloseHandle / SEH Trick
CloseHandle is a standard API function that closes a handle, e.g., to a file, thread, or process object. Normally Windows can recognize invalid handles without throwing an exception. It simply returns an error code. But with a debugger attached, a different mechanism occurs: a Structured Exception Handling (SEH) handler can catch an exception.
When CloseHandle is executed with an invalid handle inside an SEH block, under a debugger the invalid operation raises a first chance exception. Without a debugger, Windows only returns an error code. Thus a debugger can be detected if the SEH handler executes [Check Point 2023b].
1
2
3
4
5
6
7
8
9
10
11
__try
{
CloseHandle((HANDLE)0xDEADBEEF);
std::cout << "Debugger not detected!" << std::endl;
}
__except (EXCEPTION_INVALID_HANDLE == GetExceptionCode()
? EXCEPTION_EXECUTE_HANDLER
: EXCEPTION_CONTINUE_SEARCH)
{
std::cout << "Debugger detected!" << std::endl;
}
NtQueryInformationProcess
Another key function is NtQueryInformationProcess. It can query various process information by specifying a ProcessInformationClass value [MS Docs: NtQueryInformationProcess 2024]. Two values are especially interesting:
- ProcessDebugObjectHandle (0x1E): returns the debug object handle used for communication between debugger and process. Its presence is a strong indication of debugging.
- ProcessDebugPort (0x7): shows if a debug port is connected.
Both can be used to reliably detect a debugger [Check Point 2023a].
1
2
3
4
5
6
7
8
9
10
11
12
13
const DWORD ProcessDebugObjectHandle = 0x1e;
HANDLE hDebugObject;
NTSTATUS status = NtQueryInformationProcess(
GetCurrentProcess(),
ProcessDebugObjectHandle, &hDebugObject,
sizeof(hDebugObject), NULL
);
if (status >= 0 && hDebugObject)
std::cout << "Debugger detected!" << std::endl;
else
std::cout << "Debugger not detected!" << std::endl;
These queries are robust because they use system calls that access kernel structures directly. When a debugger attaches, entries for the debug port or debug object are created. Without them, the kernel cannot forward exceptions to the debugger. Manipulating these structures is very difficult, as they are deep in the OS mechanisms.
Through these techniques, programs can actively check whether they are under observation by a debugger tool. In many cases developers combine several checks to ensure reliable detection and to block debuggers as early as possible. This creates a constant cat-and-mouse game between debugger developers and anti-debugging vendors.
Countermeasures
The detection techniques described above can be bypassed by mechanisms that manipulate debug flags, process structures, or API calls. In practice, both modified debuggers and external tools are used. The goal is to make the program’s checks fail or return false information.
A popular example is the ScyllaHide plug-in used with x64dbg. Its core sets certain debug flags to FALSE so that Windows APIs like IsDebuggerPresent or CheckRemoteDebuggerPresent always report “no debugger”. Hooks are also placed in functions to modify return values. For example, NtQueryInformationProcess can be patched so that ProcessDebugObjectHandle or ProcessDebugPort always report “no debugger”, even when debugging is active [x64dbg 2025].
ScyllaHide also implements Instrumentation Callbacks. These can be used in user mode to temporarily take control after a system call and modify return data. Since some programs use direct syscalls instead of ntdll stubs, API hooks alone may fail. With instrumentation callbacks, return values can still be intercepted at the syscall level. But this can be bypassed if the target program disables or overwrites callbacks before the call [wlan 2018].
A completely different approach is a VEH-based debugger. It does not use the Win32 Debugging API. Instead, it only registers in the vectored exception handler list. From that point it can intercept all exceptions without Windows setting debug flags. No debug port entry, no BeingDebugged flag, and no debug objects are created. Since most anti-debugging checks look for exactly these indicators, a VEH-based debugger stays mostly invisible. Breakpoints (INT 3) and other exceptions can be handled through the vectored handler, giving full control without the target program knowing.