windbg init script

There are a couple of commands that I always run every time I start up windbg and it just dawns on me that perhaps it's time to put these commands to the script and have windbg execute it automatically.

So in this post, let me show you how to do that.
First, create a file that will contain all the commands that you would like to run.
For this example, let me create a file called 'dbg-prep.cmd'

C:\Users\ilhoye\Desktop\WinDbg> type dbg-prep.cmd
.symfix
.reload
.load mex
.load kdexts
aS !pr !process

Once we have this, we can just launch windbg with '-c' option. '-c' is a command to execute when windbg starts up but for our case, we want to execute several command and that is why I created a script in the first place.
To do that, we still use '-c' option but now this time we want to provide file path as follows.

windbg.exe -c "$$>< C:\Users\ilhoye\Desktop\WinDbg\dbg-prep.cmd

Please note that '-c' option needs to be quoted like the above.
Of course, it is cumbersome to type all these so it would be best to create a shortcut for this and in fact, as for me here is my shortcut command which also specifies the connection for kernel debugging.

"C:\Program Files\Debugging Tools for Windows (x64)\windbg.exe" -k 1394:channel=2 -c "$$>< C:\Users\ilhoye\Desktop\WinDbg\dbg-prep.cmd"

You can also add arguments to the script and for more information, please refer to msdn.

WINDBG: Setting breakpoints for user-mode process from kernel mode debugger

When working with kernel debugger, sometimes we may want to set a breakpoint in the user-mode. Can we do it? Yes, we can. :)

So in this post let me show you how to do that using notepad as an example.
First, let us connect to kernel debugger and in my case I use 1394 debugger connection. Once we are connected, look for a process that we want to set a breakpoint for.

0: kd> !process 0 0 notepad.exe
PROCESS fffffa8012256980
    SessionId: 1  Cid: 0990    Peb: 7f72f0bf000  ParentCid: 07ac
    DirBase: 1babb8000  ObjectTable: fffff8a007c7a040  HandleCount:  68.
    Image: notepad.exe

Once we have located the process that we are interested in. Follow these steps.

0: kd> .process /i fffffa8012256980
You need to continue execution (press 'g' <enter>) for the context
to be switched. When the debugger breaks in again, you will be in
the new process context.
0: kd> g
Break instruction exception - code 80000003 (first chance)
nt!DbgBreakPointWithStatus:
fffff802`46e8f930 cc

.process command will set a process context to notepad and '/i' option means that the target process is to be debugged invasively. In other words, once we execute this command, it prompts us to type 'g'. When we type 'g', it will set the target process to be active process and in this context, we can set a user-mode breakpoint.

For instance, let us set a breakpoint at NtCreateFile of ntdll but before we do that, we need to reload the symbols. This will not only reload kernel symbols but it will also reload user-mode symbols which we need to set a breakpoint for.

3: kd> .reload
Connected to Windows 8 9200 x64 target at (Mon Oct  8 18:10:21.107 2012 (UTC - 7:00)), ptr64 TRUE
Loading Kernel Symbols
...............................................................
................................................................
...................
Loading User Symbols
.........................
Loading unloaded module list
......
3: kd> bp /p fffffa8012256980 ntdll!ntcreatefile

Now, let us resume and this time the debugger should be able to break into the user-mode process.

3: kd> g
Breakpoint 3 hit
ntdll!ZwCreateFile:
0033:000007fb`891a30f0 4c8bd1          mov     r10,rcx
1: kd> kcn
 # Call Site
00 ntdll!ZwCreateFile
01 ntdll!LdrpNtCreateFileUnredirected
02 ntdll!LdrpMapResourceFile
03 ntdll!LdrLoadAlternateResourceModuleEx
04 ntdll!LdrpLoadResourceFromAlternativeModule
05 ntdll!LdrpSearchResourceSection_U
06 ntdll!LdrFindResource_U
07 KERNELBASE!FindResourceExW
08 COMDLG32!FindResourceExFallback
09 COMDLG32!FindResourceExMirrorFallback
0a COMDLG32!CFileOpenSave::_GetDialogTemplate
0b COMDLG32!CFileOpenSave::Show
0c notepad!ShowOpenSaveDialog

You can see from the above that the breakpoint was hit and the callstack is actually from the user-mode.
This technique can be used in many different places and one of the place could be when we want to break into certain function of the service process when it was being loaded. I guess that there are probably other places but this will definitely save you from some work of coordinating two different debuggers.

Second Level Address Translation - EPT/NPT

This post is more or less to summarize what I have found out so that I can refer to it once I forget. (I know that I will forget this)

A few posts earlier I described how page table walk occurs and here let me briefly describe how that occurs with virtualization software such as Hyper-V. In this post. I will just describe the overall process without any debugger examples.

First of all, overall page walk with virtualization is similar to regular page table walk. Let me put out the regular page table walk diagram from the wiki page.

However, with virtualization things are a bit different. Keep in mind that whatever guest physical address that the OS thinks cannot be real physical address as hypervisor is the one that manipulates the real hardware. So what has to happen is another set of translations from the guest physical addresses to system physical addresses. Both Intel and AMD provides a solution to this address translation and they call these in two different names i.e. EPT and NPT but they are essentially the same thing.

So with guest physical addresses in our hand, we can traverse the similar data structures to obtain the system physical addresses. On Intel, these data structures are traversed via PML4 table - Page Directory Pointer table(PDPT) - Page Directory(PD) - Page Table(PT).

There are a couple of twists here to watch out though.

• If bit 7 of the EPT PDPT entry is '1', the EPT PDPTE maps 1-Gbyte page. Otherwise, it maps to 2-Mbyte page.
• For each entry of the table, we need to know the processor's physical-address width to obtain the physical address of the next table. 

We can get processor's address width by executing __cpuid with 0x80000008 in EAX and the the physical address width is returned in bits 7:0 of EAX. Well, that does not sound easy. Here is what I did. Just go to MSDN __cpuid page and copy the code and create a C++ source file and use that to obtain the value. On my machine, I got 36 so I know that my machine supports upto 36bit width.

So once we have the guest physical address and EPTP, it is just a matter of translating each address using the entry that we get to and the interpretation for each entry is subject to the tables given in chapter 28. VMX Support for Address Translation of Intel Manual.

In order to verify this page table walk, we need EPTP address and guest physical address but I have not found a way to obtain VMCS from the debugger easily. I will follow up on this if I find a way to obtain this pointer. But for now, everything is still in theory.

Enable Remote Desktop from PowerShell/WMI

The other day I had to connect to my dev box at work from home but I realized that I did not enable remote desktop on my machine at work so I could not connect to it. However, I knew that I have a remote desktop access to another machine in my office so I connected to it and tried to see if there is any way I can enable remote desktop on my dev box via WMI interface using PowerShell. I looked up online and found bits of information here and I enabled remote desktop but somehow I could not still remote desktop to my dev box. When I came to work next day I learned that I also had to enable firewall setting for remote desktop service. So here is my post on how to enable remote desktop on Windows 8.
First, we need to keep our credential information so that we do not have to enter it every time we connect to the remote machine.

$cred = Get-Credential

Then, let us check if remote desktop is enabled. This will tell us if we need to enable it.

PS C:\temp> Invoke-Command -computername dev-pc -scriptblock {(gwmi -class win32_terminalservicesetting -namespace "root\cimv2\terminalservices").allowtsconnections} -Credential $cred

If we need to enable it, run the following command.

PS C:\temp> Invoke-Command -computername dev-pc -scriptblock {(gwmi -class win32_terminalservicesetting -namespace "root\cimv2\terminalservices").setallowtsconnections(1)} -Credential $cred

Then, run the following one to make sure that authentication is required for remote desktop access.

PS C:\temp> invoke-command -computername dev-pc -scriptblock {(gwmi -class win32_tsgeneralsetting -namespace "root\cimv2\terminalservices").setuserauthenticationrequired(1)} -Credential $cred

The above step basically enables Remote Desktop on my dev-pc but the trouble I had was that I still could not connect to my dev box because of firewall so we need to take care of that as well. To do that, we will need to run one more command.

PS C:\temp> invoke-command -computername dev-pc -scriptblock {netsh advfirewall firewall set rule group="remote desktop" new enable=Yes } -Credential $cred

Once we execute the last command, we are now able to remote desktop to remote computer.
Lastly, let me put two websites that I got the above scripts from to enable remote desktop.

• Microsoft support page: How to use the netsh advfirewall firewall
• Checking and enabling Remote Desktop with PowerShell

Network Virtualization with SR-IOV in simple terms

It's exciting time for Hyper-V. There are so many new features with Windows 8 Hyper-V that it is worth mentioning. One of interesting new feature is SR-IOV which stands for "Single-Root I/O Virtualization". This feature is to offload all the work done by CPU to network card. Previously, Host OS had to process all the incoming/outgoing network packets which means that it required lots of CPU time just to process which VM the packet belongs to. However, with SR-IOV we can avoid that overhead!

Again, this is another cool feature that hardware brings into the virtualization space. From high-level point of view what this does is that the network card itself has virtual functions that act like ports. For regular network card, we just see one physical port as in the picture below.

However, SR-IOV network card has implemented virtual ports associated with physical port. Therefore, when the VM starts up, we can assign these virtual ports to the VM and from then on the VM can directly talk to network card. This way we do not have to consume all the CPU to figure out which VM the network packet belongs to. Instead we can cut down all the overhead and directly connect to the network card. Following is a diagram that I borrowed from MSDN.

Before I conclude today's post, let me introduce one more terminology that is equally important to get this feature working properly. That is IOMMU / Intel VT-d. When these network card interacts with the rest of the system, it either uses interrupt or DMA to either read from or write to memory location. With these virtual functions talking directly to VMs, it is also required that we process interrupts and memory access properly. What does this mean? For instance, when the network card wants to read the data from memory location that belongs to the VM, it accesses the memory location as if that is real physical address. However, that address might not be the right address for the VM as we have to share the memory address among all the VMs. Therefore, we need to translate the memory address and this is done by IOMMU. This is very similar to page table walk to translate virtual address to physical address. Yet, this is for virtual machines. Likewise, we have to map interrupts from these network device to the appropriate CPU by finding out which virtual port the interrupt is associated with and forwards it to the appropriate CPU.

I copied the following diagram from wiki page. Hopefully the idea will make sense with the picture below.

In summary, SR-IOV enables to offload all the processing work from CPU with the help from both network card and CPU feature.

• Virtual Ports/VIrtual Functions from SR-IOV network card
• DMA/Interrupt remapping from CPU

Virtual Machine Extensions - second post

Earlier I described briefly how the current virtualization technology works. Today let me try to explain little bit further on the same topic.

I showed the diagram that I got from Intel manual and basically that shows how VMM/Hypervisor interacts with Guest OS. In summary, it uses several instructions such as VM Exit or VM Entry to move in and out of Guest OS. To a degree that is very similar to how system call works. In other words, on 64-bit machine when we execute 'syscall' instruction, it causes the change to kernel mode and the kernel knows which system service the user mode application requested because the system service number is passed via EAX register. In the similar fashion, when we execute VM Exit, that causes the change to VMM/Hypervisor mode and VMM/Hypervisor does what's necessary to provide the guest OS virtualized environment.

If we look at disassembly of NtReadFile from ntdll.dll, we can see that it is calling 'syscall' as follows.

0:013> u ntdll!ntreadfile
ntdll!NtReadFile:
000007fe`747d2e40 4c8bd1          mov     r10,rcx
000007fe`747d2e43 b804000000      mov     eax,4
000007fe`747d2e48 0f05            syscall

Then the natural question would be what about virtualization? Do we also generate similar code for VM Exit or VM Entry? The answer is 'No'. The way it works is somewhat different and you can find the detailed information in chapters 25-27 of Intel Software Developer Manual but let me briefly describe how that works here.

Basically we tell processors that VMM/Hypervisor wants to gain the control when the guest OS executes certain instruction. Or we specify that when the guest OS touches certain parts of memory, we want to obtain the control so that we can provide appropriate information to the guest OS.
For instance, we might want to control the time inside guest OS and the way OS may obtain the time information is via TSC. So for this we want to gain the control when the guest OS executes 'rdtsc' or 'rdtscp' instruction. So essentially there is a certain data structure called virtual-machine control structures that the processor uses when it executes instructions and all we need to do is that we program this data structure. Once we do that the mode will be changed to VMM/Hypervisor when the given instruction is executed on the guest OS. Similarly, when we gain the control, processor provides us information with regard to the reason why VM Exit occured and the guest addresses at the time it was exiting so that we can use those information. So essentially this makes things a lot easier to create virtualized environment compared to binary patching technology.

Next, let me briefly describe another hardware support for virtualization.
That's the support for memory address translation. If you think about it, we cannot really give guest OS to control entire memory as that could mean that the given guest OS can view pages belonging to other guest OSs. Hence, we will have to control guest OS memory access in VMM/Hypervisor. How do we do that? Earlier we did this by using some sort of software page table that was hidden from the guest OS. What does this mean? This means that when the application in guest OS attempts to read from the memory, that address has to be translated to physical address as the address the application uses is virtual address. However, in our case the physical address that the guest wants to use is actually fake physical address from VMM/Hypervisor's point of view. Hence, we needed a way to provide real hardware physical address so that the guest OS can access to the correct memory address. This additional translation was done in software and its technology is normally called 'shadow paging'. Obviously, keeping up with all the hidden page tables and execute translation caused more memory consumption and slow execution compared to native execution case. So the hardware vendor such as Intel or AMD came up with the hardware support for this so that these are done hardware behind the scene so that the software does not have to concern this translation task. This technology is called extended page table (EPT) from Intel and nested page table (NPT) from AMD.

Here is the diagram that I stole from one of Intel presentation slides:  Intel Virtualization Technology Roadmap and VT-d Support in Xen. I think the diagram does a great illustrating the point. As you can see, there is now new EPT base register that points to EPT page table

In addition, hardware vendor added the Virtual Processor Identifiers (VPIDs) so that we do not throw out the TLB cache for those that belong to other processors as it makes sense that we want to keep these cache mapping data to improve the overall performance.

So that's all for now and here is the summary of this post:

• How to configure processor to gain control back when the guest OS executes certain instructions
• Hardware support for hidden page table to translate guest physical to machine physical address
• Keeping the address mapping data by using virtual processor id

I hope that this was useful to those who stop by my page. Thank you.

How to debug debugger?

Yes, title says it all.
If you've ever tried to create your own debugger extension to ease the task of debugging your own program/kernel module, you probably wanted to know how to debug your own debugger extension as that is running within debugger.

The answer is pretty simple. You can use a couple of windbg commands to accomplish your task.
First, traditional '.dbgdbg' command. This will spawn CDB session and create remote session so you can either just use cdb session to debug or use windbg and connect to the remote session.
But as you can see, cdb session is not so friendly so I often start new windbg session to debug my own debugger extension. This works but requires a couple of steps.

Then, today I learned new command. That is '!debugme -lw'. What this will do is that it simply creates windbg session attached to your own debugger so it cuts out the extra CDB and remote. Very simple and nicely done!

So next time you need to debug your own debugger extension, try '!debugme -lw'

In this post I assumed that you know how to develop windbg debugger extension. Please refer to Advanced Windows Debugging for more information on that topic. The book has a nice example where you can probably just replace some of the code to create your own debugger extension.