Search This Blog

Wednesday, February 10, 2016

WPP trace sample - USEPREFIX and USESUFFIX

Just recently I had to add some WPP traces and found that macro preprocessing makes it somewhat difficult to use it. I know that it has certain advantages but I did not find much information nor good samples for my own needs so I would like to share some of my findings.

Essentially, what I wanted to do is that I want to call some function to extract certain information from the arguments that I passed to WPP macro. The issue is that WPP preprocesses its own macros and produces another set of code before the build kicks in so and the other thing is that existing code already used debugoutput API so I wanted to minimize the code churn as much as possible. While working on this, I learned that I can use PREFIX and SUFFIX but I did not find any good samples.

Following code basically helps me to call foo() function with EXP parameter and SUFFIX will output whatever the information that I obtained from foo.
There is one thing to keep in mind.
When creating PRE and POST macros, you need to put parameter names as part of the names in the middle. Essentially, you can put a lot of code in PRE macro but let us remember that this is macro so your code size will grow pretty quickly and if you happen to use this in kernel driver, you will also need to be careful of APIs that you use in PRE to make sure that you follow IRQ level.


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
//MACRO: MYTRACE
//
//begin_wpp config
//USEPREFIX (MYTRACE, "%!STDPREFIX!");
//FUNC MYTRACE(LEVEL, EXP, ...);
//USESUFFIX (MYTRACE, "Result=%!HRESULT! %s",EXP, pszId);
//end_wpp

#define MYTRACE(level, exp, ...)                    \
    do {                                            \
        TraceEvents(level, VA_ARGS(__VA_ARGS__));   \
    } while (0)

#define WPP_LEVEL_EXP_PRE(level, exp)   \
    {                                   \
        PSTR pszId = foo(exp);          \
#define WPP_LEVEL_EXP_POST(level, exp) ;}

#define WPP_LEVEL_EXP_ENABLED(LEVEL, HR) WPP_FLAG_ENABLED(LEVEL)
#define WPP_LEVEL_EXP_LOGGER(LEVEL, HR) WPP_FLAG_LOGGER(LEVEL)

Once you have the above defined, you can then start to call your WPP macro as follows.


1
MYTRACE(TRACE_LEVEL_ERROR, hr, "hello, world");

This way, we can be more flexible using WPP traces. Hope this helps someone.

Saturday, December 27, 2014

Studying for PowerShell and Regex

At work, PowerShell with Regex is used for log search and hence, I am putting hours to learn these two together. I look at this as a good opportunity to learn these two as this is something that requires time to practice and having opportunities to use these two are great ways to learn them.

I tried to learn PowerShell a few times but never really reached a level where I felt comfortable but at this time, I am reading through Windows PowerShell Cookbook and finding that PowerShell has gone through many changes and many new commands.
For instance, I just ran into Invoke-WebRequest whose aliases are 'iwr', 'curl', or 'wget' I remember that we had to create new object using System.Net.WebRequest but with this new command, I don't see the needs. Good addition! In fact, there are many new commands and their alias makes it easy to transition to PowerShell world and as for me this script from the book really helped me to learn alias as I learn PowerShell.
But the most surprising addition to the latest PowerShell is readline library. With readline library, you can turn cmd window into something completely different. One of PowerShell author at Microsoft has implemented really nice module based on this PsReadline library and you should get it from here if you have done so. This will make PowerShell learning quite pleasant.

For regular expression we all know that it is more powerful and yet complicated than wildcard matching. I've looked through a book and some MSDN site but I have found Mastering Regular Expression much more complete than any other sources. So I picked up this book and practice it via PowerShell interface. In PowerShell we can run Regex using "-match" operator in PowerShell. Here is an example.

 PS C:\Temp> "12" -match "\b([1-9]|1[012])\b"  
 True  
 PS C:\Temp> "15" -match "\b([1-9]|1[012])\b"  
 False  

At first, I did not use "\b" which is word boundary. Without this it was producing incorrect result. I won't explain all the details of Regex here as I don't know enough but again point you to the Mastering Regular Expression. ;-)

If anyone is interested in learning these two, try these two books. ;-)

Thursday, December 4, 2014

Different ways of using windbg: Learn the code flow and verify the change.

Recently I switched to a different group and hence, I've been trying to ramp up quickly and started to work on bugs to learn the product.
While working on bugs, I learned that debugger is really helpful to understand the code flow and verify the change. Typically, people might use debugger to debug the issue and that's correct. However, there is another advantage of using a debugger. That is to learn how the code flows and verify the change you are making.
What do I mean by that?
First, let me illustrate with an example to show how debugger can be helpful to learn the code.
Say, you understand what the certain function does and noticed that this function is called several places but not sure exactly where the call is coming from and where it goes.
Then, set a breakpoint on that function and try to reproduce the issue. Sure enough, this will hit the function but at the same time, you might be surprised to find how this function is called from different places for other purposes.
As an example, we are learning NtCreateFile function and wanted to see how this is used. Then, set a breakpoint at NtCreateFile and see the callstack as follows:
 0:000> kcn8  
  # Call Site  
 00 ntdll!NtCreateFile  
 01 KERNELBASE!CreateFileInternal  
 02 KERNELBASE!CreateFileW  
 03 KERNEL32!GetDefaultSortFileMapping  
 04 KERNEL32!SetupDefaultSortTables  
 05 KERNEL32!InternalSortGetHandle  
 06 KERNEL32!SortGetHandle  
 07 KERNELBASE!GetSortVersionHandle  

As a trick, you can also give certain commands to breakpoints. For example, you can say, each time we hit NtCreateFile, display callstack and continue.
 bp ntdll!ntcreatefile "kcn; gc"  

Or if we are only interested in NtCreateFile() coming from certain function like CreateFileInternal as in the above example, we can do the following trick.
 0:000> bl  
  0 e 00007ff8`9b641bc0   0001 (0001) 0:**** ntdll!NtCreateFile "bd 0; x; gc"  
  1 e 00007ff8`98af72d0   0001 (0001) 0:**** KERNELBASE!CreateFileInternal "be 0;gc"  

Basically, what we are doing here is that we enable breakpoint #0 when we hit breakpoint #1 and as soon as we hit breakpoint #0, we display local variables and disable itself. This way we can suppress any other noise and quickly understand the code flow for your purpose.

Now that you understand the code and came up with the fix but sometimes it might be pretty difficult to reproduce the issue. Sometimes we can use debugger to change the code flow.
For instance, you have a function that returns boolean value and in order to hit the newly added code path, you need true from the function but you can't really reproduce the issue locally. In that case, you can again take advantage of breakpoint.
Here is how. Say you have a function and try to disassemble the function and see the address of 'ret' instruction. Once you find it, you know that the return value is passed via eax register so you can do the following.
Here is the disassembled code:
 notepad!IsTextUTF8+0x3c:  
 00007ff6`59596e0e 488b5c2408   mov   rbx,qword ptr [rsp+8]  
 00007ff6`59596e13 c3       ret  

Here is breakpoint:
 0:000> bl  
  2 e 00007ff6`59596e13   0001 (0001) 0:**** notepad!IsTextUTF8+0x41 "r eax=1;gc"  

Of course, this may not work all the time but sometimes this could be very handy to verify your change quickly.

In summary, a debugger can be very helpful in learning the code and verifying the fix. Spend some time learning how to use the debugger. It can save much of your time.

Thursday, October 16, 2014

PowerShell commands to configure Generation 2 VM kernel debug

Generation 2 VM has UEFI rather than BIOS. This means that secure boot is enabled by default but kernel debugging and secure boot are mutually exclusive.
Therefore, we will need to disable that first. Generation 2 VM still supports serial ports. We can set up serial ports for debugging again through PowerShell.

Here are two PowerShell commands to accomplish this:

 PS C:\> Set-VMFirmware -VMName testvm -EnableSecureBoot Off  
 PS C:\> Set-VMComPort testvm 1 \\.\pipe\vmdebug  

Switching to a different group - How I prepared for it.

I have been working in my current team for about three years so far and wanted to change the team to do different type of work. While I enjoyed debugging some hard problems and coming up with the fix, I have been wanting to consistently work on certain code base to learn the technology and deepen domain knowledge. Hence, I reached out those who I used to work with to gather some input/advice before I start to look around and today's blog is about what I have learned/gathered during this process.
First, do I really want a change?
One person whom I spoke to said it very clearly. He said that he could have made more impact on the product if he had stayed in my current team but he knew that he's not learning new technologies as much as he wanted in the current team. As such, he made a move and that has challenged him a lot for the past one year and felt that he grew so much. His new role/work is involved in big data/scale out work which helps him to be exposed to the latest hot topics and he's enjoying what he's doing. He mentioned that the process has been somewhat painful but that was intentional and he's glad that he made the move. As I was listening to him, I felt that I am in the similar situation that he was in a year ago and I want to have the similar changes in my own career. After that, I reached out to few more guys that I used to work with to gather some more information on how they found their new jobs. This helped me to what to look for and what's ahead when looking for jobs. With this, I started to look around opportunities inside the company mostly and started to send out my application with my resume based on HR job lists.

Second, what should I prepare for the interview?
You might say, "Interview?" Yes, we may still need to go through the interview process even if we change jobs inside the company. I learned that if I change jobs within the same group, I don't have to go through the interview but if I want to switch to different group, the interview process is required. As you guessed, preparing for the interview is not an easy task in computer science field. You will have to grab those programming algorithm books that I used in school and try to brush up your problem solving skills. Sometimes I wonder if this is really necessary in that we may never get to use some of these algorithms/data structures. Yet, that's the way it is so you will need to adapt to the system for the time being. :) Thanks to my friends, I came to know two very useful resources to prepare the interview and they are leetcode.com and EPI(Element of Programming Interview).

These two helped me immensely to be ready for technical interviews so let me describe these two a bit more in detail. leetcode.com is very helpful in that it has online judge site where you can solve the problems and submit your code to run the solution against test cases to see if your solution is correct and efficient. On top of that, you can also peek into other people's solution in discuss forum to see if there is any better solution. From this site I was able to check if my solution is correct and also learned how others solved the problem to enhance my solutions. It has about 150 problems to solve and you will surely learn much and confirm your answer via this web site to get ready for the interview.
Now, let me talk about EPI. In my opinion, EPI is one of the best books out there to prepare for the technical interview. It has a lot of questions and these questions are not easy but through these questions I felt that I am getting ready for the interview. It has questions that I never thought about so that helps you to think about such problems so that your brain gets exercised. In fact, some of questions that I studied from this book came up during the interview so that was very helpful.
I know that there are many more problems out there and I have heard from friends that some other companies tend to ask more difficult questions than what I have found from the above two resources. However, I think that these two have to be the base to start with so I highly recommend them!

Additionally, I would like to emphasize the amount of time you want to spend to prepare. Personally, it is very helpful if you can concentrate in studying the above two within short period of time so that you do not lose the context. It might be difficult to find time to study but I feel that you should treat this preparation as part of your job and take it seriously. Probably, it is a good idea to study for about three to four hours each day and seven to eight hours during weekends so that you can make significant process.


Third, which company/team to choose?
I did not have a lot of choices as I had four different interviews total. While going through the interview and getting to know each team/company, it became clear to me which team/company I would like to join. I believe that the interview process is not just one sided. It is not just the interviewee trying to impress upon the interviewer and pass the interview. It is also the interview's responsibility to sell their team/company so that the interviewee would like to come and work with them. I interacted with four different team/company, I felt comfortable with one particular team and decided to join that team. To be clear, I did not get offers from all these four places I had an interview with.

Lastly and perhaps most importantly, I would like to mention the importance of networking.
Making a career change is not an easy task but I have received a lot of advice and tips from friends/ex teammates that I worked with. I feel that keeping these relationships well is very important aspect of our career and as such, my advice to everyone is to reach out to friends/ex teammates for advice and you will be surprised how much you can learn from your peers to succeed in making career change. So if you are thinking of making changes in your career, send an email to your friend for lunch and ask them for their input.

All the best!

Saturday, January 18, 2014

Visual Studio makes it real easy to write drivers

I have heard sometime ago that visual studio now fully supports driver development. Years ago, I found some web resource to modify visual studio settings to make this happen but now that it officially supports driver development, I decided to give it a try.

In order to do this, you will need to install WDK from this msdn web site. Once you install WDK, start up Visual Studio and you will find Driver project when you start new project.
Not just that! In Visual Studio, you can even deploy your own driver to the remote computer!

MSDN also provides some hello world driver to help you to jump start.
Before deploying make sure to enable testsigning on remote computer and restart the remote computer.

However. if you want to do it manually, you will have to first install test certificate and after that right click inf file and click 'install'. That will install and start up your driver.

In case you have trouble installing driver, please check setupapi.dev.log file under C:\Windows\inf folder. If you have to debug your driver, then windbg is your friend.

Thursday, December 26, 2013

Enabling PowerShell remoting

I am experimenting PowerShell remoting lately and wanted to share simple tip to enable remoting.
To enable remoting, you will want to run 'Enable-PsRemoting' cmdlet and once you run it, it asks you several questions to which you will probably want to answer 'yes'. If you want to skip this, you can run 'Enable-PsRemoting -force' and that skips question steps.

Once you have the machine ready, by default it only allows admin to execute the script so you might want to add yourself so you can execute commands from remote computer.
To add yourself, run 'Set-PSSessionConfiguration Microsoft.PowerShell -ShowSecurityDescriptorUI' cmdlet and UI will pop up and you can use that to add yourself.

Now that you have enabled remoting, let's give it a try:

PS C:\Users\ilhoye> invoke-command -computer localhost -scriptblock {ps | select -first 5 | ft}

Handles  NPM(K)    PM(K)      WS(K) VM(M)   CPU(s)     Id ProcessName
-------  ------    -----      ----- -----   ------     -- -----------
    124      11     2368       6068    46            2192 Acmengine
    890      42    16560      44128   150            3416 CcmExec
     46       7     1820       5140    57     0.19    532 conhost
     46       7     1844       5184    57            1664 conhost
     46       7     1824       5168    57     0.23   2844 conhost

Finally, if you still have a trouble setting up remoting, a good place to start debugging is to run 'get-help about_Remote_Troubleshooting'. This has detailed information on most issues you might encounter.


Friday, November 15, 2013

PCI Config Space with windbg

I've been working on some PCI issue and as a result I learned a bit about PCI configuration stuff. Let me summarize few things about PCI with respect to configuration.

A processor is not capable of directly accessing these config space to read from or write to.
Instead, Root complex knows how to do this when a processor makes either IO or memory access.
For example, when processor attempts to read from config space, it will try to read from certain memory-mapped IO address. This request is latched to PCI root complex which then decodes the address and figures out whether it needs to re-route the packet to appropriate secondary bus.
While the packet is in transit, its type is set to TYPE 1 but once it reaches to PCI bridge where its one of connected bus is destination bus, it changes its type to be TYPE 0.

As mentioned, we can access config space either by IO port or memory-mapped IO. To make an access via IO port, we use 0xCF8 (address port) and 0xCFC (data port).
To view if these are allocated, we can use windbg to see.
 0: kd> !arbiter 1  
 DEVNODE fffffa8009782cb0 (HTREE\ROOT\0)  
  Port Arbiter "RootPort" at fffff802e555c5a0  
   Allocated ranges:  
    0000000000000000 - 00000000000003af    
     0000000000000000 - 00000000000003af SC  fffffa80097d1d30 (pci)  
     0000000000000000 - 00000000000003af SC  fffffa80097d1d30 (pci)  
     0000000000000000 - 000000000000000f CB  fffffa8009789a50   
     0000000000000020 - 0000000000000021 CB  fffffa8009789a50   
     0000000000000040 - 0000000000000043 CB  fffffa8009789a50   
     0000000000000048 - 000000000000004b CB  fffffa8009789a50   
     0000000000000070 - 0000000000000071 CB  fffffa8009789a50   
     0000000000000080 - 000000000000008f CB  fffffa8009789a50   
     0000000000000092 - 0000000000000092 CB  fffffa8009789a50   
     00000000000000a0 - 00000000000000a1 CB  fffffa8009789a50   
     00000000000000c0 - 00000000000000cf CB  fffffa8009789a50   
     00000000000000f0 - 00000000000000ff CB  fffffa8009789a50   
    00000000000003b0 - 00000000000003df S   fffffa80097d1d30 (pci)  
    00000000000003e0 - 0000000000000cf7 S   fffffa80097d1d30 (pci)  
    0000000000000cf8 - 0000000000000cff  B  fffffa8009789a50   
    0000000000000d00 - 0000000000000fff S   fffffa80097d1d30 (pci)  
    0000000000001000 - 000000000000efff S   fffffa80097d1d30 (pci)  
   Possible allocation:  
    < none >  

We see that we have allocated resource between cf8 and cff under RootPort. Let us run a couple more debugger commands to connect again between root port and resources.
 0: kd> !devobj fffffa8009789a50  
 Device object (fffffa8009789a50) is for:  
  00000010 \Driver\PnpManager DriverObject fffffa80097560f0  
 Current Irp 00000000 RefCount 0 Type 00000004 Flags 00001040  
 Dacl fffff9a10010dd91 DevExt fffffa8009789ba0 DevObjExt fffffa8009789bb0 DevNode fffffa8009749010   
 ExtensionFlags (0x00000800) DOE_DEFAULT_SD_PRESENT  
 Characteristics (0x00000080) FILE_AUTOGENERATED_DEVICE_NAME  
 AttachedDevice (Upper) fffffa80097a34b0 \Driver\ACPI_HAL  
 Device queue is not busy.  
 0: kd> !devnode fffffa8009749010 6  
 DevNode 0xfffffa8009749010 for PDO 0xfffffa8009789a50  
  Parent 0xfffffa8009782cb0  Sibling 0xfffffa800979bd30  Child 0xfffffa800970b010    
  InstancePath is "ROOT\ACPI_HAL\0000"  
  State = DeviceNodeStarted (0x308)  
  Previous State = DeviceNodeEnumerateCompletion (0x30d)  
  StateHistory[05] = DeviceNodeEnumerateCompletion (0x30d)  
  StateHistory[04] = DeviceNodeEnumeratePending (0x30c)  
  StateHistory[03] = DeviceNodeStarted (0x308)  
 [snip]  
  StateHistory[08] = Unknown State (0x0)  
  StateHistory[07] = Unknown State (0x0)  
  StateHistory[06] = Unknown State (0x0)  
  Flags (0x0c0001f5) DNF_MADEUP, DNF_HAL_NODE,   
            DNF_ENUMERATED, DNF_IDS_QUERIED,   
            DNF_HAS_BOOT_CONFIG, DNF_BOOT_CONFIG_RESERVED,   
            DNF_NO_RESOURCE_REQUIRED, DNF_NO_LOWER_DEVICE_FILTERS,   
            DNF_NO_LOWER_CLASS_FILTERS  
  DisableableDepends = 1 (from children)  
  BootResourcesList at 0xfffff8a000069ae0 Version 0.0 Interface 0 Bus #0  
   Entry 0 - Interrupt (0x2) Driver Exclusive (0x2)  
    Flags (0000) - LEVEL_SENSITIVE   
    Level 0, Vector 0, Group 0, Affinity 0xff  
    Range starts at 0x92 for 0x1 bytes  
   Entry 56 - Port (0x1) Driver Exclusive (0x2)  
    Flags (0x11) - PORT_MEMORY PORT_IO 16_BIT_DECODE   
    Range starts at 0xa0 for 0x2 bytes  
 [snip]  
    Range starts at 0xf0 for 0x10 bytes  
   Entry 59 - Port (0x1) Driver Exclusive (0x2)  
    Flags (0x11) - PORT_MEMORY PORT_IO 16_BIT_DECODE   
    Range starts at 0xcf8 for 0x8 bytes  
   Entry 60 - Memory (0x3) Driver Exclusive (0x2)  
    Flags (0000) - READ_WRITE   
    Range starts at 0x00000000fec00000 for 0x400 bytes  
 [snip]  

So we can see how 0xCF8 and 0xCFC ports are allocated under Root Port.
If you want to see if we ever use these ports to access config space, we can set a brekpoint but these are legacy way so most likely we won't hit the breakpoint this case.
At any rate, here is how you set the breakpoint:
 0: kd> ba i4 0xcfc  
 0: kd> bl  
  1 e 00000000`00000cfc i 4 0001 (0001)  

Wednesday, October 30, 2013

Some simple change to prevent future problem

This sounds so basic but I think we always want to have a second look on our code to see if we can improve the code. For instance, I had a piece of code that takes index as input and return the value from the array. The following is hypothetical example.

 int map_func(ULONG index)  
 {  
   assert(index < max);  
   return array[index];  
 }  

When I call this function, I make sure that index is within the range. But my code interfaces with other code and as the time goes on, the code becomes pretty complicated that I had a case where input 'index' was out of range and hence, the program crashed.
So I fixed that and after a few months later, the similar issue occurred. Only then, I realized that I should have changed 'assert' to the check that would survive even in free build.
Here is the new change.
 int map_func(ULONG index)  
 {  
   if (index >= max) {  
     assert(FALSE);  
     log("error occurred: %d\n", index);  
     return -1;  
   }  
   return array[index];  
 }  

Now, I can avoid program crash in free build and also have the assert in checked build. I know this is such a simple case but I only came up with this resolution when I stopped and thought about the fix one more time.

I think I will need this stop and think moment for all my works.

Saturday, November 10, 2012

Toaster device - installing wdm driver

I have been trying to understand how toaster wdm device works and apparently I spent so much time trying to install these drivers on my VM. First of all, its readme file is helpful but I found it lacking in some of its explanations. At first, I used devcon.exe to install bus driver but I bumped into a couple of issues so I could not install bus driver properly.
I searched online and found this msdn page where on the bottom it explains how to install toaster bus driver.

So that was helpful and on to the next issue: function driver. I could start up toaster by using enum.exe but it could not find some files that function driver was not installed. The message was not really helpful in that it does not say which files missing and part of it is that I do not fully understand what's needed for this install to happen. I could have spent time understanding installing package requirement but my goal was to understand Power management using this toaster before anything else.
Then, again from online I learned that I can look at setupapi.dev.log for more clue. By the way, this file is located in C:\Windows\inf directory.
Here is the error message I found from log file:
!!!  flq:                               Error installing file (0x00000002)
!!!  flq:                               Error 2: The system cannot find the file specified.
!    flq:                                    SourceFile   - 'c:\work\toaster\device\amd64\tostrco2.dll'
!    flq:                                    TargetFile   - 'C:\Users\ILHOYE~1.RED\AppData\Local\Temp\{6d5b7b46-959d-0823-c45e-094dc5a9816c}\amd64\tostrco2.dll'


By now, it is clear that I am missing tostrco2.dll. I don't know what file is for but I know that I need it. So I grabbed this from WDK and after that I could install toaster function driver. Now, I can happily debug toaster to understand the code flow.

How about traces? Is there any traces available with toaster driver? Yes, many of kernel drivers use either ETW or WPP tracing and hence by providing appropriate information you can turn on/off debug messages. These messages can be captured and saved to the file or you can actually see them if you have the debugger attached to your target machine/VM. In fact, toaster driver readme file describes the steps to do it but let me repeat that here. First of all, start the trace session by executing the following command where toaster.ctl contains "C56386BD-7C67-4264-B8D9-C4A53B93CBEB toaster"

c:\temp>tracelog -start toaster -rt -kd -ft 1 -guid toaster.ctl -flags 0xff

After that, in the kernel debugger you need to set the wmi path to refer to the TMF file location. What's TMF file? It is the file that contains information to translate the debug message to human readable strings. You can generate tmf file from pdb file by running 'tracepdb -f toaster.pdb' Here is how to set the path and enable debugging message.

kd>!wmitrace.searchpath + path_of_TMF_files
kd> !wmitrace.strdump
(WmiTracing)StrDump Generic
  LoggerContext Array @ 0x80BF1760 [64 Elements]
    Logger Id  2 @ 0x820C5000 Named 'MSDTC_TRACE_SESSION'
    Logger Id  3 @ 0x81AAF000 Named 'toaster'

kd> !wmitrace.enable 3
With that, you should be able to see the trace messages. Of course, you can always set the breakpoint where you are interested in to look into more details but knowing how to leverage existing traces should be helpful.

For toaster bus driver, if you want to see the debug messages, you will need to use chk build and use dbgview to enable kernel verbose debugging. However, once you turn on kernel verbose debugging, it will generate all sorts of debugging messages that you may not care about. Toaster bus driver uses DbgPrint for debug messages and that is essentially same as the following.

DbgPrintEx ( DPFLTR_DEFAULT_ID, DPFLTR_INFO_LEVEL, Format, arguments )

Therefore, we need to enable mask and level according to our mask and level. We can do this either updating registry or updating values via kernel debugger. For more information, please refer to MSDN page 'Reading and Filtering Debugging Messages' that describes how to enable certain component.


Sunday, October 14, 2012

Knowing what to practice

This week I read a book called "Talent is overrated". In the book author talks about how important it is to be deliberate in our practice to enhance our skills but what's even more important is to "KNOW" what to practice. Author said that many people do not have a clear picture of KNOWING what to practice. But those who accomplished knew exactly what they need to practice as they knew where they are lacking. I remember that my high school teacher said that the more you study, the more things that you see the needs to study. But if you do not study, you do not even know what to study. I think that it's a nice book to read and it does challenge the reader to be aware of one's status in terms of their achievement in their fields.


Just looking into where I am at right now, I feel that I do not really have a clear goal and I do not even know what to practice. Of course, there are many things out there I can/should study but my problem is that I do not see the needs so even though I may start out, I became lame and lose the passion to continually practice. I've got several books in front of me. C#, C++, Java Script and some OS/kernel books. I've looked at them to a degree but I've never mastered any of them. When I look at one subject, I feel like I should try another subject. That's pretty bad.

I hope and pray that I will be more consistent in my practice/study and be deliberate as the book suggests. I will keep posting my progress in my blog here so that I can keep track of my status. At this point, let me pick up one language and OS/kernel for the next one month. Perhaps I can write some simple apps like some command line tools, weather app, or stopwatch app. Hopefully, I will have a positive result in the next month.


Friday, October 12, 2012

String permutation with backtracking

This is probably one of classic interview questions. There are many solutions for this problem out there but today, a friend of mine mentioned this problem and I know that I solved this long time ago but my memory was fading so I decided to try this out.

As I thought about the problem, I decided to tackle this with backtracking. Basically the idea is that I take one character out from the initial word and mark that character to note that the character has been taken out for permutation. All along, I am passing just one character array and each time I reach the end, I simply print out the resulted string.

So let me show my code.
void perm_internal(string word, char *output, int n, int k)
{
    if (n == k) {
        cout << "[" << output << "]" << endl;
        return;
    }

    for (int i = 0; i < n; i++) {
        if (word[i] == 0) {
            continue;
        }
        char tmp = word[i];
        output[k] = tmp;
        word[i] = 0;
        perm_internal(word, output, n, k+1);
        word[i] = tmp;
    }
}

void perm(string word)
{
    int n = word.length();
    char *output;

    if (n == 0)
        return;
    output = (char *)malloc(n+1);
    memset(output, 0x0, n+1); 
    cout << "input word: " << word << "(" << n << ")" << endl;
    perm_internal(word, output, n, 0);
}


I want to try out different approaches to solve this problem but it's getting late so I will do that next time.

Wednesday, October 10, 2012

windbg init script

There are a couple of commands that I always run every time I start up windbg and it just dawns on me that perhaps it's time to put these commands to the script and have windbg execute it automatically.

So in this post, let me show you how to do that.
First, create a file that will contain all the commands that you would like to run.
For this example, let me create a file called 'dbg-prep.cmd'

C:\Users\ilhoye\Desktop\WinDbg> type dbg-prep.cmd
.symfix
.reload
.load mex
.load kdexts
aS !pr !process

Once we have this, we can just launch windbg with '-c' option. '-c' is a command to execute when windbg starts up but for our case, we want to execute several command and that is why I created a script in the first place.
To do that, we still use '-c' option but now this time we want to provide file path as follows.

windbg.exe -c "$$>< C:\Users\ilhoye\Desktop\WinDbg\dbg-prep.cmd

Please note that '-c' option needs to be quoted like the above.
Of course, it is cumbersome to type all these so it would be best to create a shortcut for this and in fact, as for me here is my shortcut command which also specifies the connection for kernel debugging.

"C:\Program Files\Debugging Tools for Windows (x64)\windbg.exe" -k 1394:channel=2 -c "$$>< C:\Users\ilhoye\Desktop\WinDbg\dbg-prep.cmd"

You can also add arguments to the script and for more information, please refer to msdn.

Monday, October 8, 2012

WINDBG: Setting breakpoints for user-mode process from kernel mode debugger

When working with kernel debugger, sometimes we may want to set a breakpoint in the user-mode. Can we do it? Yes, we can. :)

So in this post let me show you how to do that using notepad as an example.
First, let us connect to kernel debugger and in my case I use 1394 debugger connection. Once we are connected, look for a process that we want to set a breakpoint for.

0: kd> !process 0 0 notepad.exe
PROCESS fffffa8012256980
    SessionId: 1  Cid: 0990    Peb: 7f72f0bf000  ParentCid: 07ac
    DirBase: 1babb8000  ObjectTable: fffff8a007c7a040  HandleCount:  68.
    Image: notepad.exe


Once we have located the process that we are interested in. Follow these steps.

0: kd> .process /i fffffa8012256980
You need to continue execution (press 'g' <enter>) for the context
to be switched. When the debugger breaks in again, you will be in
the new process context.
0: kd> g
Break instruction exception - code 80000003 (first chance)
nt!DbgBreakPointWithStatus:
fffff802`46e8f930 cc

.process command will set a process context to notepad and '/i' option means that the target process is to be debugged invasively. In other words, once we execute this command, it prompts us to type 'g'. When we type 'g', it will set the target process to be active process and in this context, we can set a user-mode breakpoint.

For instance, let us set a breakpoint at NtCreateFile of ntdll but before we do that, we need to reload the symbols. This will not only reload kernel symbols but it will also reload user-mode symbols which we need to set a breakpoint for.

3: kd> .reload
Connected to Windows 8 9200 x64 target at (Mon Oct  8 18:10:21.107 2012 (UTC - 7:00)), ptr64 TRUE
Loading Kernel Symbols
...............................................................
................................................................
...................
Loading User Symbols
.........................
Loading unloaded module list
......
3: kd> bp /p fffffa8012256980 ntdll!ntcreatefile


Now, let us resume and this time the debugger should be able to break into the user-mode process.

3: kd> g
Breakpoint 3 hit
ntdll!ZwCreateFile:
0033:000007fb`891a30f0 4c8bd1          mov     r10,rcx
1: kd> kcn
 # Call Site
00 ntdll!ZwCreateFile
01 ntdll!LdrpNtCreateFileUnredirected
02 ntdll!LdrpMapResourceFile
03 ntdll!LdrLoadAlternateResourceModuleEx
04 ntdll!LdrpLoadResourceFromAlternativeModule
05 ntdll!LdrpSearchResourceSection_U
06 ntdll!LdrFindResource_U
07 KERNELBASE!FindResourceExW
08 COMDLG32!FindResourceExFallback
09 COMDLG32!FindResourceExMirrorFallback
0a COMDLG32!CFileOpenSave::_GetDialogTemplate
0b COMDLG32!CFileOpenSave::Show
0c notepad!ShowOpenSaveDialog


You can see from the above that the breakpoint was hit and the callstack is actually from the user-mode.
This technique can be used in many different places and one of the place could be when we want to break into certain function of the service process when it was being loaded. I guess that there are probably other places but this will definitely save you from some work of coordinating two different debuggers.

Thursday, October 4, 2012

Second Level Address Translation - EPT/NPT

This post is more or less to summarize what I have found out so that I can refer to it once I forget. (I know that I will forget this)

A few posts earlier I described how page table walk occurs and here let me briefly describe how that occurs with virtualization software such as Hyper-V. In this post. I will just describe the overall process without any debugger examples.

First of all, overall page walk with virtualization is similar to regular page table walk. Let me put out the regular page table walk diagram from the wiki page.



However, with virtualization things are a bit different. Keep in mind that whatever guest physical address that the OS thinks cannot be real physical address as hypervisor is the one that manipulates the real hardware. So what has to happen is another set of translations from the guest physical addresses to system physical addresses. Both Intel and AMD provides a solution to this address translation and they call these in two different names i.e. EPT and NPT but they are essentially the same thing.

So with guest physical addresses in our hand, we can traverse the similar data structures to obtain the system physical addresses. On Intel, these data structures are traversed via PML4 table - Page Directory Pointer table(PDPT) - Page Directory(PD) - Page Table(PT).

There are a couple of twists here to watch out though.

  • If bit 7 of the EPT PDPT entry is '1', the EPT PDPTE maps 1-Gbyte page. Otherwise, it maps to 2-Mbyte page.
  • For each entry of the table, we need to know the processor's physical-address width to obtain the physical address of the next table. 

We can get processor's address width by executing __cpuid with 0x80000008 in EAX and the the physical address width is returned in bits 7:0 of EAX. Well, that does not sound easy. Here is what I did. Just go to MSDN __cpuid page and copy the code and create a C++ source file and use that to obtain the value. On my machine, I got 36 so I know that my machine supports upto 36bit width.

So once we have the guest physical address and EPTP, it is just a matter of translating each address using the entry that we get to and the interpretation for each entry is subject to the tables given in chapter 28. VMX Support for Address Translation of Intel Manual.

In order to verify this page table walk, we need EPTP address and guest physical address but I have not found a way to obtain VMCS from the debugger easily. I will follow up on this if I find a way to obtain this pointer. But for now, everything is still in theory.

Thursday, August 23, 2012

Enable Remote Desktop from PowerShell/WMI

The other day I had to connect to my dev box at work from home but I realized that I did not enable remote desktop on my machine at work so I could not connect to it. However, I knew that I have a remote desktop access to another machine in my office so I connected to it and tried to see if there is any way I can enable remote desktop on my dev box via WMI interface using PowerShell. I looked up online and found bits of information here and I enabled remote desktop but somehow I could not still remote desktop to my dev box. When I came to work next day I learned that I also had to enable firewall setting for remote desktop service. So here is my post on how to enable remote desktop on Windows 8.
First, we need to keep our credential information so that we do not have to enter it every time we connect to the remote machine.
$cred = Get-Credential
Then, let us check if remote desktop is enabled. This will tell us if we need to enable it.
PS C:\temp> Invoke-Command -computername dev-pc -scriptblock {(gwmi -class win32_terminalservicesetting -namespace "root\cimv2\terminalservices").allowtsconnections} -Credential $cred
If we need to enable it, run the following command.
PS C:\temp> Invoke-Command -computername dev-pc -scriptblock {(gwmi -class win32_terminalservicesetting -namespace "root\cimv2\terminalservices").setallowtsconnections(1)} -Credential $cred
Then, run the following one to make sure that authentication is required for remote desktop access.
PS C:\temp> invoke-command -computername dev-pc -scriptblock {(gwmi -class win32_tsgeneralsetting -namespace "root\cimv2\terminalservices").setuserauthenticationrequired(1)} -Credential $cred
The above step basically enables Remote Desktop on my dev-pc but the trouble I had was that I still could not connect to my dev box because of firewall so we need to take care of that as well. To do that, we will need to run one more command.
PS C:\temp> invoke-command -computername dev-pc -scriptblock {netsh advfirewall firewall set rule group="remote desktop" new enable=Yes } -Credential $cred
Once we execute the last command, we are now able to remote desktop to remote computer.
Lastly, let me put two websites that I got the above scripts from to enable remote desktop.

Friday, June 15, 2012

Network Virtualization with SR-IOV in simple terms

It's exciting time for Hyper-V. There are so many new features with Windows 8 Hyper-V that it is worth mentioning. One of interesting new feature is SR-IOV which stands for "Single-Root I/O Virtualization". This feature is to offload all the work done by CPU to network card. Previously, Host OS had to process all the incoming/outgoing network packets which means that it required lots of CPU time just to process which VM the packet belongs to. However, with SR-IOV we can avoid that overhead!

Again, this is another cool feature that hardware brings into the virtualization space. From high-level point of view what this does is that the network card itself has virtual functions that act like ports. For regular network card, we just see one physical port as in the picture below.
However, SR-IOV network card has implemented virtual ports associated with physical port. Therefore, when the VM starts up, we can assign these virtual ports to the VM and from then on the VM can directly talk to network card. This way we do not have to consume all the CPU to figure out which VM the network packet belongs to. Instead we can cut down all the overhead and directly connect to the network card. Following is a diagram that I borrowed from MSDN.


Before I conclude today's post, let me introduce one more terminology that is equally important to get this feature working properly. That is IOMMU / Intel VT-d. When these network card interacts with the rest of the system, it either uses interrupt or DMA to either read from or write to memory location. With these virtual functions talking directly to VMs, it is also required that we process interrupts and memory access properly. What does this mean? For instance, when the network card wants to read the data from memory location that belongs to the VM, it accesses the memory location as if that is real physical address. However, that address might not be the right address for the VM as we have to share the memory address among all the VMs. Therefore, we need to translate the memory address and this is done by IOMMU. This is very similar to page table walk to translate virtual address to physical address. Yet, this is for virtual machines. Likewise, we have to map interrupts from these network device to the appropriate CPU by finding out which virtual port the interrupt is associated with and forwards it to the appropriate CPU.

I copied the following diagram from wiki page. Hopefully the idea will make sense with the picture below.


In summary, SR-IOV enables to offload all the processing work from CPU with the help from both network card and CPU feature.
  • Virtual Ports/VIrtual Functions from SR-IOV network card
  • DMA/Interrupt remapping from CPU

Monday, May 28, 2012

Virtual Machine Extensions - second post

Earlier I described briefly how the current virtualization technology works. Today let me try to explain little bit further on the same topic.

I showed the diagram that I got from Intel manual and basically that shows how VMM/Hypervisor interacts with Guest OS. In summary, it uses several instructions such as VM Exit or VM Entry to move in and out of Guest OS. To a degree that is very similar to how system call works. In other words, on 64-bit machine when we execute 'syscall' instruction, it causes the change to kernel mode and the kernel knows which system service the user mode application requested because the system service number is passed via EAX register. In the similar fashion, when we execute VM Exit, that causes the change to VMM/Hypervisor mode and VMM/Hypervisor does what's necessary to provide the guest OS virtualized environment.

If we look at disassembly of NtReadFile from ntdll.dll, we can see that it is calling 'syscall' as follows.

0:013> u ntdll!ntreadfile
ntdll!NtReadFile:
000007fe`747d2e40 4c8bd1          mov     r10,rcx
000007fe`747d2e43 b804000000      mov     eax,4
000007fe`747d2e48 0f05            syscall

Then the natural question would be what about virtualization? Do we also generate similar code for VM Exit or VM Entry? The answer is 'No'. The way it works is somewhat different and you can find the detailed information in chapters 25-27 of Intel Software Developer Manual but let me briefly describe how that works here.

Basically we tell processors that VMM/Hypervisor wants to gain the control when the guest OS executes certain instruction. Or we specify that when the guest OS touches certain parts of memory, we want to obtain the control so that we can provide appropriate information to the guest OS.
For instance, we might want to control the time inside guest OS and the way OS may obtain the time information is via TSC. So for this we want to gain the control when the guest OS executes 'rdtsc' or 'rdtscp' instruction. So essentially there is a certain data structure called virtual-machine control structures that the processor uses when it executes instructions and all we need to do is that we program this data structure. Once we do that the mode will be changed to VMM/Hypervisor when the given instruction is executed on the guest OS. Similarly, when we gain the control, processor provides us information with regard to the reason why VM Exit occured and the guest addresses at the time it was exiting so that we can use those information. So essentially this makes things a lot easier to create virtualized environment compared to binary patching technology.

Next, let me briefly describe another hardware support for virtualization.
That's the support for memory address translation. If you think about it, we cannot really give guest OS to control entire memory as that could mean that the given guest OS can view pages belonging to other guest OSs. Hence, we will have to control guest OS memory access in VMM/Hypervisor. How do we do that? Earlier we did this by using some sort of software page table that was hidden from the guest OS. What does this mean? This means that when the application in guest OS attempts to read from the memory, that address has to be translated to physical address as the address the application uses is virtual address. However, in our case the physical address that the guest wants to use is actually fake physical address from VMM/Hypervisor's point of view. Hence, we needed a way to provide real hardware physical address so that the guest OS can access to the correct memory address. This additional translation was done in software and its technology is normally called 'shadow paging'. Obviously, keeping up with all the hidden page tables and execute translation caused more memory consumption and slow execution compared to native execution case. So the hardware vendor such as Intel or AMD came up with the hardware support for this so that these are done hardware behind the scene so that the software does not have to concern this translation task. This technology is called extended page table (EPT) from Intel and nested page table (NPT) from AMD.

Here is the diagram that I stole from one of Intel presentation slides:  Intel Virtualization Technology Roadmap and VT-d Support in Xen. I think the diagram does a great illustrating the point. As you can see, there is now new EPT base register that points to EPT page table


In addition, hardware vendor added the Virtual Processor Identifiers (VPIDs) so that we do not throw out the TLB cache for those that belong to other processors as it makes sense that we want to keep these cache mapping data to improve the overall performance.

So that's all for now and here is the summary of this post:

  • How to configure processor to gain control back when the guest OS executes certain instructions
  • Hardware support for hidden page table to translate guest physical to machine physical address
  • Keeping the address mapping data by using virtual processor id

I hope that this was useful to those who stop by my page. Thank you.

Wednesday, May 16, 2012

How to debug debugger?

Yes, title says it all.
If you've ever tried to create your own debugger extension to ease the task of debugging your own program/kernel module, you probably wanted to know how to debug your own debugger extension as that is running within debugger.

The answer is pretty simple. You can use a couple of windbg commands to accomplish your task.
First, traditional '.dbgdbg' command. This will spawn CDB session and create remote session so you can either just use cdb session to debug or use windbg and connect to the remote session.
But as you can see, cdb session is not so friendly so I often start new windbg session to debug my own debugger extension. This works but requires a couple of steps.

Then, today I learned new command. That is '!debugme -lw'. What this will do is that it simply creates windbg session attached to your own debugger so it cuts out the extra CDB and remote. Very simple and nicely done!

So next time you need to debug your own debugger extension, try '!debugme -lw'

In this post I assumed that you know how to develop windbg debugger extension. Please refer to Advanced Windows Debugging for more information on that topic. The book has a nice example where you can probably just replace some of the code to create your own debugger extension.

Wednesday, April 18, 2012

Short introduction to Virtual Machine Extensions

Things have been changed greatly in the virtualization area. A few years ago, we had to emulate hardware by either binary patch at runtime or modify the guest OS. So not all the instruction was run natively and the goal was to run as much instruction as possible natively on the actual processors without the VMM intervention.

Nowadays, there is a better way to do virtualization. Namely, hardware support! Hardware vendors came up with many new features in CPU so that we do not have to do all the work in the software any more. At the heart of this technology is VMX in Intel and SVM in AMD. Here is high level overview of these technology. The goal is that from time to time VMM needs a way to gain control to feed different information to guest.
First, system software will have to register with processors that it is interested in gaining control when the guest OS executes such and such instruction. We do this by configuring virtual-machine control structure (VMCS).

Once this is set up and the guest OS actually executes instructions we registered, it will cause the transition from VMX guest operation to VMX root operation so that it can give processed information to the guest. This is called VM exits.

Following is a diagram from Intel Manual with regard to the interactions between VMM and VM guests.


There are some other ways to enter VMM mode but that's essentially how the virtualization with hardware support works in the core. Here I have skipped how the system software discovers VMX support in the processor and how it enables the features. For more information, please refer to Intel/AMD documentation. Hopefully, I will have more to say about those in the next posts.