Search This Blog

Sunday, November 20, 2011

Powershell works differently

For years I tried to pick up powershell but everytime I tried, I miserably failed to feel comfortable about it. But I felt strongly that I need to learn some scripting language to automate some works on Windows so I decided to give it a try one more time.

This blog is by no means to say that I have mastered it but I think that I finally understood why I could not learn the powershell. I think that is because I approached powershell from my bash scripting experience. What do I mean by it? When we use bash shell, we work with string but when we work with powershell, that does not work well. Powershell always returns objects not string and hence when we want to process output of certain output, it is necessary that we approach it from this perspective. So let me illustrate this.

Often times we want to list something and then 'grep' certain words from the list output. For instance, let us try 'alias' command on powershell.


Alias command produces lots of output that its output is more than a page. So probably I want to see if alias output has the line that contains the word 'content' in some easy way. Readily I think of 'grep' command and I remember that select-string works somewhat similarly. So let us try Select-String.


Hmm. Nothing... But I know that there is a line that has the word 'content' What's going on here? Apparently, Select-String is only looking for words to match from Name column. So let us now try 'asnp' to verify it.


Ok, that is better but I want to get the entire line. What should I do?
This is where I realized that powershell works on objects not string. In principle, powershell operates on objects and hence when the powershell returns the output for alias command, it actually returned the list of objects. So if we want to get the whole line which in this case is an object, we want to try to match the entire object. Here is the trick.



Yes, we got what we wanted. Again, this is because we were looking for object where its definition contains 'content' and it returns the list of those object.

In summary, powershell operates on objects so when you use powershell, keep that in mind!

Thursday, October 20, 2011

Increase the productivity with raid0

Yesterday I was busy setting up my machine at work and one of my co-workers shared the tip to increase the machine throughput so let me share here with all of you.
Every time we build the projects or do some IO intensive works, things get slower because of the bottleneck with the disks. Hence if we can avoid writing to/reading from the disk, the productivity increases and there are numerous examples out there with this idea and probably one of well-known example is memcache.

So what can we do about disk IO? If we have to write to the disk in any way, there is a way to improve the situation with raid0. Normally whenever we think about the raid, we tend to think about redundancy but raid0 is not so much about that. What raid0 is doing is striping. According to wiki page, it is defined as follows.
A RAID 0 (also known as a stripe set or striped volume) splits data evenly across two or more disks (striped) with no parityinformation for redundancy.


In my machine, I happen to have two additional disks and hence my co-worker suggested me to use raid 0 to boost the IO and that is what I did. So how did I do it? It turned out that it is fairly easy to do it on Windows. Here is the steps.
First let us fire up 'Computer Management' by first going to 'Control Panel' and selecting 'Administrative Tools' and finally choosing 'Computer Management' icon. Once you have it running, click 'Disk Management' under 'Storage'. When you do that, you will see all your disks and its partitions and there choose two disks that you want to use for raid0 and right click mouse and you will see the menu as follows.


Choose 'New Striped Volume...' and when you do that, you will be presented a couple of windows to configure the raid0 such as allocating the size of the disk for raid0.
Once you do that, you are done. Yes, I know it is too simple. But that's all it takes. Now when you look at the 'Disk Management' console, you will see something like the following.


I created the raid0 and label it as E: drive and hence I put my source code there so that all the IO work happens there to isolate the IO from other OS or app activities. So if you are a windows user and have some extra disks on your machine, give it a try!

Monday, September 12, 2011

Signed or Unsigned?

I found it confusing at times to understand signedness of integer and hence I have attempted to tackle some of these issues here. First, the task I want to discuss here is the internal representation of a given number in a signed and unsigned integer type and I think that this will help me to have a better understanding of signedness.
To illustrate the point, let us assume that we need to develop a function that should return the absolute value of input number. For this task, can we just return the same value but specify that the return type of unsigned as follows?

Unfortunately, this will not work properly. For example, if we pass a negative value to the above function and assign the returned value to the unsigned type of integer, we will see that the function somehow did the trick. However, if we assign the returned value to the signed type of integer, it will still display the negative value. Why is that? That is because we have not actually changed anything. Let me explain with an example. Say, we pass '-1' to unsigned_abs() function, this passed value is represented as 32 1's in a binary format. This value is interpreted as '-1' in a signed integer but that same value is interpreted as 0xffffffff in a unsigned integer.

So then what should we do? Here is one way to implement abs() function.


This will actually update the internal value of input value and return the real absolute value. To verify this, I wrote a function to display the binary value of an input integer as follows.

The above function simply uses STL stack to store binary values of the input value and prints out on the screen and when I executed the returned values from unsigned_abs() and signed_abs(), I could see that the returned value of unsigned_abs() is displayed as 32 1's but the returned value of signed_abs() is displayed as '1' when I called binaryrep().

In conclusion, we need to be careful of what's really taking place when we use either unsigned and signed and in my opinion, if we are not sure, I would recommend to just use signed integer.

Tuesday, August 23, 2011

Take advantage of chromium project!

This post is going to be very short. In this post I just wanted to share my experience with chromium browser project. Not that I have ever contributed to chromium project but as I read/study the code for my own edification, I found it very useful.


I found that there are many parts of code that I can use for my various needs and as we know the code quality must be good.
For instance, I recently found that how to dump stack trace on Windows. Earlier last year I tried to do the similar thing on Linux but I never knew that the similar API exists on Windows. But thanks to these projects, I found the working example code of stacktrace code. The list can go on and on and what's neat about this project is that it has the code that is doing the same thing on Linux, Windows, and Mac so if you need to write code for all these platforms in C/C++, check it out! Lastly, I hope that I will be able to contribute to the project in the near future.

Saturday, August 6, 2011

Breadth First Search and Backtracking

This week I had a chance to look at Topcoder.com tutorial and read about BFS/Backtracking. Sometimes it is not clear if we need to use BFS or Recursion/backtracking approach to solve a given problem and the following is the BFS description from Topcoder.com

Breadth First Search (BFS):
Problems that use BFS usually ask to find the fewest number of steps (or the shortest path) needed to reach a certain end point (state) from the starting one. Besides this, certain ways of passing from one point to another are offered, all of them having the same cost of 1 (sometimes it may be equal to another number). Often there is given a N x M table (formed of N lines and M columns) where certain cells are passable and others are impassable, and the target of the problem is to find the shortest time/path needed to reach the end point from the start one. Such tables may represent mazes, maps, cities, and other similar things. These may be considered as classical BFS problems. Because BFS complexity is in most cases linear (sometimes quadratic, or N logN), constraints of N (or M) could be high - even up to 1 million.

Having said that let me post two problems that sound somewhat similar but require different approach to solve. First question. We have a robot sitting somewhere in the matrix and we want to find the shortest path to get to a certain position. This is a kind of problem that needs BFS approach as described above. Whenever we see the shortest and the cost to move from one state to another is same, it is safe to assume that we can use BFS. As we know from graph theory, we use BFS to find the shortest path and hence we can apply BFS to solve the problem and the key to remember is that BFS requires us to use queue-like data structure. At each node, we need to enqueue the neighboring nodes for the future processing and we do this this until the queue is empty. When the queue is empty, we know that we have covered all the nodes that could be reached from the source node. Lastly, for this particular problem we want to keep the number of paths taken so far and use that to calculate the next one by simply adding one.

Second question. How do we find the possible number of paths from one node to another for a robot to take? In this problem, we do not concern the shortest path but would like to know how many paths exist from the source node to the destination node. This is somewhat simple problem in that we just need to recursively call ourselves until we get to the destination. For this particular problem, we need backtracking because we want to keep track of states between each call and thus, we will need to use backtracking technique.
Here is the better description of backtracking from Topcoder.com:

Backtracking:
This technique may be used in many types of problems. Just take a look at the limits (N, M and other main parameters). They serve as the main hint of a backtrack problem. If these are very small and you haven't found a solution that's easier to implement - then just don't waste your time on searching it and implement a straight-forward backtracking solution. 

Usually problems of this kind ask you to find (similarly to Brute Force):

  1. Every possible configuration (subset) of items. These configurations should respect some given rules.
  2. The "best" configuration (subset) that respects some given rules.
What does that mean? It means that we do not want to explore the path that we have already visited. In other words, we keep the states so far and add the next route and try the next route that's available. Why? In the next move, we normally want to know where we have visited to make a decision. However, when we come back from the next move, we want to take out the next move we just added and add another route that we have not tried yet.

Again, it is possible that the question may ask to calculate the shortest path but the cost for each move differs. For Backtracking problem, it normally asks for finding all the possible paths or some sorts so let us keep that in mind to differentiate.

So in conclusion, look for some keywords when we need to solve problems.
If it states the shortest path with the same cost for each move, try BFS. But if it asks us to find all possible paths or each cost differs, try to see if we can use Backtracking.

p.s. Let me add two samples: one for bfs and the other for backtracking for future reference.

Friday, July 22, 2011

WinHttpSendRequest is not so easy to use

The other day I tried to look up WinHttpSendRequest() API for more information and found that the second entry in google search result was my question that I posted at MSDN forum two years ago with regard to its usage example.
At that time I was trying to understand how to use WinHttpSendRequest() API to construct Http POST request and could not find any good example on the web and hence, I decided that I should write something about it in my blog.

Followings are the basic flow of WinHttp calls to create Http Post request.

  • Prepare WinHttp session handle and connection handle. We can reuse these two handles to send multiple requests. For this use WinHttpOpen and WinHttpConnect API
  • Create a specific WinHttp request handle. Once we have the request handle, we can associate certain options with this handle. Use WinHttpOpenRequest.
  • Construct POST request and details are as follows. This is where it gets real tricky.

In order to successfully send our Http Request, we need two APIs: WinHttpAddRequestHeaders() and WinHttpSendRequest()

First, I assume that we want to send multipart form HTTP Post request to upload a file along with some fields. For this task, we need to specify our intention but what's tricky is the fact that we need to specify the boundary of http header so that Web server knows where the header ends and how to parse our request. Here is an example of its usage.

WinHttpAddRequestHeaders(
    hRequest, 
    "Content-Type:multipart/form-data; boundary=----------ThIs_Is_tHe_bouNdaRY_$", 
    -1L, 
    WINHTTP_ADDREQ_FLAG_ADD
);

As you can see, we have put certain string such as "boundary=----------ThIs_Is_tHe_bouNdaRY_$" to inform the web server to look for these string to parse the request correctly.

Now, it is time to construct all the fields. Let me give you one example of form field.
Say we have a field called "username" and want to specify its value "ilho". Following is the sequence of string we need to construct and just to make it easy to read, I added 'new line' after '\r\n' but in reality you need to have all the strings together as one string. You need to add the same format for all the fields to the same string.

------------ThIs_Is_tHe_bouNdaRY_$\r\n
Content-Disposition: form-data; name="username"\r\n\r\n
ilho\r\n

For data file that we want to upload, we need special care. Now, assume that we want to upload some text file. Here is the string that you want to build.

------------ThIs_Is_tHe_bouNdaRY_$\r\n
Content-Disposition: form-data; name="personal_data";\r\n filename="personal.txt"\r\n
Content-Type: text/plain; charset=utf-8\r\n\r\n

Finally, you will need to end your request by specifying the ending as follows.

\r\n------------ThIs_Is_tHe_bouNdaRY_$--\r\n\r\n

I know that these look really ugly and I do not know if there is any better way to do this. Unless I am mistaken, I think for this specific part, I prefer to use curl because it is a lot easier to use curl to construct POST request than WinHttp.

But for now, let me throw out some code example to illustrate what we need.

StringCbPrintf(
    ffield,
    sizeof(ffield),
    "------------ThIs_Is_tHe_bouNdaRY_$\r\n"
    "Content-Disposition: form-data; name="username"\r\n\r\n"
    "ilho\r\n"
);

StringCbPrintf(
    filefield,
    sizeof(filefield),
    "------------ThIs_Is_tHe_bouNdaRY_$\r\n"
    "Content-Disposition: form-data; name=\"personal_data\"; filename=\"personal.txt\"\r\n"
    "Content-Type: text/xml; charset=utf-8\r\n\r\n"
);

StringCbPrintf(formdata, sizeof(formdata), "%s%s", ffield, filefield);

Once we have the form ready, we need to calculate the entire size of request which includes the actual text file. With all these information, here is WinHttpSendRequest.

total_length = strlen(formdata) + filesize + strlen("\r\n------------ThIs_Is_tHe_bouNdaRY_$--\r\n\r\n");


WinHttpSendRequest(
    hRequest,
    WINHTTP_NO_ADDITIONAL_HEADERS,
    0,
    (LPVOID)formdata,
    (DWORD)strlen(formdata),
    total_length,
    0
);

Once we send our HTTP request, we can start to send our text file using WinHttpWriteData followed by boundary string to finish the HTTP Post request. Hence, in minimum we need to call two WinHttpWriteData.

WinHttpWriteData(file content);
WinHttpWriteData("\r\n------------ThIs_Is_tHe_bouNdaRY_$--\r\n\r\n");  // boudary

At this point, we just need to receive the Web server response to your request by calling WinHttpReceiveResponse() API and we are pretty much done by now.

In the above example, I tried to present the steps in a rather raw format but in reality you may want to create either inline functions or macros to to reduce the typings and create a nice interface to work with WinHttp. Like I said, if you need to work on the similar code for both Windows and Linux, it is better off to use curl and I hope that MS will come up with better APIs or interfaces for the developers to work with Http requests.

Tuesday, July 12, 2011

Windows Startup Routine

It's has been a while since I looked at Windows startup routine and I realized that I don't remember the overview of it. So I grabbed Windows Internals and summarized it here for my future reference.

1. POST (Power On Self Test)

2. BIOS loads MBR into memory

3. MBR reads the first sector from the bootable partition and the code in the first sector loads the Bootmgr into memory

4. Bootmgr loads the boot loader, Winload.exe (Ntldr in the past)
Winload.exe
- loads the appropriate kernel and HAL images (Ntoskrnl.exe and Hal.dll)
- reads registry to determine boot device drivers and loads them.
- calls KiSystemStartup in Ntoskrnl.exe

Ntokrnl.exe
- phase 0
 prepares data structures for services to run in phase 1
 calls the object manager, security reference monitor, process
manager, user-mode debugging framework, and the Plug and Play manager initialization
- phase 1
 calls more initializations i.e. power manager, memory manager
 I/O manager initialization
 - creates I/O manager object types i.e. device, driver
 - perform boot-start drivers initialization and system-start device drivers are loaded and initialized
 - the Session manager(Smss) process is started. Smss is responsible for creating user-mode environment that provides the visible interface to Windows

Smss
 - start the Windows subsystem
 - start Csrss which is Client Server Runtime Process
  - CSRSS: user-mode portion of the Win32 subsystem; Win32.sys is the kernel-mode portion.
 - launches Windows Initialization Process (Wininit) and start the interactive   logon manager process (Winlogon)

Wininit
 - performs startup steps such as creating the initial window station and desktop objects
 - creates SCM which initializes auto-start services and drivers

Once Winlogon validates the logon, Winlogon sets up the user environment and starts the shell(explorer)



That's it for now and hopefully, in the future I will try to add some diagram to describe what I wrote in the above.