Close Encounters with AppVerifier

With the release of one of our products, one of the customers came to a standstill.  Obvious reason was the new version couldn’t sustain on older platform Win 2003.  The mixed mode application died before it could start. Thinking it could be a slam-dunk bombarded the entire arsenal available including following.

•  Use of certified vcredist_x86  (VS2008 SP1) (target machines were still 32bit)
•  Removal of older registry
•  Windows patching
•  Manually copying VC libraries (CRT,MFC etc) from working machine into target Win 2003 winsxs folder.
• Interestingly older version of the product works fine, so verified all binaries closely with versions and their dependency.
• Tried with comparing outputs of Process monitor, but log outputs of working and non-working were almost same, no clue.
• Installed .NET4.0 x86 , although required was only 3.5 SP1

The documents said recommended was .NET3.5 SP1, but I as per my knowledge the CLR loaded with the application was 2.0. Of course .NET 3.5 SP1 still holds CLR2.0. This was replicable on all Win 2003 machines, and I still had no clue how to go about this issue.

Not even Windbg gave any clues, the stack just died before it could load any useful information. Since App could not start it was difficult to attach a debugger. From search I could get some info using Gflags , so I set the Gflags pointing the .exe under image options and then checking the debugger and its path.

Now when I click the .exe , Windbg pops up and it loads the first DLL. Using next go commands a few DLL’s been loaded, but at one point loading a third party DLL, it threw AV c0000005, and unloaded DLL. That’s it. Still in no man’s land!

After few more searches, I came across MSDN WIT blogs, which used AppVerifier and thought to give a try. Although I had just seen this once,  but never used it. Attached the filename of exe to AppVerifier and checked few basic options. Checking the logs showed nothing, but same old access violation error. Decided to check all possible options, and voila!!  The app started! 

So something had done the trick. Now it was time to uncheck one by one settings, and catch the culprit (insane, he is the hero for me) who made it work. Surprisingly its HigherversionLie which makes it work on Win 2003. MSDN showed the HigherversionLie is used for integration testing , and this one actually raises the Major/Minor version to +2 while testing. So in all it seemed the app was trying to find version of OS before startup using GetVersion/GetVersionEx possibly and failed if it returned < 5. Since HigherversionLie raised this bar with +2 , it worked! Rest all in Dev hands to check..Good night, and yes Merry Christmas – Happy Holidays…

.NET DLL Loading

It is frustrating to see workaround/fixes doesn’t work at customers end, with same environments
on which support replicates. To add fuel to the Fire, our tools too don’t show up big expected 
results when client is showing issue :(

Towards last couple of months , I have seen this recurring issues of DLL loading, not working
as expected, .NET security permissions on code executions etc.

           In one of scenario, the application using .NET DLL, fails to run at customer while 
executing a particular process. Application calls a process from .NET DLL to run on a context 
menu. At customer it fails to run, even DLL entry points seems not reached. Locally one can run 
without issues, but at customer its restricted, the paths are on application server. Also this application uses partially signed DLL's , instead of strong names (placed in GAC) which makes things more painful for debugging.

Now most of relevant investigation being done including following :

1) DLL registered with regasm on path (app shared)
2) Full admin rights on shared path
3) Fulltrust granted to .NET code on shared path
4) Removed any unwanted conflicting paths of DLL's from registry, where CLSID for the DLL was 
registered.
5) Fusion log viewer shows same paths are been picked successfully from where it has to.

Another hitch here is that, there is nothing such a flag / return value to check if DLL's are
registered successfully. So either one has to manual scan registry hive or code to check if 
DLL's instance can be used to create an object. Both are painful. Even legacy DLL registration which internally calls DllRegisterServer, has no way to check if registration got successful. See this blog.

Nevertheless, one can debug to attach running process to a debugger to check DLL loading/paths 
etc, or much easier, I would say is Sysinternals Process explorer, or another tool I saw was
from Nirsoft's RegDllView.exe. For RegDllView, from my testing, I could not see called DLL's 
instantly being displayed, but for procexp.exe, I am 99% sure it should work. So just went ahead
launch my application , called .NET DLL, and ran procexp.exe, locate the app.exe from list of 
processes.  User View menu to check lower pane, then use View->lower pane view->DLL's.
Now click on app.exe, at the bottom you will see list of DLL's loaded and information like 
version, path, manufacturer etc. I think this is a clear win here. See process explorer in action below...

 

Res-Q'd by Spy++

Today while searching for a file on a windows XP machine, I launched desktop search (win+F), and as usually I don’t use this, I preferred original search to be used. After finishing the search, as I was about to close desktop search, I see following error and unable to close this window. Also mysteriously this window had lost its title, just showing folder icon.

This window is busy. Closing this window may cause some problems. Do you want to close it anyway. OK/Cancel ?

Thinking not such a big deal, I threw up task manger to get process to be killed, much to my surprise the related processes viz searchindexer.exe, cidaemon.exe, SearchProtocolHost.exe does not had impact on the hung window after killing them. Lost in thinking what would be process name for desktop search ???

Neither Google provided any related names that I could think of. Launching more and more desktop search windows just stood hung there without being able to be closed. I thought to launch Sysinternals process explorer to see under which process this might be disguising. I find NOTHING L

I kept closing all unnecessary application keeping just those windows there. In the midst of trying to check which process is launched or highlighted in process explorer, I had already gathered 10 windows of hung desktop search. What the heck, it was already 1 hour!

Played another half hour researching my toolbox items to see if there would be any tool which can display which process is hiding desktop exe. Find handle search did not give any results. The find process window of procexp too showed up process this time Explorer.exe. Something was now at least clear. But the properties window showed many Non-existent processes. Hmm!

Finally sometime later I came up with Spy++ to track down messages. You may know that this is a great tool to look messages sent by windows and also shows related threads. Using its find window option quickly showed me that Explorer was using desktop search. Scanning all threads was not a difficult job as each thread had the messages and button names similar to desktop search.

 

Now it was easy to grab the thread ID like the one shown below (hex), use calc.exe to convert hex to decimal.

And later used process explorer, track down explorer .exe, double click to show open the properties. Check the thread Id converted above, and use Kill option to kill the thread

Wow, the hung window killed! Alas 2 hours of fruitful research don’t know how that thread hung? And what were other possible consequences of killing the thread, anyhow I did not face anything unusual, as desktop search worked as before J

kirann