Hall of fame ... CCWS !

An era has come to an end, RIP CCWS..tears! Seems I have entered a endless black hole.
After a long agony, it was decided to finalize dooms day for CCWS. Hundreds of people
would be experiencing this live shit. For so many years this has be source of bread and butter and biryani and chai.

 The partial closure would soon follow dumping of whole traditional
ASP/SQL CCWS. My deep condolences for all those days gone by thousands of hours spent on tons
of customer problems. Many good and sad memories for so many customers and their reactions goes down into dust with CCWS.

I lost the privilege to put my last adieu CCWS, nevertheless my unknown feeling will always be with you. Retrospective be it, but I always felt very simple and useful to work with. "Purana jayega, naya aayega, dekte hai SF kya karega.."

 

Close Encounters with AppVerifier

With the release of one of our products, one of the customers came to a standstill.  Obvious reason was the new version couldn’t sustain on older platform Win 2003.  The mixed mode application died before it could start. Thinking it could be a slam-dunk bombarded the entire arsenal available including following.

•  Use of certified vcredist_x86  (VS2008 SP1) (target machines were still 32bit)
•  Removal of older registry
•  Windows patching
•  Manually copying VC libraries (CRT,MFC etc) from working machine into target Win 2003 winsxs folder.
• Interestingly older version of the product works fine, so verified all binaries closely with versions and their dependency.
• Tried with comparing outputs of Process monitor, but log outputs of working and non-working were almost same, no clue.
• Installed .NET4.0 x86 , although required was only 3.5 SP1

The documents said recommended was .NET3.5 SP1, but I as per my knowledge the CLR loaded with the application was 2.0. Of course .NET 3.5 SP1 still holds CLR2.0. This was replicable on all Win 2003 machines, and I still had no clue how to go about this issue.

Not even Windbg gave any clues, the stack just died before it could load any useful information. Since App could not start it was difficult to attach a debugger. From search I could get some info using Gflags , so I set the Gflags pointing the .exe under image options and then checking the debugger and its path.

Now when I click the .exe , Windbg pops up and it loads the first DLL. Using next go commands a few DLL’s been loaded, but at one point loading a third party DLL, it threw AV c0000005, and unloaded DLL. That’s it. Still in no man’s land!

After few more searches, I came across MSDN WIT blogs, which used AppVerifier and thought to give a try. Although I had just seen this once,  but never used it. Attached the filename of exe to AppVerifier and checked few basic options. Checking the logs showed nothing, but same old access violation error. Decided to check all possible options, and voila!!  The app started! 

So something had done the trick. Now it was time to uncheck one by one settings, and catch the culprit (insane, he is the hero for me) who made it work. Surprisingly its HigherversionLie which makes it work on Win 2003. MSDN showed the HigherversionLie is used for integration testing , and this one actually raises the Major/Minor version to +2 while testing. So in all it seemed the app was trying to find version of OS before startup using GetVersion/GetVersionEx possibly and failed if it returned < 5. Since HigherversionLie raised this bar with +2 , it worked! Rest all in Dev hands to check..Good night, and yes Merry Christmas – Happy Holidays…

.NET DLL Loading

It is frustrating to see workaround/fixes doesn’t work at customers end, with same environments
on which support replicates. To add fuel to the Fire, our tools too don’t show up big expected 
results when client is showing issue :(

Towards last couple of months , I have seen this recurring issues of DLL loading, not working
as expected, .NET security permissions on code executions etc.

           In one of scenario, the application using .NET DLL, fails to run at customer while 
executing a particular process. Application calls a process from .NET DLL to run on a context 
menu. At customer it fails to run, even DLL entry points seems not reached. Locally one can run 
without issues, but at customer its restricted, the paths are on application server. Also this application uses partially signed DLL's , instead of strong names (placed in GAC) which makes things more painful for debugging.

Now most of relevant investigation being done including following :

1) DLL registered with regasm on path (app shared)
2) Full admin rights on shared path
3) Fulltrust granted to .NET code on shared path
4) Removed any unwanted conflicting paths of DLL's from registry, where CLSID for the DLL was 
registered.
5) Fusion log viewer shows same paths are been picked successfully from where it has to.

Another hitch here is that, there is nothing such a flag / return value to check if DLL's are
registered successfully. So either one has to manual scan registry hive or code to check if 
DLL's instance can be used to create an object. Both are painful. Even legacy DLL registration which internally calls DllRegisterServer, has no way to check if registration got successful. See this blog.

Nevertheless, one can debug to attach running process to a debugger to check DLL loading/paths 
etc, or much easier, I would say is Sysinternals Process explorer, or another tool I saw was
from Nirsoft's RegDllView.exe. For RegDllView, from my testing, I could not see called DLL's 
instantly being displayed, but for procexp.exe, I am 99% sure it should work. So just went ahead
launch my application , called .NET DLL, and ran procexp.exe, locate the app.exe from list of 
processes.  User View menu to check lower pane, then use View->lower pane view->DLL's.
Now click on app.exe, at the bottom you will see list of DLL's loaded and information like 
version, path, manufacturer etc. I think this is a clear win here. See process explorer in action below...

 

NET interview questions 6th edition released..

Just sharing new release of my favorite book  from Shivprasad here at

http://www.dotnetinterviewquestions.in/article_net-interview-questions-6th-edition-sixth-edition-by-shivprasad-koirala_144.html

The new release has some exciting stuff, like questions on debugging and releases. Will check definitely check this topic, since at production this is very much needed and handy for interview.

keep reading more...

cheers
kirann

Oracle Banner

This is an offside article, but as I could not find much on web, I thought this should be blogged even though simple enough info. You never know who might hit with this.

Recently while working on a Oracle connection issue, Customer confirmed their issue was resolved by turning off the Oracle banner. This was the first time I heard of banner being turned off and on.
I had to research along with DBA to enable disable banner in Oracle, even though we could not confirm whether banner was culprit for our application connection. Not much info on web, had to struggle.

Eventually found that Oracle 11.2.xx.xx has this facility. One can simple put a banner text file
on disk, and insert into sqlnet.ora below

SEC_USER_AUDIT_ACTION_BANNER = c:\banner.txt

sqlnet.ora file is typically found under ..product\11.2.0\dbhome_1\NETWORK\ADMIN

After this bounce database services and login into Oracle to check banner text something similar to below.
Note here that we are talking this on Windows platform, on unix flavors path is different.

 

Res-Q'd by Spy++

Today while searching for a file on a windows XP machine, I launched desktop search (win+F), and as usually I don’t use this, I preferred original search to be used. After finishing the search, as I was about to close desktop search, I see following error and unable to close this window. Also mysteriously this window had lost its title, just showing folder icon.

This window is busy. Closing this window may cause some problems. Do you want to close it anyway. OK/Cancel ?

Thinking not such a big deal, I threw up task manger to get process to be killed, much to my surprise the related processes viz searchindexer.exe, cidaemon.exe, SearchProtocolHost.exe does not had impact on the hung window after killing them. Lost in thinking what would be process name for desktop search ???

Neither Google provided any related names that I could think of. Launching more and more desktop search windows just stood hung there without being able to be closed. I thought to launch Sysinternals process explorer to see under which process this might be disguising. I find NOTHING L

I kept closing all unnecessary application keeping just those windows there. In the midst of trying to check which process is launched or highlighted in process explorer, I had already gathered 10 windows of hung desktop search. What the heck, it was already 1 hour!

Played another half hour researching my toolbox items to see if there would be any tool which can display which process is hiding desktop exe. Find handle search did not give any results. The find process window of procexp too showed up process this time Explorer.exe. Something was now at least clear. But the properties window showed many Non-existent processes. Hmm!

Finally sometime later I came up with Spy++ to track down messages. You may know that this is a great tool to look messages sent by windows and also shows related threads. Using its find window option quickly showed me that Explorer was using desktop search. Scanning all threads was not a difficult job as each thread had the messages and button names similar to desktop search.

 

Now it was easy to grab the thread ID like the one shown below (hex), use calc.exe to convert hex to decimal.

And later used process explorer, track down explorer .exe, double click to show open the properties. Check the thread Id converted above, and use Kill option to kill the thread

Wow, the hung window killed! Alas 2 hours of fruitful research don’t know how that thread hung? And what were other possible consequences of killing the thread, anyhow I did not face anything unusual, as desktop search worked as before J

kirann

Activation context being deactivated is not the most recently activated one..

July 1 2011 was a dooms day for our customers! Early morning this date struck initially APJ reason, soon spreading the error across globally by afternoon.Crisis started with application not being able to start and popping the error

A quick search in for the exception code said C015000F is The activation context being deactivated

and had something to do with MFC's thread state activation-deactivation.
Following this info I reproduce the error and attached the debugger to get a clear call stack.

0:000> kb


*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\Program Files\TEST\MyApp MyApp1.6\Bin\MSVCR90.dll -

ChildEBP RetAddr Args to Child
 WARNING: Stack unwind information not available. Following frames may be wrong.
0013b8cc 7857dca0 c015000f 0013b8e4 00642c38 Foundation!FUserInterruptException::FUserInterruptException+0x532
0013b918 7857d4eb 0013bb08 0013f374 0013bb28 MSVCR90!_CxxLongjmpUnwind+0x101
0013b94c 7857d8b6 0013bb08 0013f374 0013bb28 MSVCR90!_CxxExceptionFilter+0x217
0013b9ac 7857d9d3 0013bb08 0013f374 0013bb28 MSVCR90!_CxxExceptionFilter+0x5e2
0013b9e0 7857db8a 0013bb08 0013f374 0013bb28 MSVCR90!_CxxExceptionFilter+0x6ff
0013ba1c 7c9032a8 0013bb08 0013f374 0013bb28 MSVCR90!_CxxFrameHandler3+0x26
3ba40 7c90327a 0013bb08 0013f374 0013bb28 ntdll!ExecuteHandler2+0x26
0013baf0 7c90e48a 00000000 0013bb28 0013bb08 ntdll!ExecuteHandler+0x24
0013baf0 7c9446da 00000000 0013bb28 0013bb08 ntdll!KiUserExceptionDispatcher+0xe
0013be54 7c80a73d 00000000 129c001a 0013def0 ntdll!RtlDeactivateActivationContext+0x15f
0013be64 78a43f5d 00000000 129c001a 78590abe KERNEL32!DeactivateActCtx+0x31
0013def0 78a3ea8e 00000000 002d1434 073754c0 mfc90u+0x63f5d
0013df64 78a44d56 00000000 002d1434 073754c0 mfc90u+0x5ea8e
0013dfa8 78a44fcf 002d1434 073754c0 00cf8000 mfc90u+0x64d56
0013dff0 78a489f4 07363f88 00cf8000 00000000 mfc90u+0x64fcf
0013e00c 00405b35 00000001 00cf8000 00000000 mfc90u+0x689f4
0013f380 78a4aef9 006c3c3c 00000001 00000000 MyApp!FMyApp::InitInstance+0x465 [c:\MyApp\sources\rg_previous_160\MyApp\MyApp\MyApp.cpp @ 427]
0013f394 00589608 00400000 00000000 00020d6a mfc90u+0x6aef9
0013f428 035bb213 19f4ee06 00000000 00000001 MyApp!__tmainCRTStartup+0x150 [f:\dd\vctools\crt_bld\self_x86\crt\src\crtexe.c @ 578]


*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscorwks.dll -


0013f440 79e71b4c 00000000 0026fac8 0013f48c +0x35bb212


So after call to InitInstance there are sequence of MFC90u.dll calls finally calling DeactivateActCtx.
Still no clue why would DeactivateActCtx would fail  :(
Between a nice article on MSDN blogs here for DeactivateActCtx.

I then look back upwards before native call stack , as soon as got the exception and debugger break, and I see below
.....


ModLoad: 77920000 77a13000 C:\WINDOWS\system32\SETUPAPI.dll
(4b98.529c): CLR exception - code e0434f4d (first chance)
(4b98.529c): Unknown exception - code c015000f (first chance)
(4b98.529c): Break instruction exception - code 80000003 (first chance)
eax=002b6710 ebx=00642c38 ecx=d2017b54 edx=00000001 esi=0013bb08 edi=00031f30
eip=01d29f72 esp=0013b7bc ebp=0013b8cc iopl=0 nv up ei pl nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202
....

Ok now seems good point to check some managed stuff, as we see that a CLR exception is raised.

0:000> .loadby sos mscorwks


0:000> !CLRStack
PDB symbol for mscorwks.dll not loaded
OS Thread Id: 0x529c (0)
ESP EIP
0013f458 01d29f72 [NDirectMethodFrameStandalone: 0013f458] ._wWinMainCRTStartup()
0013f688 01d29f72 [GCFrame: 0013f688]

 So nothing there on the managed stack. Lets check the exception thrown

0:000> !pe


Exception object: 04df0f7c
Exception type: System.Exception
Message: Component usage expired
 InnerException:
StackTrace (generated):
SP IP Function


0013C988 06ED8113 !ThirdPartyDll.xrl.Processor.v4xrlProcessor..ctor()+0x10b
0013C9BC 06ED7FE8 !xrl.xrlProcessor..ctor()+0x18
0013C9C8 06ED7F79 Environment!.xrlProcessorWrapper.{ctor}(xrlProcessorWrapper*)+0x59
0013C9F8 06ED7EF7 Environment!.xrlProcessorWrapperBase.CreatexrlProcessorInstance()+0x27

 tackTraceString:


HResult: 80131500


0:000> !do 04df0f7c
Name: System.Exception
MethodTable: 79330cb8
EEClass: 790ed72c
Size: 72(0x48) bytes
(C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)

Fields:
MT Field Offset Type VT Attr Value Name
79330b24 40000b5 4 System.String 0 instance 00000000 _className
7932ff98 40000b6 8 ...ection.MethodBase 0 instance 00000000 _exceptionMethod
79330b24 40000b7 c System.String 0 instance 00000000 _exceptionMethodString
79330b24 40000b8 10 System.String 0 instance 04df0fc4 _message
7932a480 40000b9 14 ...tions.IDictionary 0 instance 00000000 _data
79330cb8 40000ba 18 System.Exception 0 instance 00000000 _innerException
79330b24 40000bb 1c System.String 0 instance 00000000 _helpURL
79330740 40000bc 20 System.Object 0 instance 04df1064 _stackTrace
79330b24 40000bd 24 System.String 0 instance 00000000 _stackTraceString
79330b24 40000be 28 System.String 0 instance 00000000 _remoteStackTraceString
79332d70 40000bf 34 System.Int32 1 instance 0 _remoteStackIndex
79330740 40000c0 2c System.Object 0 instance 00000000 _dynamicMethods
79332d70 40000c1 38 System.Int32 1 instance -2146233088 _HResult
79330b24 40000c2 30 System.String 0 instance 00000000 _source
793333ec 40000c3 3c System.IntPtr 1 instance 0 _xptrs
79332d70 40000c4 40 System.Int32 1 instance -532459699 _xcode


0:000> !do 04df0fc4
Name: System.String
MethodTable: 79330b24
EEClass: 790ed65c
Size: 64(0x40) bytes
(C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String: Component usage expired
Fields:
MT Field Offset Type VT Attr Value Name
79332d70 4000096 4 System.Int32 1 instance 24 m_arrayLength
79332d70 4000097 8 System.Int32 1 instance 23 m_stringLength
79331804 4000098 c System.Char 1 instance 43 m_firstChar
79330b24 4000099 10 System.String 0 shared static Empty
>> Domain:Value 04965080:04d01198 <<
79331754 400009a 14 System.Char[] 0 shared static WhitespaceChars
>> Domain:Value 04965080:04d017b4 <<


We see that same exception thrown into popup 'Component usage expired' that comes from a DLL 'ThirdPartyDll'

Since we do not have control on this, consulted dev, who gave us new version of DLL which
undoutedly resolved the error.