Activation context being deactivated is not the most recently activated one..

July 1 2011 was a dooms day for our customers! Early morning this date struck initially APJ reason, soon spreading the error across globally by afternoon.Crisis started with application not being able to start and popping the error

A quick search in for the exception code said C015000F is The activation context being deactivated

and had something to do with MFC's thread state activation-deactivation.
Following this info I reproduce the error and attached the debugger to get a clear call stack.

0:000> kb


*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\Program Files\TEST\MyApp MyApp1.6\Bin\MSVCR90.dll -

ChildEBP RetAddr Args to Child
 WARNING: Stack unwind information not available. Following frames may be wrong.
0013b8cc 7857dca0 c015000f 0013b8e4 00642c38 Foundation!FUserInterruptException::FUserInterruptException+0x532
0013b918 7857d4eb 0013bb08 0013f374 0013bb28 MSVCR90!_CxxLongjmpUnwind+0x101
0013b94c 7857d8b6 0013bb08 0013f374 0013bb28 MSVCR90!_CxxExceptionFilter+0x217
0013b9ac 7857d9d3 0013bb08 0013f374 0013bb28 MSVCR90!_CxxExceptionFilter+0x5e2
0013b9e0 7857db8a 0013bb08 0013f374 0013bb28 MSVCR90!_CxxExceptionFilter+0x6ff
0013ba1c 7c9032a8 0013bb08 0013f374 0013bb28 MSVCR90!_CxxFrameHandler3+0x26
3ba40 7c90327a 0013bb08 0013f374 0013bb28 ntdll!ExecuteHandler2+0x26
0013baf0 7c90e48a 00000000 0013bb28 0013bb08 ntdll!ExecuteHandler+0x24
0013baf0 7c9446da 00000000 0013bb28 0013bb08 ntdll!KiUserExceptionDispatcher+0xe
0013be54 7c80a73d 00000000 129c001a 0013def0 ntdll!RtlDeactivateActivationContext+0x15f
0013be64 78a43f5d 00000000 129c001a 78590abe KERNEL32!DeactivateActCtx+0x31
0013def0 78a3ea8e 00000000 002d1434 073754c0 mfc90u+0x63f5d
0013df64 78a44d56 00000000 002d1434 073754c0 mfc90u+0x5ea8e
0013dfa8 78a44fcf 002d1434 073754c0 00cf8000 mfc90u+0x64d56
0013dff0 78a489f4 07363f88 00cf8000 00000000 mfc90u+0x64fcf
0013e00c 00405b35 00000001 00cf8000 00000000 mfc90u+0x689f4
0013f380 78a4aef9 006c3c3c 00000001 00000000 MyApp!FMyApp::InitInstance+0x465 [c:\MyApp\sources\rg_previous_160\MyApp\MyApp\MyApp.cpp @ 427]
0013f394 00589608 00400000 00000000 00020d6a mfc90u+0x6aef9
0013f428 035bb213 19f4ee06 00000000 00000001 MyApp!__tmainCRTStartup+0x150 [f:\dd\vctools\crt_bld\self_x86\crt\src\crtexe.c @ 578]


*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscorwks.dll -


0013f440 79e71b4c 00000000 0026fac8 0013f48c +0x35bb212


So after call to InitInstance there are sequence of MFC90u.dll calls finally calling DeactivateActCtx.
Still no clue why would DeactivateActCtx would fail  :(
Between a nice article on MSDN blogs here for DeactivateActCtx.

I then look back upwards before native call stack , as soon as got the exception and debugger break, and I see below
.....


ModLoad: 77920000 77a13000 C:\WINDOWS\system32\SETUPAPI.dll
(4b98.529c): CLR exception - code e0434f4d (first chance)
(4b98.529c): Unknown exception - code c015000f (first chance)
(4b98.529c): Break instruction exception - code 80000003 (first chance)
eax=002b6710 ebx=00642c38 ecx=d2017b54 edx=00000001 esi=0013bb08 edi=00031f30
eip=01d29f72 esp=0013b7bc ebp=0013b8cc iopl=0 nv up ei pl nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202
....

Ok now seems good point to check some managed stuff, as we see that a CLR exception is raised.

0:000> .loadby sos mscorwks


0:000> !CLRStack
PDB symbol for mscorwks.dll not loaded
OS Thread Id: 0x529c (0)
ESP EIP
0013f458 01d29f72 [NDirectMethodFrameStandalone: 0013f458] ._wWinMainCRTStartup()
0013f688 01d29f72 [GCFrame: 0013f688]

 So nothing there on the managed stack. Lets check the exception thrown

0:000> !pe


Exception object: 04df0f7c
Exception type: System.Exception
Message: Component usage expired
 InnerException:
StackTrace (generated):
SP IP Function


0013C988 06ED8113 !ThirdPartyDll.xrl.Processor.v4xrlProcessor..ctor()+0x10b
0013C9BC 06ED7FE8 !xrl.xrlProcessor..ctor()+0x18
0013C9C8 06ED7F79 Environment!.xrlProcessorWrapper.{ctor}(xrlProcessorWrapper*)+0x59
0013C9F8 06ED7EF7 Environment!.xrlProcessorWrapperBase.CreatexrlProcessorInstance()+0x27

 tackTraceString:


HResult: 80131500


0:000> !do 04df0f7c
Name: System.Exception
MethodTable: 79330cb8
EEClass: 790ed72c
Size: 72(0x48) bytes
(C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)

Fields:
MT Field Offset Type VT Attr Value Name
79330b24 40000b5 4 System.String 0 instance 00000000 _className
7932ff98 40000b6 8 ...ection.MethodBase 0 instance 00000000 _exceptionMethod
79330b24 40000b7 c System.String 0 instance 00000000 _exceptionMethodString
79330b24 40000b8 10 System.String 0 instance 04df0fc4 _message
7932a480 40000b9 14 ...tions.IDictionary 0 instance 00000000 _data
79330cb8 40000ba 18 System.Exception 0 instance 00000000 _innerException
79330b24 40000bb 1c System.String 0 instance 00000000 _helpURL
79330740 40000bc 20 System.Object 0 instance 04df1064 _stackTrace
79330b24 40000bd 24 System.String 0 instance 00000000 _stackTraceString
79330b24 40000be 28 System.String 0 instance 00000000 _remoteStackTraceString
79332d70 40000bf 34 System.Int32 1 instance 0 _remoteStackIndex
79330740 40000c0 2c System.Object 0 instance 00000000 _dynamicMethods
79332d70 40000c1 38 System.Int32 1 instance -2146233088 _HResult
79330b24 40000c2 30 System.String 0 instance 00000000 _source
793333ec 40000c3 3c System.IntPtr 1 instance 0 _xptrs
79332d70 40000c4 40 System.Int32 1 instance -532459699 _xcode


0:000> !do 04df0fc4
Name: System.String
MethodTable: 79330b24
EEClass: 790ed65c
Size: 64(0x40) bytes
(C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String: Component usage expired
Fields:
MT Field Offset Type VT Attr Value Name
79332d70 4000096 4 System.Int32 1 instance 24 m_arrayLength
79332d70 4000097 8 System.Int32 1 instance 23 m_stringLength
79331804 4000098 c System.Char 1 instance 43 m_firstChar
79330b24 4000099 10 System.String 0 shared static Empty
>> Domain:Value 04965080:04d01198 <<
79331754 400009a 14 System.Char[] 0 shared static WhitespaceChars
>> Domain:Value 04965080:04d017b4 <<


We see that same exception thrown into popup 'Component usage expired' that comes from a DLL 'ThirdPartyDll'

Since we do not have control on this, consulted dev, who gave us new version of DLL which
undoutedly resolved the error.