MSDN Blogs: How to Identify Why an Application Pool is Recycling in IIS 7+ and DebugDiag 2 Fails to Add Managed Breakpoint in .NET 4.6+ in one post

Identifying why Application Pool is recycling

Starting with IIS 7, application recycle for managed Web Applications does not recycle w3wp.exe (IIS Worker Process), instead it simply reload (unload and load a new one) the Application Domain. I mentioned it on another post how to identify the cause of an application pool recycle, but now I will tell how to capture a dump file when it happens.

Capturing a dump when an Application Pool recycles is as easy as capturing a dump file at System.AppDomain.Unload(System.AppDomain) via DebugDiag crash rule. However, when doing this on a machine with .NET 4.6, I was unable to capture the dump. Tinkering a bit I found how to resolve the issue via workaround which I explain later on this post.

After you capturing the dump, it is easy to find the cause if you use NetExt: Getting started with NetExt

Open dump file with WinDbg, load NetExt, index heap with !windex

Image may be NSFW.
Clik here to view.

The run !wruntime to know the cause:

0:032>; !wruntime

Runtime Settings per Application Pool

=========================================================================

Address         : 0000000DB8DDB100

First Request   : 10/29/2015 11:34:50 PM

Runing Time     : 00:00:07

App Pool User   : IIS APPPOOL\DefaultAppPool

Trust Level     : Full

App Domnain Id  : /LM/W3SVC/1/ROOT-4-130906352796685448

Debug Enabled   : False

Active Requests : 0n0

Path            : C:\inetpub\wwwroot\ (local disk)

Temp Folder     : C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files

Compiling Folder: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\e22c2559\92c7e946

Shutdown Reason : ConfigurationChange at 10/29/2015 11:34:57 PM

IIS configuration change

HostingEnvironment initiated shutdown

HostingEnvironment caused shutdownn   at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)

   at System.Environment.get_StackTrace()

   at System.Web.Hosting.HostingEnvironment.InitiateShutdownInternal()

   at System.Web.Hosting.HostingEnvironment.InitiateShutdownWithoutDemand()

   at System.Web.Hosting.PipelineRuntime.StopProcessing()

To identify the file(s) changed, run this command: !wfrom -type System.Web.FileMonitor where ($enumname(_lastAction) == "Modified") $a("==================\nPath",DirectoryMonitor.Directory+"\\"+_fileNameLong),$a("Change Time:",$tickstodatetime(_utcLastCompletion.dateData)),$a("Action",$enumname(_lastAction))

0:032>; !wfrom -type System.Web.FileMonitor where ($enumname(_lastAction) == "Modified") $a("==================\nPath",DirectoryMonitor.Directory+"\\"+_fileNameLong),$a("Change Time:",$tickstodatetime(_utcLastCompletion.dateData)),$a("Action",$enumname(_lastAction))

==================

Path: C:\inetpub\wwwroot\web.config

Change Time:: 10/29/2015 11:34:57 PM

Action: Modified

==================

1 Object(s) listed

45 Object(s) skipped by filter

Setting a managed Breakpoint for Application pool to catch managed recycling

If you try to add a DebugDiag crash rule when a managed breakpoint is hit, you may fail if .NET 4.6+ is installed. I tested it with DebugDiag 2 Update 1 and it still fails. This is a step-by-step instruction that I sent to a customer having this issue. The instructions also work for people not having the issue, just skip the steps related to .NET 4.6.

1. Download DebugDiag 2 Update 1 and install on the server showing the problem: http://www.microsoft.com/en-us/download/details.aspx?id=42933
2. Run DebugDiag 2 Collection
3. Choose Crash on Rule Type and click Next

Image may be NSFW.
Clik here to view.

4. Then choose A specific IIS web application pool and click Next

Image may be NSFW.
Clik here to view.

5. In the next window, choose the Application Pool you are having problem with (you will see different options in your environment) and click Next

Image may be NSFW.
Clik here to view.

6. In “Action type for unconfigured…” choose Log Stack Trace. In “Action Limit”, write 100. In “Maximum number of userdumps…” choose 1 as below (do not click Next yet):

Image may be NSFW.
Clik here to view.

7. Click on the button Breakpoints. Click on Add Breakpoint…

Image may be NSFW.
Clik here to view.

8. Type mscorlib.dll!System.AppDomain.Unload on Breakpoint Expression. Make sure This is a managed (.NET) breakpoint is selected.. Action Type is to be set to “Full Userdump” and action limit to be set to 1 and click Ok

Image may be NSFW.
Clik here to view.

9. If things are correct you should see the screen below. Then choose “Save & Close”:

Image may be NSFW.
Clik here to view.

10. Click Next

11. By default, the dump will be written in C:\Program Files\DebugDiag\... If you want to save in a different place, change the appropriate filed (it will create the folder if it does not exist:

Image may be NSFW.
Clik here to view.

11. Click Finish (and Yes for any window that pops up):

12. Choose not to activate the rule at this time

Image may be NSFW.
Clik here to view.

13. (ONLY IF .NET 4.6 is installed) Open Notepad as Administrator

14. (ONLY IF .NET 4.6 is installed) Open File C:\Program Files\DebugDiag\Scripts\CrashRule_WebAppPool_DefaultAppPool.vbs

15. (ONLY IF .NET 4.6 is installed) Locate Function AddBreakpoint and replace until EndFunction by this code (in fact there is only one line added to load sos).

Function AddBreakpoint(BreakpointExpression, IsManaged)

Dim LogString

Dim bpID  

    LogString = "Attempting to set "

IfNot IsManaged Then LogString = LogString & "un"

    LogString = LogString & "managed breakpoint at "& BreakpointExpression                 

    WriteToLog LogString           

' This is the ONLY line added

    WriteToLog Debugger.Execute(".loadby sos clr")

' End of modification

    bpID = Debugger.AddCodeBreakpoint(BreakpointExpression, IsManaged)              

If IsManaged Then

If bpID < -1 Then

            RecreateDeferredManagedBreakpointExpressionsCollectionIfNecessary

If IsEmptyOrNothing(g_DeferredManagedBreakpointExpressions) ThenExitFunction

If g_DeferredManagedBreakpointExpressions.Exists(BreakpointExpression) Then

                g_DeferredManagedBreakpointExpressions(BreakpointExpression) = bpID

Else

                g_DeferredManagedBreakpointExpressions.Add BreakpointExpression, bpID

EndIf

EndIf

EndIf

               DbgState("BP_"& BreakpointExpression & "_ID") = bpID

    WriteToLog "bpID = "& bpID

               AddBreakpoint = bpID

EndFunction

16. (ONLY IF .NET 4.6 is installed) Save the file

17. Right-click on the rule and choose activate

Image may be NSFW.
Clik here to view.

Other Approach

I talked to my dear DebugDiag colleague Wade Mascia and he suggested a cleaner approach: to add a custom rule for clr.dll module load. The rule would be simply load sos. This option also makes the export and import of the rule possible. Anyway, this issue should be resolved by next DebugDiag release.

Image may be NSFW.
Clik here to view.