This new series is an attempt to improve my WinDbg skills. The concept is to create faulty applications and troubleshoot the issue using WinDbg pretending that I have no prior knowledge of the code.

I’ll be using my WinDbg guide as I can never remember the commands! I’m hoping than through those challenges I’ll get to improve the guide. Today’s exercise is inspired by the excellent blog post Pinpointing a Static GC Root with SOS. The post only contains a few commands but I must admit that it took me hours to achieve the same result.

## The code

The application is available on GitHub. Clone it, I’ll wait.

It is an ASP.NET Core 2.0 project:

• Compile the solution with the Release Configuration
• Launch the SampleApi project, it should start on port 5000
• Using Kestrel will make the next part easier

Launch Process Explorer. If you don’t have this gem drop everything you’re doing and download it now! Click on the crosshair, mouse hover the process you want to target and release the button:

The Working Set is sitting just under 44 MB.

• Issue 20 GET requests to http://localhost:5000/feed/me

If you’re as lazy as me you can leverage Windows Subsystem for Linux:

The Working Set is now sitting just under 262 MB. That’s an increase of roughly 10 MB per request.

## Capture a full memory dump

The easiest option in this case is to leverage Process Explorer as we already have it opened. Right-click on dotnet.exe and select Create Full Dump...:

## Install and configure WinDbg

Follow these instructions:

• Copy the path of the directory where SampleApi.dll is located (in my case it is E:/code/me/blog-samples/windbg-static/src/SampleApi/bin/Release/netcoreapp2.0/)

• Copy the content of this directory into your symbols directory (in my case I configured sympath to include C:\symbols\local\):

Now we’re in business!

## Identifying the most problematic Type / instance

We’ll start with the DumpHeap command from the SOS extension.

Displays information about the garbage-collected heap […]. The -stat option restricts the output to the statistical type summary.

Instead of listing every single object present in the heap(s), this will group them by Class Name and provide us with an instance Count and TotalSize taken (in bytes). Let’s run it:

It looks like we have a winner! There are 158 instances of System.Int32[] for a TotalSize of 251802384 bytes. As we have only 158 instances it’s likely we have a few big instances, let’s list the ones that are bigger than 1000 bytes:

As it turns out one instance is 134217752 bytes which is roughly 134 MB. I suggest we start investigating this one.

## Determine what is keeping our instance alive

We’ll use another SOS command: GCRoot.

Displays information about references (or roots) to an object at the specified address.

This reads bottom to top, our Int32[] is referenced by a List<Int32>. This makes sense as List<T> is using an array internally:

The List<T> class is the generic equivalent of the ArrayList class. It implements the IList<T> generic interface by using an array whose size is dynamically increased as required.

In turn this List<Int32> is referenced by a System.Object[]. I was hoping to get the name of one of my class but I’ll have to dig deeper, let’s take a closer look at this array of object.

For this we’ll rely on the DumpObj command from the SOS extension.

According to Sasha Goldshtein post this is how the CLR stores static fields:

This object array is ubiquitous, it would seem that all static root references stem from it. Indeed (and this is a CLR implementation detail), static fields are stored in this array and their retention as far as the GC is concerned is through it.

Let’s now determine where in the array is our List referenced. We’ll use the Search Memory command which is the first WinDbg command we used today!

• -q: we’re looking for a QWORD (the address is 64 bit)
• L: this is a Range, we’re starting to search at the address 0000019419341038 (the beginning of the array) and we search the whole array (1ff8 is the size of the array as indicated in the previous command output)
• 0000019019412bf0 is the address of the List

Sadly the lead stops there. We know this is a static field but we don’t know which class it belongs to.

## Fishing with dynamite

There is one last thing we can try, we could look for references to 0000019419342830 in memory. This section is completely stolen from Sasha’s excellent post as I never did something like this before.

The SOS command DumpDomain comes in handy:

Enumerates each Assembly object that is loaded within the specified AppDomain object address.

SampleApi.dll is located at 00007ffac5d04d38 so it does make sense to start searching at 00007ffa00000000. Remember the Search Memory command we used above? We’ll put it to good use again:

• L: this is a Range, we’re starting to search at the address 00007ffa00000000
• As we’re searching for a QWORD the unit is 8 bytes (64 bit), so we’re looking ahead for 40000000 * 8 = 320 MB

Bingo! Wow I didn’t think it would be that easy. We have a reference! Let’s use the WinDbg Unassemble command to look at the instructions:

Looks like I might have celebrated prematurely. Let’s extend the range:

Same result!

Again, this is where Sasha comes to the rescue:

The problem is that we might miss unaligned references to that address, which may occur if it is hardcoded into some instruction (e.g. a MOV). So instead we should be looking for the individual byte sequence, and remember that we are on a little-endian architecture

The command is the same than the previous one except for two differences:

• This time we’re searching for bytes -b
• As we’re on a little-endian architecture, 0000019419342830 turn into 30 28 34 19 94 01 00 00

I’ve already unassembled the first address, let’s look at the two other ones:

That’s much nicer, there is a reference to one of my class: SampleApi.Controllers.FeedController. What about the other address:

This goes a step farther as it references the static constructor of FeedController (SampleApi.Controllers.FeedController..cctor()). We now have enough information to inspect the code but first let’s take a deeper look at the FeedController class.

Let’s use the SOS Name2EE command.

Displays the MethodTable structure and EEClass structure for the specified type or method in the specified module. […] This command supports the Windows debugger syntax of <module>!<type>. The type must be fully qualified.

We’ll use the SOS DumpClass command.

Displays information about the EEClass structure associated with a type.

So, it turns out the FeedController has a static field named MemoryHog. Probably not my finest piece of coding to be honest.

## Conclusion

I learned how to trace back a static field to a class. I’m sure this will come in handy later.

I might have made some mistakes around Ranges as this is an area I’m still unfamiliar with but it shouldn’t prevent you from achieving the same result.

Until next time!