This new series is an attempt to improve my WinDbg skills. The concept is to create faulty applications and troubleshoot the issue using WinDbg pretending that I have no prior knowledge of the code.
I’ll be using my WinDbg guide as I can never remember the commands! I’m hoping than through those challenges I’ll get to improve the guide. Today’s exercise is inspired by the excellent blog post Pinpointing a Static GC Root with SOS. The post only contains a few commands but I must admit that it took me hours to achieve the same result.
The application is available on GitHub. Clone it, I’ll wait.
It is an ASP.NET Core 2.0 project:
Compile the solution with the ReleaseConfiguration
Launch the SampleApi project, it should start on port 5000
Using Kestrel will make the next part easier
Launch Process Explorer. If you don’t have this gem drop everything you’re doing and download it now! Click on the crosshair, mouse hover the process you want to target and release the button:
The Working Set is sitting just under 44 MB.
Issue 20GET requests to http://localhost:5000/feed/me
We’ll start with the DumpHeapcommand from the SOS extension.
Displays information about the garbage-collected heap […]. The -stat option restricts the output to the statistical type summary.
Instead of listing every single object present in the heap(s), this will group them by Class Name and provide us with an instance Count and TotalSize taken (in bytes). Let’s run it:
It looks like we have a winner! There are 158 instances of System.Int32 for a TotalSize of 251802384 bytes. As we have only 158 instances it’s likely we have a few big instances, let’s list the ones that are bigger than 1000 bytes:
As it turns out one instance is 134217752 bytes which is roughly 134 MB. I suggest we start investigating this one.
Displays information about references (or roots) to an object at the specified address.
This reads bottom to top, our Int32 is referenced by a List<Int32>. This makes sense as List<T> is using an array internally:
The List<T>class is the generic equivalent of the ArrayListclass. It implements the IList<T> generic interface by using an array whose size is dynamically increased as required.
In turn this List<Int32> is referenced by a System.Object. I was hoping to get the name of one of my class but I’ll have to dig deeper, let’s take a closer look at this array of object.
For this we’ll rely on the DumpObjcommand from the SOS extension.
According to Sasha Goldshtein post this is how the CLR stores static fields:
This objectarray is ubiquitous, it would seem that all static root references stem from it. Indeed (and this is a CLR implementation detail), static fields are stored in this array and their retention as far as the GC is concerned is through it.
Let’s now determine where in the array is our List referenced. We’ll use the Search Memorycommand which is the first WinDbg command we used today!
-q: we’re looking for a QWORD (the address is 64 bit)
L: this is a Range, we’re starting to search at the address 0000019419341038 (the beginning of the array) and we search the whole array (1ff8 is the size of the array as indicated in the previous command output)
0000019019412bf0 is the address of the List
Sadly the lead stops there. We know this is a static field but we don’t know which class it belongs to.
Fishing with dynamite
There is one last thing we can try, we could look for references to 0000019419342830 in memory. This section is completely stolen from Sasha’s excellent post as I never did something like this before.
Enumerates each Assembly object that is loaded within the specified AppDomain object address.
SampleApi.dll is located at 00007ffac5d04d38 so it does make sense to start searching at 00007ffa00000000. Remember the Search Memory command we used above? We’ll put it to good use again:
L: this is a Range, we’re starting to search at the address 00007ffa00000000
As we’re searching for a QWORD the unit is 8 bytes (64 bit), so we’re looking ahead for 40000000 * 8 = 320 MB
Bingo! Wow I didn’t think it would be that easy. We have a reference! Let’s use the WinDbgUnassemblecommand to look at the instructions:
Looks like I might have celebrated prematurely. Let’s extend the range:
Again, this is where Sasha comes to the rescue:
The problem is that we might miss unaligned references to that address, which may occur if it is hardcoded into some instruction (e.g. a MOV). So instead we should be looking for the individual byte sequence, and remember that we are on a little-endian architecture
The command is the same than the previous one except for two differences:
This time we’re searching for bytes -b
As we’re on a little-endian architecture, 0000019419342830 turn into 30 28 34 19 94 01 00 00
I’ve already unassembled the first address, let’s look at the two other ones:
That’s much nicer, there is a reference to one of my class: SampleApi.Controllers.FeedController. What about the other address:
This goes a step farther as it references the static constructor of FeedController (SampleApi.Controllers.FeedController..cctor()). We now have enough information to inspect the code but first let’s take a deeper look at the FeedControllerclass.
Displays the MethodTable structure and EEClass structure for the specified type or method in the specified module. […] This command supports the Windows debugger syntax of <module>!<type>. The type must be fully qualified.