Capture network packets with netsh

Another day, another “interesting” issue at a customer. After deploying our product we were left with a partially working web application. The product has been developed over many years and is a mix of ASP Classic, Web Forms, MVC and Web API. In this case ASP Classic pages were broken and would throw an error.

Ensuring ASP Classic is configured properly

The first step is to ensure that IIS has been configured to execute ASP Classic and this is done easily by adding a dummy ASP page to the web application. After deploying this page I was able to confirm that it was working as expected.

Enabling failed request tracing

The features written in ASP Classic have been written many years ago and the developers didn’t consider logging as a key part of the development process. The end result being that when something goes wrong no logs get written by the application or to the event viewer.

The second step is to turn on the “Failed Request Tracing Rules“ and reload the failing page. Internet has a lot of tutorials around this but they’re all missing key steps, I’ll focus on those as you can find everything else easily.

“Failed Request Tracing Rules” will not be available in the IIS Manager if you didn’t turn on the Tracing feature in Windows Features:

Another thing is that multiple sites could be writing traces at the same time. Each site will be writing to a different sub folder suffixed with the site ID:

Finally you can copy the log files back to your machine, don’t forget to copy the freb.xsl file too, you’ll then be able to open the XML files in Internet Explorer and look at a human readable representation of the log.

All ASP Classic pages are calling an API endpoint in order to get a token (long story short: don’t ask - the user is signing-in in an AngularJS app backed by Web API and is then able to use seamlessly the pages hosted on MVC, Web Forms and Classic ASP). What is strange in this situation is that Internet Explorer is marking the TLS certificate as valid, so does Chrome. Even worse: the same ASP Classic page hosted on my machine calling the token endpoint on the remote server is successful! The Windows Certificate Manager is displaying the same message for the root CA, intermediate CA and certificate: ”This certificate is OK.”.

I then suspected the certificate might be using unsupported ciphers but it turned out that it wasn’t the case. I quickly wrote a C# Console application calling the same token endpoint - the HttpClient class is throwing meaningful errors - but to my dismay the C# code was able to call the endpoint successfully!

Armed with the ErrorCode “80072f0d” and the Description “The certificate authority is invalid or incorrect” I scoured Internet for some potential solutions. Everything I could find was related to invalid and self-signed certificates.

Capturing packets on a Windows Server

When people think “packet capture” they always assume they need to install Wireshark (or another similar tool) whereas Windows Server is shipping with the ability to capture network packets with netsh since Windows Server 2008 R2. The advantage of this solution is that you don’t need to install anything on the machine. To see if it’s available, all you need to do is open a command prompt and type:

netsh trace

Now that we know that trace is available we need to start capturing the packets and reproduce the problem. Launch an elevated command prompt and type:

netsh trace start tracefile="C:\tmp\traces\classic.etl" scenario=internetclient capture=yes maxsize=200 filemode=circular overwrite=yes

Note: the path needs to exist beforehand.

Starting and stopping the trace is actually slower that what is demonstrated above but I didn’t want lo lose your attention! And of course you would need to reproduce the issue before issuing:

netsh stop

Microsoft Message Analyzer

Note: Microsoft Message Analyzer has been retired since I wrote this post.

We now need to analyze this trace and this is done with the Microsoft Message Analyzer (can be downloaded here). The Analyzer takes a long time to open the smallest trace but once the trace is loaded you can search quickly.

We’ll first look for an HTTP CONNECT, use this filter:

(HTTP.Method == "CONNECT") And
(HTTP.Uri.Host == "domain.name") And
(HTTP.Uri.Port == "port")

As we can see the CONNECT was successful. Let’s investigate the TLS handshake now, this is handled by the TLS module so all we need to do is filter on this module only:

TLS

This is what was captured when the C# application connected to the token endpoint:

This is matching closely what is described in the RFC 5246 (TLS 1.2).

Let’s now capture the traffic when the VB code is trying to call the token endpoint.

Great Scott! The server is sending ServerHello as expected but the client doesn’t reply with ClientKeyExchange. I then removed the filter and started to look at the messages below. My reasoning was that I should be finding some kind of error message soon after and here it was:

The error message was:

A certificate chain processed, but terminated in a root certificate which is not trusted by the trust provider. (0x800B0109)

As it turned out someone had messed up with the certificate store and removed the intermediate CA from the “Intermediate Certification Authorities”. As the root CA was still present in the “Trusted Root Certification Authorities” it was good enough for Internet Explorer and C# but it wasn’t for VB! I added the intermediate CA to the store and things started to work again.