While we wait for Magnus Karlsen to make his first move (E4?), here’s a solution walk-through of the second Packet Capture Analysis challenge.

Spoilers ahead

Here we go

#1 – How many hops can we assume there are between the client and the server?

If we assume a default minimum IP datagram TTL of 64 (WHAT? See https://tools.ietf.org/html/rfc1700#page-64), the server 204.152.184.134 is 14 hops away because the IP datagram TTL is 50 in all TCP segments received by the client.

hops = def.ttl – rcv.ttl

How to find the IP TTL value:

#2 – Using fingerprinting techniques, what OS is the server likely running? (1 point per technique)

One way is to look at the HTTP 200 OK response from the server. Usually the application give away some hints on OS and software used:

It claims to be a FreeBSD host.

Another indicator of OS type/version can be found by looking at the advertised IP TTL and TCP Window Size values:

A combination of IP TTL 64 and Window Size 65535 is often found in *nix/FreeBSD systems. More information here and here.

But wait, the Windows client is using the exact same values!? Yes, implementation “standards” are nice

#3 – What is the average RTT delay between the client and the server? (1 point)

Start off by changing View > Time Display Format to Time Since Previous Displayed Packet.

Then we can look at the SYN/ACK response during a TCP handshake to determine the RTT, around 20 ms in this case.

Why the SYN/ACK handshake step? Because that segment is generated by the remote end TCP/IP stack, without any added application delay.

#4 – Following frame #14886, what TCP sequence number (relative) is the client expecting to receive next? (1 point + 1 bonus)

Looking at frame #14886 we see a TCP sequence number of 4641056. That segment had a payload of 1460 Bytes. The next sequence number in the stream is 4641056 + 1460 = 4642516.

As we see in frame #14889 the client does not receive the expected segment and TCP SACK kicks in.

Bonus – In what frame does it receive the expected TCP segment?

Since we know that the sequence number is 4642516, we can apply a specific Display Filter using tcp.seq:

Frame #14981!

5# – At the beginning of the file transfer there is a delay lasting around 3 seconds. Why? (2 points)

A graphical representation of the transfer halting
Statistics > TCP Stream Graphs > Time Sequence (Stevens):

Awkward silence.

The client TCP receive buffer is considered full in frame #8662. The client is reporting a window size of 536 Bytes which does not allow for a full TCP segment of 1460 Bytes. For some reason the client application (browser) is not pulling data fast enough from its buffer.

The transfer halts for around 3 seconds until the client has finished processing the data and reports a window of 64 KB (256KB as seen in Wireshark is due to Window Scaling). The server continues transferring at frame #8665.

That’s it!

Hope you had some fun solving these. More to come

Have a nice day! 

The Sharkminator returns.

Round two of #Wiresharklympics is here! Having survived round one, you know the drill. The tiny specs of data that allow services like Facebook and the Internet to work will be put under heavy scrutiny.

Wireshark has reached a stable release of 2.2.1 and is eagerly awaiting new challenges.

We will yet again use a sample packet capture from Netresecs list.

The packet capture sample can be downloaded here:
https://dl.dropboxusercontent.com/u/1185688/blog/wireshark2.pcapng

It’s safe to download. VirusTotal concur.

Enough yada yada

Five questions. Should be at least 8 points up for grabs

Start off by applying this display filter “tcp.stream == 16”.

1. How many hops can we assume there are between the client and the server? (1 point)
2. Using fingerprinting techniques, what OS is the server likely running? (1 point per technique)
3. What is the average RTT delay between the client and the server? (1 point)
4. Following frame #14886, what TCP sequence number (relative) is the client expecting to receive next? (1 point + 1 bonus)
   Bonus – In what frame does it receive the expected TCP segment?
5. At the beginning of the file transfer there is a delay lasting around 3 seconds. Why? (2 points)

Easy peasy?

Please send me your questions and answers via a communication transport of your liking. A comment here, the social medias or email. Doesn’t matter!

The winner will of course receive loads of street cred. Way better than Facebook likes.

Have a great day

It’s time for a walk-through of how to solve the first Packet Analysis Challenge.

Happy to see that there are plenty of fine Wireshark warriors out there! Hope you had some fun 

The fastest one to answer all the questions including the bonus round was a local champion named Martin Karlsen! Lots of packet level street cred is hereby sent your way. He also has a website up at exceededintransit.net that you can check out. Don’t expect to find cooking recipes there. Next-gen Jedi that one.

Yes, OK, just give me the answers

#1 – How many non-broadcast IPv4 nodes is Wireshark seeing?

Go to Statistics > Endpoints to list all the endpoints that Wireshark is seeing across all TCP/IP layers. Select IPv4:

With 255.255.255.255 being a Limited Broadcast address (rfc5735#section-4), this leaves us with a total of 11 IPv4 nodes.

#2 – The client downloads an EXE file, twice. From which countries is it downloading the file from?

Apply a display filter to only focus on HTTP requests containing the word “exe”:

Frame number 842 and 8292 initiates the downloads and list two different destination IPv4 addresses. Run a whois query against them to find out where they originate:

$ whois 61.8.0.17 | grep Country
Country: AU
$ whois 204.152.184.134 | grep Country
Country: US

There you have it, Australia and the United States.

#3 – How many Bytes is the client expecting to download for each EXE file?

We can find this information by looking at the HTTP 200 OK response from the server (frames 844 and 8294):

The HTTP header Content-Length indicates the size of the entity-body in Bytes (rfc2616#section-14.13).

The client therefore expect to download 78597807 Bytes (78.5 MB).

#4 – Looking at the fastest of the two transfers, at what speed is the file downloaded on average in kbps, kilobit per second?

Go to Statistics > Conversations and select TCP (since the two transfers used HTTP, which is transported over TCP).

Click on the column Bytes to sort the table. The top two conversations list the file transfers.

Next, find the column Bits/s B -> A (meaning the download direction from the client perspective).

The server providing the fastest file transfer is the US host 204.152.184.134. It delivers the EXE file at an average speed of 2344 Kbps or 2.34 Mbps.

#5 – One node is not accepting the use of full TCP segments. Which one?

A 100% fully utilized TCP segment carry 1460 Bytes. You can load the HTML payload of http://ulv.no/ almost three times in such a segment. Plenty of data.

If there’s a need for a limit, it is set during the initial TCP handshake in a separate TCP header option:

https://tools.ietf.org/html/rfc793#page-19

Maximum Segment Size Option:
If this option is present, then it communicates the maximum receive segment size at the TCP which sends this segment. This field must only be sent in the initial connection request (i.e., in segments with the SYN control bit set). If this option is not used, any segment size is allowed.

Meaning, if Node A set its MSS to 1100 and Node B set its MSS to 1460 in their handshake, any segment sent to Node A must be limited to 1100 Bytes. Node B however is happy to receive up to 1460 Bytes of payload.

So which one is it?

First, apply a display filter to single out all TCP handshakes. All handshakes must have the SYN flag set.

Next select a frame, then in the Packet Details pane, expand the subtree of Transmission Control Protocol. Scroll down to Options and look for Maximum segment size.

Expand it and right-click on MSS Value and select Apply as Column:

This allow us easily see all the TCP MSS values set during TCP handshakes.

By sorting using this new shiny MSS Value column, we find one outlier:

The host 216.251.114.10 in frames 6 and 30 limit any TCP segments sent its way to 1380 Bytes.

Optional solution:
Apply a display filter to list all MSS values below 1460 Bytes using “tcp.options.mss_val < 1460“.

BONUS – How many Bytes is the client potentially missing out on per round-trip?

If a full TCP segment is 1460 Bytes, then we’re missing out on 80 Bytes of data in each TCP segment sent towards 216.251.114.10.

But wait a minute, you said client! Hahaa!

Yes I did. And in retrospect it was a weird example to use since 216.251.114.10 is a web server and not a client. Or is it..? This could turn into a nice client-server model debate.

But, because of this ambiguity – my bad. Or as the Urban Dictionary would put it:

“I did something bad, and I recognize that I did something bad, but there is nothing that can be done for it now, and there is technically no reason to apologize for that error, so let’s just assume that I won’t do it again, get over it, and move on with our lives.”

Have a great day!

The Sharkminator

Vacation’s over. Your networks have been underutilized for a good long month now. Time to get back to the trenches. Why not start things off with a proper packet analysis challenge? At least fire up Wireshark to see if there’s an auto-update waiting for you?

Thank you Netresec for providing a huge list of packet captures to play with!

We will borrow a 13 MB packet capture from the excellent book “Practical Packet Analysis“.

$ shasum wireshark1.pcapng
b8060f2b946f33b79833710db458368cd382d06c wireshark1.pcapng

Please go ahead and download the pcap file. Yes, it’s safe to download.

Ready?

<gong sound>

Five questions + one bonus. One point per question:

1. How many non-broadcast IPv4 nodes is Wireshark seeing?
2. The client downloads an EXE file, twice. From which countries is it downloading the file from?
3. How many Bytes is the client expecting to download for each EXE file?
4. Looking at the fastest of the two transfers, at what speed is the file downloaded on average in kbps, kilobit per second?
5. One node is not accepting the use of full TCP segments. Which one?
   1. BONUS – How many Bytes is the client potentially missing out on per round-trip?

Easy peasy?

Please send me your answers via a communication platform of your liking. The social medias or email. Doesn’t matter!

The winner will get loads of street cred as defined by Urban Dictionary:

He’s been thru it all. His street cred is undeniable.

That’s all you need. Get to it!

Today, the phone is a touch screen that allow you to interact with apps. Everything is done through apps. We use them to buy tickets, share pictures with grandparents, access our bank account, post a picture of today’s dinner. So what happens when your favorite app stops working all of a sudden? How do you troubleshoot it?

Once I had an issue with a banking app on my iPhone. After upgrading to iOS 9, logging in using two-factor authentication stopped working. It had persisted for several months and when I asked the company I got the default reply that “it’s a known issue and the developers are working a fix”. Ok great.

This is what greeted me when I tried logging in:

You could sit and look at that white square for as long as you wanted. Nothing would happen. No error message or spinning beach ball.

Cursed app

So then I decided to troubleshoot it myself. The same way I troubleshoot applications professionally. Not the developer way of debugging through Xcode. The network engineer way by looking at packets. Shouldn’t be too hard, right?

Two hurdles:

1. The app was on my iPhone which meant wireless communication.
2. The communication was likely using an encrypted channel making eavesdropping impossible*.

* No, nothing is impossible. You can use mitmproxy or similar tools.

Using a Mikrotik RB750GL I mirrored all traffic sent and received from my iPhone to my laptop. Got Wireshark up and running hoping to find something that indicated an error.

Filtering the output down to the exact moment when I tapped “Login” and the two-factor module was supposed to load:

Lots of weird text. Don’t give up just yet.

There’s a clue here! On line #6, Wireshark is trying to help us by showing an error message “Handshake Failure”. A fatal alert. That can’t be good.

Following the TLS handshake failure the server tears down the TCP connection. Nothing happens in the app UI. So much for error handling.

More on Transport Layer Security (TLS) Protocol Version 1.2 Handshakes https://tools.ietf.org/html/rfc5246#section-7.3.

Failing handshakes

What does this mean? It means that the client is sending something that the server does not agree with.

Then I compared the authentication and handshake mechanism with another banking app. One that worked. It turned out that the two were sending two different cipher suites in their Client Hello:

Failing app (12 ciphers)

TLS_EMPTY_RENEGOTIATION_INFO_SCSV
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA

Working app (same 12 ciphers + 6 additional)

TLS_EMPTY_RENEGOTIATION_INFO_SCSV
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
TLS_RSA_WITH_AES_256_GCM_SHA384
TLS_RSA_WITH_AES_128_GCM_SHA256
TLS_RSA_WITH_AES_256_CBC_SHA256
TLS_RSA_WITH_AES_256_CBC_SHA
TLS_RSA_WITH_AES_128_CBC_SHA256
TLS_RSA_WITH_AES_128_CBC_SHA

The cipher chosen for the working app was “TLS_RSA_WITH_AES_256_CBC_SHA”.

After some research I learned that Apple introduced a new security feature in iOS 9 called App Transport Security (ATS). Even though it killed my app it’s a great feature forcing the use of Transport Layer Security (TLS) version 1.2, better ciphers, forward secrecy (FS) among other things.

So the theory now was:

“after upgrading to iOS 9 the client is sending ciphers unsupported by the server”

Every theory should be tested so I decided to use the excellent Qualys SSL Labs SSL Report tool.

Lo and behold, both server endpoints got a grade of C because of two things:

1. “No support for TLS 1.2, which is the only secure protocol version.”
2. “Handshake Simulation: Apple ATS 9 / iOS 9 – Protocol or cipher suite mismatch – Fail”

I got the same results using nscurl showing that TLS 1.0 was the only supported version:

$ /usr/bin/nscurl --ats-diagnostics --verbose https://login.bank.com/"

TLSv1.0 with PFS disabled
ATS Dictionary:
{
    NSExceptionDomains =     {
        "login.bank.com" =         {
            NSExceptionMinimumTLSVersion = "TLSv1.0";
            NSExceptionRequiresForwardSecrecy = false;
        };
    };
}
Result : PASS

Show me the fix

The problem had two possible solutions:

• Enable support of TLS 1.2 and Apple ATS 9 handshakes on the server side – GOOD
• Offer ciphers outside of the Apple ATS 9 scope on the client side – BAD

Based on my findings the app was updated to version 4.1.0 “authorisation issue on iOS 9 for Norwegian region”.

Checked again today to see what had changed:

• The client is now sending 22 ciphers instead of the ATS 9 default of 12.
• The server side now get a rating of A and B by Qualys. TLS 1.2 is now supported. Apple ATS 9 is still unsupported.

The world is not perfect, but we’re getting there. Bit by bit 

If you are looking for a troubleshooting expert, have a look at my Consulting Services.

Have a nice day!