As requested you can now find information about my Consulting Services.

Get in touch if you’re interested in discussing business opportunities.

Have a nice day!

Today, the phone is a touch screen that allow you to interact with apps. Everything is done through apps. We use them to buy tickets, share pictures with grandparents, access our bank account, post a picture of today’s dinner. So what happens when your favorite app stops working all of a sudden? How do you troubleshoot it?

Once I had an issue with a banking app on my iPhone. After upgrading to iOS 9, logging in using two-factor authentication stopped working. It had persisted for several months and when I asked the company I got the default reply that “it’s a known issue and the developers are working a fix”. Ok great.

This is what greeted me when I tried logging in:

You could sit and look at that white square for as long as you wanted. Nothing would happen. No error message or spinning beach ball.

Cursed app

So then I decided to troubleshoot it myself. The same way I troubleshoot applications professionally. Not the developer way of debugging through Xcode. The network engineer way by looking at packets. Shouldn’t be too hard, right?

Two hurdles:

1. The app was on my iPhone which meant wireless communication.
2. The communication was likely using an encrypted channel making eavesdropping impossible*.

* No, nothing is impossible. You can use mitmproxy or similar tools.

Using a Mikrotik RB750GL I mirrored all traffic sent and received from my iPhone to my laptop. Got Wireshark up and running hoping to find something that indicated an error.

Filtering the output down to the exact moment when I tapped “Login” and the two-factor module was supposed to load:

Lots of weird text. Don’t give up just yet.

There’s a clue here! On line #6, Wireshark is trying to help us by showing an error message “Handshake Failure”. A fatal alert. That can’t be good.

Following the TLS handshake failure the server tears down the TCP connection. Nothing happens in the app UI. So much for error handling.

More on Transport Layer Security (TLS) Protocol Version 1.2 Handshakes https://tools.ietf.org/html/rfc5246#section-7.3.

Failing handshakes

What does this mean? It means that the client is sending something that the server does not agree with.

Then I compared the authentication and handshake mechanism with another banking app. One that worked. It turned out that the two were sending two different cipher suites in their Client Hello:

Failing app (12 ciphers)

TLS_EMPTY_RENEGOTIATION_INFO_SCSV
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA

Working app (same 12 ciphers + 6 additional)

TLS_EMPTY_RENEGOTIATION_INFO_SCSV
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
TLS_RSA_WITH_AES_256_GCM_SHA384
TLS_RSA_WITH_AES_128_GCM_SHA256
TLS_RSA_WITH_AES_256_CBC_SHA256
TLS_RSA_WITH_AES_256_CBC_SHA
TLS_RSA_WITH_AES_128_CBC_SHA256
TLS_RSA_WITH_AES_128_CBC_SHA

The cipher chosen for the working app was “TLS_RSA_WITH_AES_256_CBC_SHA”.

After some research I learned that Apple introduced a new security feature in iOS 9 called App Transport Security (ATS). Even though it killed my app it’s a great feature forcing the use of Transport Layer Security (TLS) version 1.2, better ciphers, forward secrecy (FS) among other things.

So the theory now was:

“after upgrading to iOS 9 the client is sending ciphers unsupported by the server”

Every theory should be tested so I decided to use the excellent Qualys SSL Labs SSL Report tool.

Lo and behold, both server endpoints got a grade of C because of two things:

1. “No support for TLS 1.2, which is the only secure protocol version.”
2. “Handshake Simulation: Apple ATS 9 / iOS 9 – Protocol or cipher suite mismatch – Fail”

I got the same results using nscurl showing that TLS 1.0 was the only supported version:

$ /usr/bin/nscurl --ats-diagnostics --verbose https://login.bank.com/"

TLSv1.0 with PFS disabled
ATS Dictionary:
{
    NSExceptionDomains =     {
        "login.bank.com" =         {
            NSExceptionMinimumTLSVersion = "TLSv1.0";
            NSExceptionRequiresForwardSecrecy = false;
        };
    };
}
Result : PASS

Show me the fix

The problem had two possible solutions:

• Enable support of TLS 1.2 and Apple ATS 9 handshakes on the server side – GOOD
• Offer ciphers outside of the Apple ATS 9 scope on the client side – BAD

Based on my findings the app was updated to version 4.1.0 “authorisation issue on iOS 9 for Norwegian region”.

Checked again today to see what had changed:

• The client is now sending 22 ciphers instead of the ATS 9 default of 12.
• The server side now get a rating of A and B by Qualys. TLS 1.2 is now supported. Apple ATS 9 is still unsupported.

The world is not perfect, but we’re getting there. Bit by bit 

If you are looking for a troubleshooting expert, have a look at my Consulting Services.

Have a nice day!

That’ll slow things down! #NetworkHumor pic.twitter.com/WpbTyynTaD

— Juniper Networks (@JuniperNetworks) April 10, 2016

A young Michael Caine enjoying Juniper labs.

Let’s say you have a Juniper EX switch that you want to connect to your new virtual lab. Maybe you’re training for the JNCIP-ENT. Maybe you need to verify reachability to a production network over an IPsec VPN tunnel. How do you connect your virtual Juniper lab to the world?

Start by listing all the available local interfaces (requires a VirtualBox setup):

$ VBoxManage list bridgedifs | grep ^Name
Name: en1: Wi-Fi (AirPort)
Name: en0: Ethernet
Name: en2: Thunderbolt 1
Name: p2p0
Name: bridge0

Select an interface from that list, then modify your Vagrantfile to connect vsrx1 to the outside world:

  config.vm.define "vsrx1" do |vsrx1|
    vsrx1.vm.host_name = "vsrx1"
    vsrx1.vm.network "private_network",
                     ip: "10.99.12.1",
                     virtualbox__intnet: "1-2"
    vsrx1.vm.network "private_network",
                     ip: "10.99.31.1",
                     virtualbox__intnet: "1-3"
    vsrx1.vm.network "public_network",
                     bridge: "en1: Wi-Fi (AirPort)"
  end

We have now bridged a physical interface, in this case my Macbook Wi-Fi interface, to vsrx1’s interface ge-0/0/3.0:

$ vagrant ssh vsrx1
--- JUNOS 12.1X47-D15.4 built 2014-11-12 02:13:59 UTC
root@vsrx1% cli
root@vsrx1> show configuration interfaces ge-0/0/3
unit 0 {
    family inet {
        dhcp;
    }
}
root@vsrx1> show interfaces terse ge-0/0/3.0
Interface               Admin Link Proto    Local                 Remote
ge-0/0/3.0              up    up   inet     10.24.5.207/24
root@vsrx01> ping 8.8.8.8 count 3
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=44 time=35.096 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=44 time=23.366 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=44 time=36.630 ms

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 23.366/31.697/36.630/5.924 ms
root@vsrx01>

Your lab can now reach The Internet through vsrx1 ⚡️

Get in touch if you are looking for automation and Juniper consulting services.