On September 1st the Norwegian Juniper Elite partner nLogic AS hosted an event called “Ansible i praksis“, entirely focused on network automation using Ansible. Lots of interesting presentations and discussions from the Norwegian automation scene. A whole day of Juniper, automation and Ansible! Does it get any better?

I had the pleasure of leading a technical workshop at the end of the event where the attendees were challenged with common operations tasks worthy of automating.

A fun experience and all-in-all a great day!

Presentations

Juniper Networks had the first presentation talking about their current automation portfolio and how they are embracing Ansible. Leading by example Juniper publishes fully working playbooks on Github – github.com/JNPRAutomate.

Next up was the Norwegian Meteorological Institute (MET) talking about their Ansible implementation and showcasing everything in a live demo how they modify their Leaf-and-Spine DC fabric including firewall rulesets, on the fly, of course.

Then Uninett showed how they are planning to roll out their new core network using Ansible and how they are saving loads of time by automating the initial preparation of routers, before shipping them out to their educational and research institution partners.

Workshop

The last part of the event was the two-hour workshop. The attendees got access to a nine-node Juniper QFX topology which they were challenged to interact with using only Ansible:

With the limited amount of time our main focus was:

• Configure the network infrastructure using abstraction and templates.
• Perform an action on a device, then send that information to an external web service.
• Export information from your infrastructure for inventory or compliance purposes.

Often it’s the small and simple tasks that yields the greatest automation value.

If you want to try some similar scenarios, Juniper have published great examples on Github – github.com/JNPRAutomate/ansible-junos-examples . You can spin up a two-node topology using Vagrant in minutes and start testing.

So, what will YOU automate this week?

Verifying NTP settings in your infrastructure? The planned cloud deployment? The upcoming security compliance check? That single configuration change, that needs to be typed in on 100 nodes?

Start small and scale up later. The important key is that you start automating something.

Unsure if your infrastructure is automation friendly? Need help finding some proper automation candidates? Fear not – check out my consulting services.

Have a great day! 

The MikroTik model chosen was the “hAP ac lite” or RB952Ui-5ac2nD which supports 2.4Ghz and 5GHz (802.11 a/b/g/n/ac). Their OS, RouterOS, supports lots of security, wireless and routing features to play with. All this for a price tag of around 50 USD. Sweet!

To solve this challenge we need some way to execute actions on the MikroTik, then extract that output data into some useful format which we then can analyze and store. Then scale this up to n devices and repeat indefinitely.

Since I’m not a hard-core programmer able to hack out a solution in golang during breakfast, I’ll go with using the tools I’m familiar with, namely Ansible and Elastic Stack. Ansible is a tool to perform system/server/network automation using a set of tasks then executing these on a group of hosts. Elastic Stack is a collection of tools allowing you to retrieve, store and visualize data.

Here is a visual representation of how these components would work together:

Splunk was already being used for data storage, so why not have Ansible spread the love and send JSON to both services simultaneously. The more the merrier!

First we need to have all of the nodes at a certain configuration level.

Setup.yml:

 - hosts: mt-setup
   gather_facts: no
   connection: paramiko

   tasks:
     - name: Create the bwtest user allowing access from Ansible
       raw: "/user add name=bwtest group=full password=\"bwpass\" address=11.22.33.44/32"

     - name: Temporary static route for bandwidth-server if needed
       raw: ":if (([:len [/ip route find where comment=bwtest]]) < 1) do={/ip route add dst-address=55.66.77.88/32 gateway=1.1.1.1 comment=bwtest}"

     - name: Wireless 2.4Ghz setup
       raw: "/interface wireless set [ find default-name=wlan1 ] disabled=no distance=indoors frequency=auto ssid=\"2GHz SSID\" wireless-protocol=802.11 bridge-mode=disabled mode=station"

     - name: Wireless 5GHz setup
       raw: "/interface wireless set [ find default-name=wlan2 ] band=5ghz-a/n disabled=no distance=indoors frequency=auto ssid=\"5GHz SSID\" wireless-protocol=802.11 bridge-mode=disabled mode=station"

     - name: Disable 2.4 GHz wlan bridge interface
       raw: "/interface bridge port disable [/interface bridge port find where interface=wlan1]"

     - name: Disable 5GHz wlan bridge interface
       raw: "/interface bridge port disable [/interface bridge port find where interface=wlan2]"

     - name: DHCP client settings 2.4GHz
       raw: ":if (([:len [/ip dhcp-client find where interface=wlan1]]) < 1) do={/ip dhcp-client add add-default-route=no dhcp-options=hostname,clientid disabled=no interface=wlan1}"

     - name: DHCP client settings 5GHz
       raw: ":if (([:len [/ip dhcp-client find where interface=wlan2]]) < 1) do={/ip dhcp-client add add-default-route=no dhcp-options=hostname,clientid disabled=no interface=wlan2}"

We now have all the MikroTiks configured with two active wireless interfaces, DHCP client running on both interfaces and a user for Ansible to continue its work.

Then it’s time to perform the actual performance testing and data collection.

Benchmark.yml:

 - hosts: mt-prod
   gather_facts: no
   connection: paramiko

   vars:
     tikhost: "{{ ansible_host }}"
     tikuser: "bwtest"
     tikpw: "bwpass"
     tikbwhost: "55.66.77.88"
     tikduration: "10"
     tiktx: "100M"
     tikrx: "100M"
     tikdirection: "both"
     colon: ":"

   tasks:
     - name: Disable 5GHz wlan interface
       raw: "/interface wireless set [ find default-name=wlan2 ] disabled=yes"

     - name: Waiting for wireless connection to complete
       pause: seconds=5

     - name: Set static route for 2.4GHz testing
       raw: "ip route set [/ip route find where comment=bwtest] gateway=[/ip dhcp-client get [find interface=wlan1] gateway]"

     - name: Include TCP test instructions
       include: include/bwtest.yml tikproto="tcp" tikfreq="2.4GHz"

     - name: Include UDP test instructions
       include: include/bwtest.yml tikproto="udp" tikfreq="2.4GHz"

     - name: Disable 2.4GHz wlan interface
       raw: "/interface wireless set [ find default-name=wlan1 ] disabled=yes"

     - name: Enable 5GHz wlan interface
       raw: "/interface wireless set [ find default-name=wlan2 ] disabled=no"

     - name: Waiting for wireless connection to complete
       pause: seconds=5

     - name: Set static route for 5GHz testing
       raw: "ip route set [/ip route find where comment=bwtest] gateway=[/ip dhcp-client get [find interface=wlan2] gateway]"

     - name: Include TCP test instructions
       include: include/bwtest.yml tikproto="tcp" tikfreq="5GHz"

     - name: Include UDP test instructions
       include: include/bwtest.yml tikproto="udp" tikfreq="5GHz"

     - name: Enable 2.4GHz wlan interface
       raw: "/interface wireless set [ find default-name=wlan1 ] disabled=no"

We run the bandwidth tests over TCP and UDP using 2.4GHz and 5Ghz. We move the static route around to select our benchmarking interface.

As shown in Benchmark.yml we also include a file called bwtest.yml that executes the actual testing command and formats the output. Including allow us to reuse code instead duplicating it in our playbook.

Bwtest.yml:

- name: Random sleep so we don't DDoS the server
  local_action: shell sleep {{ item }}
  with_random_choice:
    - "0s"
    - "5s"
    - "10s"

- name: Connect to MT and run {{ tikproto }} bandwidth test over {{ tikfreq }}
  local_action: shell ./tikjson.rb {{ tikhost }} {{ tikuser }} {{ tikpw }} /tool/bandwidth-test =address={{ tikbwhost }} =duration={{ tikduration }} =direction={{ tikdirection }} =protocol={{ tikproto }} =local-tx-speed={{ tiktx }} =remote-tx-speed={{ tikrx }} | jq -c '.[][] | .' | sed 'x;$!d'
  register: cmd

- name: Copy JSON output to file
  local_action: copy content="{{ cmd.stdout }}" dest="output/bwtest_{{ tikproto }}_{{ tikhost }}.json"

- name: Rewrite tags in JSON 
  local_action: replace backup=no dest=output/bwtest_{{ tikproto }}_{{ tikhost }}.json regexp='.section' replace='section'
- name: Rewrite tags in JSON 
  local_action: replace backup=no dest=output/bwtest_{{ tikproto }}_{{ tikhost }}.json regexp='.tag' replace='tag'
- name: Rewrite tags in JSON 
  local_action: replace backup=no dest=output/bwtest_{{ tikproto }}_{{ tikhost }}.json regexp='!re' replace='proto'
- name: Rewrite tags in JSON
  local_action: replace backup=no dest=output/bwtest_{{ tikproto }}_{{ tikhost }}.json regexp='null' replace='"{{ tikproto }}"'
- name: Rewrite tags in JSON
  local_action: replace backup=no dest=output/bwtest_{{ tikproto }}_{{ tikhost }}.json regexp='}' replace=', "host"{{ colon }} "{{ tikhost }}", "freq"{{ colon }} "{{ tikfreq }}" }'

- name: Send JSON to log analyzer (Logstash)
  local_action: shell /bin/nc -q1 -w2 logstash.com 23777 < output/bwtest_{{ tikproto }}_{{ tikhost }}.json
  ignore_errors: yes

- name: Send JSON to log analyzer (Splunk)
  local_action: shell /bin/nc -q1 -w2 splunk.com 23777 < output/bwtest_{{ tikproto }}_{{ tikhost }}.json
  ignore_errors: yes

Here we invoke a MikroTik RouterOS API client written in Ruby (http://github.com/astounding/mtik).

These three YAML files put together produce the following JSON output:

{
 "status": "done testing",
 "tx-10-second-average": "1680920",
 "direction": "both",
 "tx-total-average": "1680920",
 "rx-10-second-average": "501408",
 "rx-total-average": "501408",
 "proto": "tcp",
 "duration": "11s",
 "tx-current": "684832",
 "random-data": "false",
 "section": "11",
 "tag": "3",
 "rx-current": "341304",
 "host": "14.52.63.41",
 "freq": "5GHz"
}

Now what?

With this data sent to Logstash and Splunk we can finally produce some pretty graphs and gain some insights:

Top performing wireless zone per protocol:

Performance distribution per zone over time:

Total Rx/Tx performance over time:

We’ve only scratched the surface here, but you get the idea. Ansible allows you to automate just about anything and Kibana makes it look great

Check out my Consulting Services if you’re interested in implementing something similar for your organization.

Usually I start off a lab by building the topology itself – adding all the nodes, adding the network modules to each node and then connecting them all together. When that’s done I boot the topology in the cloud, figure out a proper addressing scheme, open a console session to each device, configure the Ethernet interfaces according to the scheme, configure loopback interfaces for router IDs and LAN simulation, configure global settings for lines, IPv6, logging etc. and then finally enable one or more routing protocols on all nodes for end-to-end connectivity.

What if I could outsource these initial router configuration steps? Just create a topology and off I go?

Unable to find a proper *aaS provider or Facebook app for this particular need, I built something that solves just that:

Lab configuration
Copy Dynamips (http://www.gns3.net/dynamips/) and a Cisco IOS image to your VM. Then configure the Dynamips processes to launch at startup and prepare a timestamp log file for later:

/etc/rc.local:
==============
/root/net/dynamips/dynamips-0.2.8-RC3-community-x86.bin -H 7200 &
/root/net/dynamips/dynamips-0.2.8-RC3-community-x86.bin -H 7201 &
/root/net/dynamips/dynamips-0.2.8-RC3-community-x86.bin -H 7202 &

touch /tmp/do_timestamp.log
date +%s > /tmp/do_timestamp.log

Run as many instances of Dynamips as you want. Two or more allows for load balancing from GNS3 by distributing the virtual routers across multiple processes.

Then configure GNS3 to use your VM as an “External Hypervisor”:

Create a topology, run it through GNSparser and off you go.

Even Michael Caine enjoy Cisco labs.

That awkward feeling

Two hours later and you’re done for the day. You shutdown your laptop but forget to destroy the VM!! The meter keeps on running. Leave it running idle for another two weeks and you’ve used 20 USD on cloud air. There’s no market for cloud air, so that money is lost. There are lots of better things to invest in http://www.thinkgeek.com/interests/giftsunder20/.

Here’s one way to avoid those situations.

We will create a script that checks if there are any active TCP sessions towards the Dynamips processes. Here’s an example of ten processes ready and listening:

root@gns:~# netstat -na4
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 0.0.0.0:7200            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:7201            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:7202            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:7203            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:7204            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:7205            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:7206            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:7207            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:7208            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:7209            0.0.0.0:*               LISTEN

Should the script find TCP sessions against ports 7200-7209 in an ESTABLISHED state, it will set an initial counter with the current UNIX timestamp.

After you’ve logged out and your TCP sessions are torn down, the script will figure out that there are no ESTABLISHED sessions left. It will then start to count down for one hour.

If you haven’t reconnected to the VM (resetting the counter) within that time frame, the VM will go kamikaze on itself and issue a destroy request using the DigitalOcean API (https://www.digitalocean.com/api).

Meet the self-destructing VM
First, install the PHP5 CLI interpreter package to run PHP scripts on your server. Here’s how on Ubuntu etc.:

$ apt-get install php5-cli

Copy the following script to a suitable location:

do_destroy.php
==============
<?php

$api_key = "secret_API_key";
$api_cid = "secret_API_cid";

$file = '/tmp/do_timestamp.log';
$ts_log = file_get_contents($file);
$ts_diff = time() - $ts_log;

$vm = "gns"; // NAME OF VIRTUAL MACHINE AT DO

function destroy($vm) {
        global $api_key, $api_cid;

        $do_status = file_get_contents('https://api.digitalocean.com/droplets/?client_id=' . $api_cid . '&api_key=' . $api_key);

        $ar_status = json_decode("$do_status", true);

        foreach($ar_status['droplets'] as $droplet) {
                if($droplet['name'] == $vm) {
#                       file_get_contents('https://api.digitalocean.com/droplets/' . $droplet['id'] . '/destroy/?client_id=' . $api_cid . '&api_key=' . $api_key);
                } else {
                        continue;
                }
        }
}

function acon() {
        $netstat = exec("netstat -na4 | grep -E ':720' | grep -E 'ESTABLISHED' | wc -l");
        if($netstat > 0) {
                return 1;
        } else {
                return 0;
        }
}

if(acon() == 1) {
        $curtime = time() . "n";
        file_put_contents($file, $curtime);
} elseif(acon() == 0 && $ts_diff > 3600) {
        destroy($vm);
} else {
        return 0;
}

?>

Configure a cron job to run every 15 minutes:

crontab -e:
===========
*/15 * * * * php /root/sh/do_destroy.php > /dev/null 2>&1

With all this configured and working, create a snapshot of your VM. This way you can setup a network lab whenever you feel like it. One way is by using the Android application Basin (http://basinapp.com/) on your phone. Best of all, no need to worry about leaving the VM running. Automation will take care of the cleaning <3

Good luck!

We will use the following Tcl script:

set i "0"

foreach subnet {
 48
 50
 52
 54
} {
 puts "interface loopback $i"
 puts " ip address 192.168.$subnet.1 255.255.255.0"
 incr i 1
}

Using this script we can do the following:

Router#
Router#tclsh
Router(tcl)#set i "0"
0
Router(tcl)#foreach subnet {
+> 48
+> 50
+> 52
+> 54
+>} {
+> puts "interface loopback $i"
+> puts " ip address 192.168.$subnet.1 255.255.255.0"
+> incr i 1
+>}
interface loopback 0
 ip address 192.168.48.1 255.255.255.0
interface loopback 1
 ip address 192.168.50.1 255.255.255.0
interface loopback 2
 ip address 192.168.52.1 255.255.255.0
interface loopback 3
 ip address 192.168.54.1 255.255.255.0

Router(tcl)#exit

Notice how the script only sends the configuration to standard output. In order to actually create the interfaces we have to copy and paste the script output into configure terminal:

Router#
Router#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
Router(config)#interface loopback 0
Router(config-if)# ip address 192.168.48.1 255.255.255.0
Router(config-if)#interface loopback 1
Router(config-if)# ip address 192.168.50.1 255.255.255.0
Router(config-if)#interface loopback 2
Router(config-if)# ip address 192.168.52.1 255.255.255.0
Router(config-if)#interface loopback 3
Router(config-if)# ip address 192.168.54.1 255.255.255.0
Router(config-if)#
*Mar  1 00:01:19.331: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback0, changed state to up
*Mar  1 00:01:19.547: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback1, changed state to up
*Mar  1 00:01:19.659: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback2, changed state to up
*Mar  1 00:01:19.779: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback3, changed state to up
Router(config-if)#end
Router#sh protocols
Global values:
  Internet Protocol routing is enabled
FastEthernet0/0 is administratively down, line protocol is down
FastEthernet0/1 is administratively down, line protocol is down
Loopback0 is up, line protocol is up
  Internet address is 192.168.48.1/24
Loopback1 is up, line protocol is up
  Internet address is 192.168.50.1/24
Loopback2 is up, line protocol is up
  Internet address is 192.168.52.1/24
Loopback3 is up, line protocol is up
  Internet address is 192.168.54.1/24

Very handy when doing labs