Automate network topologies with CML, Ansible & Docker.

Intro

After many hours of reading Ansible books, looking at codes and testing, I’m continuing with the network project I started on my previous article.

On this post you will find how to create day-0 network topologies from scratch using Cisco Modelling Labs, Ansible and Docker.

Why Cisco Modelling Labs (CML)?, first and foremost, it is free (just reserve a lab). It comes with a default selection of Cisco images and won’t put a strain on your local machine’s resources. There is no need for installation or configuration. Furthermore, it is not limited to Cisco devices alone.

Prerequisites

To create the content of this post, you need:

CML instance, either personal or from Devnet sandbox.
If you don’t have a CML instance:
- Reserve a lab in the Cisco Devnet Sandbox. Is free.
- Cisco Anyconnect in your laptop. You need the VPN client to reach the network where the devices are reachable.
- A mail will be sent with the details to connect or click in “output” in the sandbox page.
Docker on the client machine.
Git, to clone the repo.

TL;DR

To get your lab, just do.

Clone the release v1.0.0 from Github.
- See below if you are not sure which command use to clone an specifc tag.
- The code in github is a living project, the main branch could have a different code than the one presented on the post, therefore all links presented in this article point to release v1.0.0
Start the container from the root directory of the project.
- docker-compose -f ./docker/cml.docker-compose.yml up -d
Create the lab from inside the container.
- ansible-playbook cisco.cml.build -e startup='host'

CML Lab Created

With that, you will be ready to go and have a lab in CML. If you want to understand how this works, keep reading.

Why?

As I’m experimenting network automation, I need a baseline to start, this post automates this baseline so I can focus on trying new ideas. The lab created, contains the day-0 config to start.

High Level Overview

The benefits of using Docker, Ansible and CML are:

Quickly create and destroy labs and containers.
Ensure consistency in network topology creation.
Write your network lab as code.

Not everything is perfect, there are a couple of manual pieces that have to be done. I would say this is more a limitation at the time of writing the post.

Prepare the Project

Go and grab the code at the GitHub repo.

1
git clone -b v1.0.0 --single-branch https://github.com/jillesca/network-automation.git

Project Structure

The project is divided in two main directories.

Ansible
- This directory contains all files related to ansible, such as inventory, playbooks, variables, etc.
Docker
- This directory has all files related to docker, such as dockerfiles, docker-compose files, environment variables.

1
╰─ tree -FL 2
2
./
3
├── LICENSE
4
├── README.md
5
├── ansible/
6
│   ├── ansible.cfg
7
│   ├── files/
8
│   ├── host_vars/
9
│   ├── inventory/
10
│   ├── playbooks/
11
│   └── templates/
12
└── docker/
13
    ├── ansible.docker-compose.yml
14
    ├── cml.docker-compose.yml
15
    └── image/
16

17
8 directories, 5 files

Ansible image

Now is time to work with Ansible in docker, there are two options:

You can build your own docker image.
Or you can reuse the image I’ve created.

The main consideration is the version of CML you are using.

For this post, CML was running virl 2.2.2+build52. If your CML is running the same version, you can skip the building part.

If you don’t know the virl version of your CML, you can do from your terminal.

1
curl https://{CML_IP}/api/v0/system_information -k

For example:

1
╰─ curl https://10.10.20.161/api/v0/system_information -k
2
{
3
  "version": "2.2.2+build52",
4
  "ready": true
5
}%

Build the Ansible container

Follow this section if you want to build your own image, otherwise you can use the image built from Docker Hub.

We start by reusing the Ansible image built in the previous post. This image will serve as base to add the additional libraries needed to manage CML.

To manage CML on Ansible, we use cisco.cml collection. which is available from Galaxy Collection. This library uses virl2-client under the hood.

If you want to know the details of this collection, check out the project in Github.

On the dockerfile we defined what is commented above; the image to use and which libraries to installed on pip and Ansible.

1
FROM jillesca/ansible:version1.1 as cml
2
RUN pip install virl2-client==2.2.1.post2 \
3
    && ansible-galaxy collection install cisco.cml

See virl2-client depends on the CML for more details.

To build the docker container use the command below from the root directory of the project. Otherwise adjust the path of the dockerfile.

1
docker build --target cml --file docker/image/ansible.dockerfile .

Add your environment variables

The project uses environment variables. This is a good way to manage secrets or special values you don’t want to store in git.

The collection cisco.cml, requires some variables. We use env vars to define these values so your credentials are out of the repo.

The default values from the sandbox (except the lab name) are used:

1
CML_USERNAME=developer
2
CML_PASSWORD=C1sco12345
3
CML_HOST=10.10.20.161
4
CML_LAB="ansible_base_lab"
5
CML_VERIFY_CERT=false

In my case, the .env file is under the docker directory in the root project. Keep in mind the location where you place this file.

1
╰─ ls -lsa docker
2
total 24
3
0 drwxr-xr-x   6 jillesca  staff  192 Dec  4 11:17 .
4
0 drwxr-xr-x  10 jillesca  staff  320 Dec  1 12:59 ..
5
8 -rw-r--r--   1 jillesca  staff  117 Nov  1 15:43 .env
6
8 -rw-r--r--   1 jillesca  staff  851 Dec  4 13:37 ansible.docker-compose.yml
7
8 -rw-r--r--   1 jillesca  staff  420 Dec  4 13:43 cml.docker-compose.yml
8
0 drwxr-xr-x   4 jillesca  staff  128 Dec  4 11:54 image

Start the container

To work locally with the container, docker compose is used. Below you can see the definition.

1
╰─ cat docker/cml.docker-compose.yml
2
version: "3.8"
3

4
services:
5
  cml:
6
    image: jillesca/ansible:cml1.1
7
    container_name: cml
8
    tty: true
9
    volumes:
10
      - ./../ansible:/home
11
    environment:
12
      CML_USERNAME: ${CML_USERNAME}
13
      CML_PASSWORD: ${CML_PASSWORD}
14
      CML_HOST: ${CML_HOST}
15
      CML_LAB: ${CML_LAB}
16
      CML_VERIFY_CERT: ${CML_VERIFY_CERT}

From the compose file, two sections are important:

volumes:
- We say to docker, “copy everything that is under the ansible dir all the time.
- This allows to do changes on the code locally, and these changes will be replicated automatically on the container.
environment:
- This section injects the variables from the .env file to the container.

Run the container from root directory:

1
docker-compose -f ./docker/cml.docker-compose.yml up -d

1
╰─ docker-compose -f ./docker/cml.docker-compose.yml up -d
2

3
[+] Running 1/1
4
 ⠿ Container cml  Started                                                                  0.4s
5

6
╰─ docker ps
7
CONTAINER ID   IMAGE                     COMMAND     CREATED         STATUS         PORTS     NAMES
8
5af70918f7b8   jillesca/ansible:cml1.1   "python3"   3 seconds ago   Up 2 seconds             cml

Now you can connect to the container shell by using docker exec -it cml /bin/sh. If you use vscode, there are also extensions that will help you with some clicks:

1
╰─ docker exec -it cml /bin/sh
2
/home # ls
3
ansible.cfg  files        host_vars    inventory    playbooks    templates
4
/home #

Verification of env vars

This is a good moment to verify your env vars are present on the container, otherwise the lab will fail. From inside the container do env | grep -i cml:

1
/home # env | grep -i cml
2
CML_USERNAME=developer
3
CML_PASSWORD=C1sco12345
4
CML_HOST=10.10.20.161
5
CML_VERIFY_CERT=false
6
CML_LAB=ansible_base_lab
7
/home #

Prepare the Bastion/Jump Host

For this lab I used a Jump Host as the connectivity between the network devices created in CML and the ansible container was not reliable. Even though IP connectivity was there, from time to time I had timeouts, so the best solution was to use a VM the sandbox provides.

VM Sandbox Details

All these steps are done from inside the container -> docker exec -it cml /bin/sh

Create the ssh keys.

1
ssh-keygen -f /home/.ssh/cml -t ed25519 -C 'CML_Automation_key' -N ''

Create an authorized_keys file with the public ssh key created.

1
cp /home/.ssh/cml.pub /home/.ssh/authorized_keys

Copy the keys to the sandbox VM. This is the VM that will work as bastion/jump host. At the time of writing the VM on the Cisco sandbox uses the IP 10.10.20.50. Replace this IP with the address of the host you want to use as bastion.

Enter the password when prompted. At the time of writing the password was C1sco12345

1
scp /home/.ssh/authorized_keys developer@10.10.20.50:/home/developer/.ssh

Execute your ssh agent.

1
eval $(ssh-agent -s)

Add your keys to your agent

1
ssh-add /home/.ssh/cml

Test ssh is using your keys. The bastion/jump host should not ask for your password.

Below you can see an output of this process.

1
/home # ssh-keygen -f /home/.ssh/cml -t ed25519 -C 'CML_Automation_key' -N ''
2
Generating public/private ed25519 key pair.
3
Your identification has been saved in /home/.ssh/cml
4
Your public key has been saved in /home/.ssh/cml.pub
5
The key fingerprint is:
6
SHA256:4z4psi8nJ0hqnwXd05R5DuVeqPUcCu7YCpZnjUHjgwQ CML_Automation_key
7
The key's randomart image is:
8
+--[ED25519 256]--+
9
|   E       .     |
10
|    .     = .    |
11
|     . o * = o   |
12
|    o = = O = .  |
13
|   . o *S+ + o   |
14
|  . . ..X.       |
15
| o . = =.=       |
16
|... O B.+        |
17
|. .o.X.o..       |
18
+----[SHA256]-----+
19
/home # cp /home/.ssh/cml.pub /home/.ssh/authorized_keys
20
/home # scp /home/.ssh/authorized_keys developer@10.10.20.50:/home/developer/.ssh
21
The authenticity of host '10.10.20.50 (10.10.20.50)' can't be established.
22
ED25519 key fingerprint is SHA256:MbXlsdtKy1J+Tj67hyVRPz5URQS/6eT2ILljoG1ihqA.
23
This key is not known by any other names
24
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
25
Warning: Permanently added '10.10.20.50' (ED25519) to the list of known hosts.
26
developer@10.10.20.50's password:
27
authorized_keys                                        100%  100     0.5KB/s   00:00
28
/home # eval $(ssh-agent -s)
29
Agent pid 85
30
/home # ssh-add /home/.ssh/cml
31
Identity added: /home/.ssh/cml (CML_Automation_key)
32
/home # ssh developer@10.10.20.50
33
(py3venv) [developer@devbox ~]$

Create the lab

All these steps are done from inside the container.

With all pre-requisites done, is time to fire up the lab. To create the lab on CML, simply do:

1
ansible-playbook cisco.cml.build -e startup='host'

Below you can see the full output of the lab creation.

1
/home # ansible-playbook cisco.cml.build -e startup='host'
2
SSL Verification disabled
3
[WARNING]: running playbook inside collection cisco.cml
4

5
PLAY [Build the topology] ***************************************************************
6

7
TASK [Check for the lab file] ***********************************************************
8
ok: [localhost]
9

10
TASK [assert] ***************************************************************************
11
ok: [localhost] => {
12
    "changed": false,
13
    "msg": "All assertions passed"
14
}
15

16
TASK [Create the lab] *******************************************************************
17
[WARNING]: Both option username and its alias user are set.
18
changed: [localhost]
19

20
TASK [Check to see if the Lab is there] *************************************************
21
skipping: [localhost]
22

23
TASK [Refresh Inventory] ****************************************************************
24
SSL Verification disabled
25

26
PLAY [Start Individual Nodes] ***********************************************************
27

28
TASK [Check for the cml_config_file] ****************************************************
29
skipping: [bridge-to-sandbox]
30
skipping: [sandbox-backend]
31
ok: [spine2-20.12 -> localhost]
32
ok: [leaf1-20.21 -> localhost]
33
ok: [leaf3-20.23 -> localhost]
34
ok: [spine1-20.11 -> localhost]
35
ok: [leaf2-20.22 -> localhost]
36
ok: [host2-20.32 -> localhost]
37
ok: [host1-20.31 -> localhost]
38
ok: [host3-20.33 -> localhost]
39

40
TASK [Read in cml_config_file] **********************************************************
41
skipping: [bridge-to-sandbox]
42
skipping: [sandbox-backend]
43
ok: [spine1-20.11]
44
ok: [spine2-20.12]
45
ok: [leaf1-20.21]
46
ok: [leaf2-20.22]
47
ok: [leaf3-20.23]
48
ok: [host1-20.31]
49
ok: [host2-20.32]
50
ok: [host3-20.33]
51

52
TASK [Start Individual Nodes] ***********************************************************
53
changed: [sandbox-backend -> localhost]
54
changed: [bridge-to-sandbox -> localhost]
55
changed: [leaf1-20.21 -> localhost]
56
changed: [spine1-20.11 -> localhost]
57
changed: [spine2-20.12 -> localhost]
58
changed: [leaf2-20.22 -> localhost]
59
changed: [host2-20.32 -> localhost]
60
changed: [host1-20.31 -> localhost]
61
changed: [leaf3-20.23 -> localhost]
62
changed: [host3-20.33 -> localhost]
63

64
PLAY [Wait for Topology to BOOT] ********************************************************
65

66
TASK [Check to see if all hosts are BOOTED] *********************************************
67
skipping: [localhost]
68

69
PLAY RECAP ******************************************************************************
70
bridge-to-sandbox          : ok=1    changed=1    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0
71
host1-20.31                : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
72
host2-20.32                : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
73
host3-20.33                : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
74
leaf1-20.21                : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
75
leaf2-20.22                : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
76
leaf3-20.23                : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
77
localhost                  : ok=3    changed=1    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0
78
sandbox-backend            : ok=1    changed=1    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0
79
spine1-20.11               : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
80
spine2-20.12               : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
81

82
/home #

From CML you can watch the progress of the lab created.

CML Lab Creation

If you examine the devices in the lab, you will see all the devices are running with their day-0 config. In our case, this day-0 config, is for the management of the device.

How the lab works?

If you are like me, you may wondering how the devices got their configs and how are the moving pieces inside Ansible. I consider this was the most difficult part for me to get my head around when starting with Ansible.

Ansible Inventory

The Ansible inventory is divided in three main components. Here is how the inventory looks like.

cml.yml

Defines the plugin and tags CML will use.

1
plugin: cisco.cml.cml_inventory
2
group_tags: clients, network

network.yml

Defines the network devices. Two main groups are defined network and iosxe_routers

1
all:
2
  children:
3
    network:
4
      children:
5
        iosxe_routers:
6
          hosts:
7
            spine1-20.11:
8
              ansible_host: 10.10.20.11
9
            spine2-20.12:
10
              ansible_host: 10.10.20.12
11
            leaf1-20.21:
12
              ansible_host: 10.10.20.21
13
            leaf2-20.22:
14
              ansible_host: 10.10.20.22
15
            leaf3-20.23:
16
              ansible_host: 10.10.20.23

system.yml

Defines the clients. Two main groups are defined clients and linux_hosts

1
all:
2
  children:
3
    clients:
4
      children:
5
        linux_hosts:
6
          hosts:
7
            host1-20.31:
8
              ansible_host: 10.10.20.31
9
            host2-20.32:
10
              ansible_host: 10.10.20.32
11
            host3-20.33:
12
              ansible_host: 10.10.20.33

Variable parsing in Ansible

Remember the lab structure presented before, now is time to see the Ansible part. Let’s see the relevant parts from another perspective.

When you execute ansible-playbook cisco.cml.build -e startup='host' the following happens:

Variable Parsing Flow

/inventory/group_vars/all/all.yml is parsed:
- Global variables are set, like ansible credentials, GW.
- ssh config for bastion is defined here.
*.yml files under /inventory are parsed (this is not default behaviour, see the file ansible.cfg on the repo).
/inventory/cml.yml
- Gets variables from /hosts_vars/cml.yml
  - The env vars for cml are parsed.
- Gets variable defined from /inventory/group_vars/all/cml.yml
  - cml_lab_file tells cisco.cml which file is use to create the topology in CML.
    - If you want to use your own lab, this is the file you need to point to.
    - The hosts defined on network.yml and system.yml match the hosts on this file.
network.yml
- Gets variables from /inventory/group_vars/iosxe_routers/iosxe_routers.yml
  - mgmt_interface sets which interface will be used for managment.
  - cml_config_file tells cisco.cml which file will be used as day-0 config.
    - This points to ansible/templates/iosxe/bootstrap.j2
    - The day-0 config is in jinja and takes several variables.
system.yml
- Gets variables from /inventory/group_vars/linux_hosts/linux_hosts.yml
  - cml_config_file tells cisco.cml which file will be used as day-0 config.
    - This points to /templates/linux_hosts/bootstrap.j2
    - The day-0 config is in jinja and takes several variables.

I would say this is the messy part, which at the beginning could be complex, but once you understand the flow it, becomes easier.

Once all variables are parsed and ready, cisco.cml proceeds to build the lab with its playbook cisco.cml.build.

Test the lab

For testing the lab we just created, we will be using cisco.ios.ios_facts The playbook below will print all facts gathered by this collection.

The goal is to be able to connect to the devices created with Ansible. And on this post, that will be through the Bastion/Jump host.

1
- name: Show cisco facts
2
  hosts: network
3
  connection: ansible.builtin.network_cli
4
  gather_facts: no
5

6
  tasks:
7
    - name: Gather cisco facts
8
      cisco.ios.ios_facts:
9
        gather_subset: all
10

11
    - name: Print cisco facts
12
      ansible.builtin.debug:
13
        var: ansible_facts

To run the playbook, simply do:

1
ansible-playbook playbooks/test_lab.yml

For example:

1
/home # ansible-playbook playbooks/test_lab.yml
2
SSL Verification disabled
3

4
PLAY [Show cisco facts] *****************************************************************
5

6
TASK [Gather cisco facts] ***************************************************************
7
ok: [spine1-20.11]
8
ok: [spine2-20.12]
9
ok: [leaf3-20.23]
10
ok: [leaf2-20.22]
11
ok: [leaf1-20.21]
12

13
TASK [Print cisco facts] ****************************************************************
14
ok: [leaf1-20.21] => {
15

16
. . . . .
17

18
ok: [spine1-20.11] => {
19
    "ansible_facts": {
20
        "net_all_ipv4_addresses": [
21
            "10.10.20.11"
22
        ],
23
        "net_all_ipv6_addresses": [],
24
        "net_api": "cliconf",
25
        "net_config": "Building configuration......."
26
        "net_filesystems": [
27
            "bootflash:"
28
        ],
29
        "net_filesystems_info": {
30
            "bootflash:": {
31
                "spacefree_kb": 5308528.0,
32
                "spacetotal_kb": 6139200.0
33
            }
34
        },
35
        "net_gather_network_resources": [],
36
        "net_gather_subset": [
37
            "interfaces",
38
            "config",
39
            "hardware",
40
            "default"
41
        ],
42
        "net_hostname": "spine1-20.11",
43
        "net_image": "bootflash:packages.conf",
44
        "net_iostype": "IOS-XE",
45
        "net_memfree_mb": 858103.16015625,
46
        "net_memtotal_mb": 1105215.80859375,
47
        "net_model": "CSR1000V",
48
        "net_neighbors": {},
49
        "net_python_version": "3.11.0",
50
        "net_serialnum": "9TZF7KV77J9",
51
        "net_system": "ios",
52
        "net_version": "17.03.02",
53
        "network_resources": {}
54
    }
55
}
56

57
PLAY RECAP ******************************************************************************
58
leaf1-20.21                : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
59
leaf2-20.22                : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
60
leaf3-20.23                : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
61
spine1-20.11               : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
62
spine2-20.12               : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
63

64
/home #

Delete the lab

To delete the lab on CML, use the playbook cisco.cml.clean, which uses the lab name defined from the env var:

1
ansible-playbook cisco.cml.clean

For example:

1
/home # ansible-playbook cisco.cml.clean
2
SSL Verification disabled
3

4
PLAY [localhost] ************************************************************************
5

6
TASK [Stop the lab] *********************************************************************
7
[WARNING]: Both option username and its alias user are set.
8
changed: [localhost]
9

10
TASK [Wipe the lab] *********************************************************************
11
changed: [localhost]
12

13
TASK [Erase the lab] ********************************************************************
14
changed: [localhost]
15

16
PLAY RECAP ******************************************************************************
17
localhost                  : ok=3    changed=3    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
18

19
/home #

If you want to specify a lab to delete, use -e cml_lab=LAB_NAME. And if you have spaces in your lab name use double quotations, for example: ' inside ""

1
ansible-playbook cisco.cml.clean -e cml_lab="'Small NXOS/IOSXE Network'"

From CML you can watch the progress of the lab deleted.

There are more options on the playbook cisco.cml.clean take a look

How can I add my own lab?

If you follow the structure layout in the project, the best way to create your own lab is the following:

Create a baseline topology:
- Create a lab in CML by hand or use one of their examples.
- Export the lab from CML in yaml format.
  - In the GUI; Bottom bar > Simulate > Download Lab
- If you want to manage day-0 config, remove any configuration for the devices.
- Add the extension .j2 to the yaml file to make it a jinja file and place it at /files/topologies/base-setup/. For example.
- You could have many labs stored there, but the lab that is picked up, is controlled by the variable cml_lab_file.
- Don’t forget to adjust the Ansible inventory based on the hosts you added on the CML file.
Day-0 config
- Day-0 config is managed based on the host group.
- the variable cml_config_file tells CML which file has the day-0 config for a device.
- Place you day-0 config on the file you defined in cml_config_file. This file has to a jinja file.

If you follow these considerations, you should be able to add as many labs as you want.

If you want to create several labs, remember to update the corresponde variables.

Conclusion

This post took way longer that I expected when I had the idea. What I thought it could be short article, became way larger than what I was thinking.

But when trying to figure out, how to automate a lab in CML with Ansible, not many resources when into the detail about how it works, just assumed you already knew, that’s why I wanted to add relevant details and try to be clear.

Additionally, once the lab process is automated, you don’t need to repeat all the steps here, eventually becomes an easy process. But when there is an issue, you will be happy there is a detailed article explaining how it works and how you can use it.

Credits

I can’t take credit for the Ansible structure of the project and how it works. When learning about cisco.cml, I found this awesome organization model-driven-devops. which is part of the book Model-Driven DevOps.

Take a look at the GitHub organization and book to learn more about how to manage networks using single source of truth and more interesting for me, working with structured data.