Ansible and ESXi-ARM on Raspberry Pi4

 Ansible and ESXi-ARM on Raspberry Pi 4




Why

When I heard about Project Monterey at VMworld 2020, especially running ESXi on Smart NICs, my head just about exploded. Amazing. Imagine network segmentation and bare metal provisioning running ON THE NIC.

Five days after VMworld 2020, the ESXi-ARM Fling was released. This got me thinking "What if I could simulate network segmentation with ESXi running on ARM?" My customers are very invested in improving Manufacturing Operational Technologies (OT). OT deals with specialized manufacturing devices as well as Internet of Things (IOT). A network segmentation experiment on ARM could apply very well to OT. Off I went to order a Raspberry Pi 4 and to download the ESXi-ARM bits.

The Condensed Version of Installing ESXi-ARM

  • Get the Bits: Download the ESXi-Arm-ISO, ESXi-Arm-Fling-Doc.pdf, Fling-on-Raspberry-Pi.pdf from here.










  • RTFM: Really. The documentation is excellent. And like those trick exams in school, read to the end before you start.
  • All the Parts: Make sure you have all the necessary Raspberry Pi4 parts (also in the docs)
    • Raspberry Pi 4 8GB
    • 3.5 Amp power supply
    • 1 x micro SD card for UEFI firmware (& Raspberry Pi OS for EEPROM update)
    • 1 x USB drive for installer ISO (I used an old thumb drive)
    • 1 x USB drive for the actual ESXi installation (I used a new USB3 thumb drive, as I did not have a large, working, USB hard drive. I ended up using NFS on a Linux laptop for guests.)
    • That weird micro-HDMI to HDMI converter that you probably don't have. (I did not)
    • USB keyboard
    • USB Mouse
    • (Also, get some sort of fan or case with fan. I did not and my Pi 4 overheated right away. I now have a small room fan blowing on it.)
  • EEPROM: Burn Raspberry Pi OS to micro SD card, boot, update EEPROM
  • Firmware/EFI: Re-use the same micro SD card to build your firmware/EFI boot and increase the EFI RAM limit beyond 3GB.
    • My mistake: All files go in the root (/) of the micro SD card not in a /boot directory. I was confused as the documentation states "copy the entire 'boot' directory onto the newly formatted SD card". 
  • ESXi-ARM Boot/Install: Burn the ESXi-ARM ISO to the USB drive. Hit ESC to tell Device Manager to boot off USB and install.
  • Post Install: Enable ESXi Shell & ssh (from the MUI), enable NTP (from the web client) [Update: If you want to attach more than one USB disk to your Raspberry Pi 4, follow the USB Arbitrator instructions in Daniele Ulrich's blog post here.]
  • Guest OS Install: URLs to Arm64 guest OSes are listed in the ESXi-Arm-Fling-Doc.pdf.
    • My mistake: My first guest OS was Photon OS. You have to scroll down the Photon OS download page to OS 3.0 Revision2 Update1 for an ISO for Arm64. It took me a while to figure that out. 















ESXi Network Segmentation with Ansible

I suppose I could have used many other automation tools (PowerCLI, Terraform, vRealize Automation), but Ansible has such simple requirements. Can you ssh into the target? Can the target run python? That's it. Here are the steps I took to pull this off. I'll spare you the two days of YAML indentation pain that always accompanies Ansible.


Set Up Password-less SSH to ESXi

[Edit. Great news. Thanks to @arielsanchezmor on Twitter, one does not have to mess with password-less ssh when one is using the PyVmoni VMware API python library. You can find Ariel's code and vBrown Bag session here.]

[Ansible executes operations over ssh to the target(s). when not using the VMware API. We have already turned on the ESX shell and ssh in ESXi-ARM. If you want to use non-VMware API Ansible playbooks, the first step is to set up password-less ssh from the Ansible control node (my Mac) to ESXi-ARM. I outlined these ssh steps in my Ansible 101 blog post. The only difference with ESXi versus Linux is that the you place your control node public key in /etc/ssh/keys-<username>/authorized_keys not .ssh/authorized_keys. This article was helpful for that difference.]

Install Ansible on Your Control Node


You should be able to find the instructions for your control node operating system on this page. For my Mac, the install command was:

> pip install ansible


Install the Python SDK for the VMware vSphere API 


PyVmoni allows user to manage ESX, ESXi, and vCenter infrastructure and removes the need for password-less ssh to ESXi. To install, run this command on your ansible control node.
 
> pip install PyVmomi

Define Your Ansible Targets (For Non-VMware API Tasks)

On your Ansible control node, edit /etc/ansible hosts and define your ESXi-ARM hosts(s). My simple setup looks like this:

[servers]
esxi-arm ansible_ssh_host=192.168.1.168

Define the User Account for Your Ansible Targets (For Non-VMware API Tasks)

Create the file (and directory if necessary) /etc/ansible/group_vars/servers. The file is named "servers" as we named our group of ESXi-ARM hosts "[servers]". In that file, add these two lines:

---
ansible_ssh_user: root


Run a Smoke Test
















Before we create our first YAML ansible playbook, a word about YAML. 😱
Many of us grew up with, and love, vi(m). You should use an editor that understands YAML indentation like Atom or VS Code or Notepad++ otherwise you may go insane. YAML, like Python, depends on _exact_ indentation to define code blocks. If your indentation is off by one space, ansible will throw errors.

Now, for a simple ansible smoke test. Edit this very simple ansible playbook, esx_kernel_version.yml:

---
- hosts: servers
  tasks:
  - debug: msg={{ ansible_kernel }}

Then run the playbook:

> ansible-playbook esx_kernel_version.yml 


PLAY [servers] **************************


TASK [Gathering Facts] ******************

ok: [esxi-arm]


TASK [debug] ****************************

ok: [esxi-arm] => {

    "msg": "7.0.0"

}


PLAY RECAP ******************************

esxi-arm                   : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   


If you successfully received the debug message "7.0.0" then your Ansible is working. Congratulations.

Automate Network Segmentation - Sort Of



The real future state of this proof of concept would be automating NSX network segmentation. Automating simple ESXi firewall rules worked for me for a proof of concept until NSX is running on ESXi-ARM. 

This was my first time automating ESXi with Ansible, so I did a lot of Googling. The Ansible VMware Plugin page was very helpful especially the Examples for each plugin. Here are the two ansible playbooks I created to add and remove subnet access to ESXi ssh:

Create Your VMware Ansible Secrets File

Secrets files allow you to embed private information in your playbooks. Here is the secrets_esxi.yml file I am using for this example"

---
username: root
password: SuperSecret01!

vm_firewall_ssh_2nets.yml

---
- hosts: esxi-arm
#  vars:
#      ansible_python_interpreter: /bin/python
  tasks:
    - name: Including Secret Environment Items
      include_vars:
        file: secrets_esxi.yml
        name: secret
#
    - name: Manage Firewall
      community.vmware.vmware_host_firewall_manager:
        esxi_hostname: 192.168.1.168
        hostname: 192.168.1.168
        username: "{{secret.username}}"
        password: "{{secret.password}}"
        validate_certs: no
        rules:
          - name: sshServer
            enabled: True
            allowed_hosts:
              all_ip: False
              ip_network:
                - 192.168.86.0/24
                - 192.168.1.0/24
      delegate_to: localhost

The "delegate_to:" line was important as the necessary python modules could not be installed on ESXi-ARM. I think this phrase is pretty important with ESXi-ARM. Again, I do not run Ansible for a living. 

Running this task will make sure that the two subnets are allowed to access ssh on the ESXi-ARM host.



















vm_firewall_ssh_1net.yml

I created this ansible playbook to simulate network segmentation by shutting down access to all but the subnet the ESXi server is on.

---
- hosts: esxi-arm
#  vars:
#      ansible_python_interpreter: /bin/python
  tasks:
    - name: Including Secret Environment Items
      include_vars:
        file: secrets_esxi.yml
        name: secret
#
    - name: Manage Firewall
      community.vmware.vmware_host_firewall_manager:
        esxi_hostname: 192.168.1.168
        hostname: 192.168.1.168
        username: "{{secret.username}}"
        password: "{{secret.password}}"
        validate_certs: no
        rules:
          - name: sshServer
            enabled: True
            allowed_hosts:
              all_ip: False
              ip_network:
                - 192.168.1.0/24
      delegate_to: localhost


After running this playbook, you will see that the 192.168.86.0 subnet has been removed from ssh access.





















Thank You

Thank you for taking the time to read this post. I hope I may have saved you some time in your own education. I am new to both ESXi-ARM and Ansible and welcome your feedback.

Comments