Getting started with CoreOS on Nutanix Community Edition

Containers seem to be the hot trend right now. I needed to get some more experience in this area, and instead of working with a single container machine, I actually wanted to get a “quick” distributed setup going. It wasn’t all that quick to start with, but I now have a working setup that can actually be rolled out and scaled in a pretty quick fashion.

Now, I’m assuming you already know what a container is, and have heard about CoreOS. Here are some quick steps to get you started. I’ll start off with the prerequisites:

• You will have your Nutanix CE cluster up and running
• You have a VLAN set up with IP address management and a DHCP server on Nutanix CE
• You downloaded the CoreOS ISO image

Your further steps are relatively simple. First off, we will create an etcd master, the most important thing we need is a fixed IP, so define which IP you want to give it. Obviously we could use the CoreOS cluster discovery mechanism and rely on an internet connection, but I decided to just use my own instance instead.

Start off by creating a cloud-config file for your etcd master:

#cloud-config
ssh_authorized_keys:
  - ssh-rsa AAAAB3NzaC1...
coreos:
  etcd2:
    name: etcdserver
    initial-cluster: etcdserver=http://<etcd-vm-ip-here>:2380
    initial-advertise-peer-urls: http://<etcd-vm-ip-here>:2380
    advertise-client-urls: http://<etcd-vm-ip-here>:2379
    # listen on both the official ports and the legacy ports
    # legacy ports can be omitted if your application doesn't depend on them
    listen-client-urls: http://0.0.0.0:2379,http://0.0.0.0:4001
    listen-peer-urls: http://0.0.0.0:2380
 units:
    - name: etcd2.service
    command: start
    - name: 00-eth0.network
    runtime: true
    content: |
      [Match]
      Name=eth0
      
      [Network]
      DNS=<your-dns-ip-here>
      Address=<etcd-vm-ip-here>/16
      Gateway=<your-gateway-ip-here>

Note that I’ve copied in the public ssh key from my laptop to get easier access to the VM. Now, save this file as a text file called user_data, and create an iso image using the ways described here. Copy that over to your container on CE using sftp to a controller VM on port 2222. You can use Prism credentials to authenticate.

Next step, create a new VM in Acropolis, attach the CoreOS ISO image as your primary CD drive, and the ISO you just created as the second CD drive, and power on the VM

Now, to create the actual CoreOS cluster, you create a second user_data file, that only contains the following:

#cloud-config
ssh_authorized_keys:
 - ssh-rsa AAAAB3NzaC1...
coreos:
  etcd2:
    proxy: on
    initial-cluster: etcdserver=http://<etcd-vm-ip-here>:2380
    listen-client-urls: http://localhost:2379
  fleet:
    etcd_servers: http://localhost:2379
    metadata: "role=etcd"
  units:
    - name: etcd2.service
    command: start
    - name: fleet.service
    command: start

For quick deployment, I’d create a VM that you use as a template to clone from. Give the VM the newly created file as the secondary drive.

Now, just create some clones, power them on and wait for them to get their IP. You should then be able to ssh into the machine using the “core” user, and check your cluster:

core@CoreOS-1 ~ $ fleetctl list-machines
MACHINE     IP             METADATA
1c24fc23... 192.168.96.248 role=etcd
8e974c05... 192.168.79.25  role=etcd
a899b944... 192.168.114.8  role=etcd
f916eb93... 192.168.3.179  role=etcd

With that, you can start implementing and rolling out your units:

core@CoreOS-1 ~ $ fleetctl load hello.service
Unit hello.service loaded on 1c24fc23.../192.168.96.248

core@CoreOS-1 ~ $ fleetctl list-units
UNIT          MACHINE                    ACTIVE   SUB
hello.service 1c24fc23.../192.168.96.248 inactive dead