manually deploying openstack with a virtual maas on ubuntu trusty (part 1)

The goal of this new few series of posts is to be able to setup virtual machines to simulate a real-world openstack deployment using maas and juju. This goes through setting up a maas-server in a VM as well as setting up maas-nodes in VMs and getting them enlisted/commissioned into the maas-server. Next juju is configured to use the maas cluster. Finally, openstack is deployed using juju.

Overview

Requirements

Ideally, a large server with 16 cores, 32G memory, 500G disk. Obviously you can tweak this setup to work with less; but be prepared to lock up lesser machines. In addition your host machine needs to be able to support nested virtualization.

Topology

Here is the basics of what will be setup for our virtual maas cluster. Each red box is a virtual machine with two interfaces. The eth0 interface in the VM connects to the NATed maas-internet network, while the VM’s eth1 interface connects to the isolated maas-management network. The number of maas-nodes should match what is required for the deployment; however it is simple enough to enlist more nodes later. I choose to use a public/private network in order to be more flexible later in how openstack networking is set up.

Setup Host Machine

Install Requirements

First install all required programs on the host machine.

sudo apt-get install libvirt-bin qemu-kvm cpu-checker virtinst uvtool

Next, check if kvm is working correctly.

kvm-ok

Ensure nested KVM is enabled. (replace intel with amd if necessary)

cat /sys/module/kvm_intel/parameters/nested

This should output Y, if it doesn’t do the following:

sudo modprobe -r kvm_intel
sudo modprobe kvm_intel nested=1

Ensure $USER is added to libvirtd group.

groups | grep libvirtd

Ensure host machine has SSH keys generated and setup. (Be careful, don’t overwrite your existing keys)

[ -d ~/.ssh ] || ssh-keygen -t rsa

Virtual Network Setup

This step can be done via virt-manager, but also done via command line using virsh.

Setup a virtual network which uses NAT to communicate with the host machine with the following parameters:

Network Name: maas_internet
Network: 192.168.100.0/24
Do _not_ Enable DHCP.
Forwarding to physical network; Any physical device; NAT

And setup an isolated virtual network the following parameters:

Network Name: maas_management
Network: 10.10.10.0/24
Do _not_ Enable DHCP.
Isolated network;

Install the MAAS Server

Download and Start the Install

Ensure you have virt-manager connected to the hypervisor.
While there are many ways we can create virtual machines, I chose the tool uvtool because it works well in Trusty and quickly creates VM based on the Ubuntu cloud image.

Sync the latest cloud trusty cloud image:

uvt-simplestreams-libvirt sync release=trusty arch=amd64

Create a maas-server VM:

uvt-kvm create maas-server release=trusty arch=amd64 --disk 20 --memory 2048 --password ubuntu

After it boots, shut it down and  edit the VM’s machine configuration.
Make the two network interfaces connect to maas_internet and maas_management respectively.

Now edit /etc/network/interfaces to have the following:

auto eth0
iface eth0 inet static
  address 192.168.100.10
  netmask 255.255.255.0
  gateway 192.168.100.1
  dns-nameservers 10.10.10.10 192.168.100.1

auto eth1
iface eth1 inet static
  address 10.10.10.10
  netmask 255.255.255.0
  dns-nameservers 10.10.10.10 192.168.100.1

And follow the instructions here:
http://maas.ubuntu.com/docs/install.html#pkg-install

Which is essentially:

sudo apt-get install maas maas-dhcp maas-dns

MAAS Server Post Install Tasks

http://maas.ubuntu.com/docs/install.html#post-install

First let’s check if the webpage is working correctly. Depending on your installation, you may need to proxy into a remote host hypervisor before accessing the webpage. If you’re working locally you should be able to access this address directly (as the libvirt maas_internet network is already connected to your local machine).

If you need to access it indirectly (and 192.168.100.0 is a non-conflicting subnet):

sshuttle -D -r <hypervisor IP> 192.168.100.0/24

Access the following:
http://192.168.100.10/MAAS
It should remind you that post installation tasks need to be completed.

Let’s create the admin user from the hypervisor machine:
ssh ubuntu@192.168.100.10

sudo maas-region-admin createadmin --username=root --email="user@host.com" --password=ubuntu

If you want to limit the types of boot images that can be created you need to edit

sudo vim /etc/maas/bootresources.yaml

Import boot images, using the new root user you created to log in:
http://192.168.100.10/MAAS/clusters/
Now click 'import boot images' and be patient as it will take some time before these images are imported.

Add a key for the host machine here:
http://192.168.100.10/MAAS/account/prefs/sshkey/add/

Configure the MAAS Cluster

Follow instructions here to setup cluster:
http://maas.ubuntu.com/docs/cluster-configuration.html

http://192.168.100.10/MAAS/clusters/
Click on ‘Cluster master’
Click on edit interface eth1.

Interface: eth1
Management: DHCP and DNS
IP: 10.10.10.10
Subnet mask: 255.255.255.0
Broadcast IP: 10.10.10.255
Router IP: 10.10.10.10
IP Range Low: 10.10.10.100
IP Range High: 10.10.10.200

Click Save Interface
Ensure Nodes Auto-Enlist

Create a MAAS key and use that to log in:
http://192.168.100.10/MAAS/account/prefs/
Click on ‘+ Generate MAAS key’ and copy that down.

Log into the maas-server, and then log into maas using the MAAS key:
maas login maas-server http://192.168.100.10/MAAS

Now set all nodes to auto accept:

maas maas-server nodes accept-all

Setup keys on the maas-server so it can access the virtual machine host

sudo mkdir -p ~maas
sudo chown maas:maas ~maas
sudo -u maas ssh-keygen

Add the pubkey in ~maas/.ssh/id_rsa.pub to the virsh servers authorized_keys and to the maas SSH keys (http://192.168.100.10/MAAS/account/prefs/sshkey/add/)

sudo cat /home/maas/.ssh/id_rsa.pub

Now install virsh to test a connection and allow the maas-server to control maas-nodes.

sudo apt-get install libvirt-bin

Test the connection to the hypervisor (replace ubuntu with hypervisor host user)

sudo -u maas virsh -c qemu+ssh://ubuntu@192.168.100.1/system list --all

Confirm Maas-Server Networking

Ensure we can reach important address via maas-server:

host streams.canonical.com
host store.juju.ubuntu.com
host archive.ubuntu.com

And that we can download charms if needed:

wget https://store.juju.ubuntu.com/charm-info

Setup Traffic Forwarding

Setup maas-server to forward traffic from eth1 to eth0:

You can type the following out manually or add it as an upstart script to ensure forwarding is setup properly each time add this file to /etc/init/ovs-routing.conf (thanks to Juan Negron):

description "Setup NAT rules for ovs bridge"

start on runlevel [2345]

env EXTIF="eth0"
env BRIDGE="eth1"

task

script
    echo "Configuring modules"
    modprobe ip_tables || :
    modprobe nf_conntrack || :
    modprobe nf_conntrack_ftp || :
    modprobe nf_conntrack_irc || :
    modprobe iptable_nat || :
    modprobe nf_nat_ftp || :

    echo "Configuring forwarding and dynaddr"
    echo "1" > /proc/sys/net/ipv4/ip_forward
    echo "1" > /proc/sys/net/ipv4/ip_dynaddr

    echo "Configuring iptables rules"
    iptables-restore <<-EOM
*nat
-A POSTROUTING -o ${EXTIF} -j MASQUERADE
COMMIT
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A FORWARD -i ${BRIDGE} -o ${EXTIF} -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
-A FORWARD -i ${EXTIF} -o ${BRIDGE} -j ACCEPT
-A FORWARD -j LOG
COMMIT
EOM

end script

Then start the service:

sudo service ovs-routing start

Setup Squid Proxy

Ensure squid proxy can access cloud images:

echo "cloud-images.ubuntu.com" | sudo tee /etc/squid-deb-proxy/mirror-dstdomain.acl.d/98-cloud-images

sudo service squid-deb-proxy restart

Install MAAS Nodes

Now we can virt-install each maas-node on the hypervisor such that it automatically pxe boots and auto-enlists into MAAS. You can adjust the script below to create as many nodes as required. I’ve also simplified things by creating everything with dual nics and ample memory and hard drive space, but of course you could use custom machines per service. Compute-nodes need more compute power, ceph nodes will need more storage, and quantum-gateway will need dual nics. In addition you could specify raw disks instead of qcow2, or use storage pools; but in this case I wanted something simple that didn’t automatically use all the space it needed.

for i in {0..19}; do
  virt-install \
    --name=maas-node-${i} \
    --connect=qemu:///system --ram=4096 --vcpus=1 --hvm --virt-type=kvm \
    --pxe --boot network,hd \
    --os-variant=ubuntutrusty --graphics vnc --noautoconsole --os-type=linux --accelerate \
--disk=/var/lib/libvirt/images/maas-node-${i}.qcow2,bus=virtio,format=qcow2,cache=none,sparse=true,size=32 \
    --network=network=maas_internet,model=virtio \
    --network=network=maas_management,model=virtio 
done

Now each node needs to be manually enlisted with the proper power configuration.
http://maas.ubuntu.com/docs/nodes.html#virtual-machine-nodes

Host Name: maas-node-${i}.vmaas
Power Type: virsh
Power Address: qemu+ssh://ubuntu@192.168.100.1/system
Power ID: maas-node-${i}

Here we need to match the machines to the mac address and update the power requirements.  You can get the mac addresses of each node by using the following on the hypervisor:

virsh dumpxml maas-node-${i} | grep "mac addr"

Here is a script that helps automate some of this process, it can be run from the maas-server (replace USER ubuntu with the appropriate value) this matches mac address from virsh to the ones in maas and then sets up the power accordingly:

#!/usr/bin/python

import sys, os, libvirt
from xml.dom.minidom import parseString
os.environ['DJANGO_SETTINGS_MODULE'] = 'maas.settings'
sys.path.append("/usr/share/maas")
from maasserver.models import Node, Tag

hhost = 'qemu+ssh://ubuntu@192.168.100.1/system'

conn = libvirt.open(hhost)
nodes_dict = {}
domains = conn.listDefinedDomains()
for node_name in domains:
    node = conn.lookupByName(node_name)
    node_xml = parseString(node.XMLDesc(0))
    node_mac1 = node_xml.getElementsByTagName('interface')[0].getElementsByTagName('mac')[0].getAttribute('address')
    nodes_dict[node_mac1] = node_name

maas_nodes = Node.objects.all()
for node in maas_nodes:
    try:
        system_id = node.system_id
        mac = node.get_primary_mac()
        node_name = nodes_dict[str(mac)]
        node.hostname = node_name
        node.power_type = 'virsh'
        node.power_parameters = { 'power_address':hhost, 'power_id':node_name }
        node.save()
    except: pass

Note you will need python-libvirt and run the above command with something like the following:

sudo -u maas ./setup-nodes.py

Setup Fastpath and Commission Nodes

You most likely want to use fast-path installer on nodes to speed up installation times. Set all nodes to use fastpath installer using another bulk action on the nodes.

After you have all this done, click bulk action commission.
You should see all your machines starting up if you set things up properly, give this some time. You should have all the nodes in the 'Ready' state in maas now!
http://192.168.100.10/MAAS/nodes/

Confirm DNS setup

One point of trouble can be ensuring DNS is setup correctly. We can test this by starting a maas node and inside of that trying the following:

dig streams.canonical.com
dig store.juju.ubuntu.com

If we can’t hit those, we’ll need to ensure the maas server is setup correctly.
Go to: http://192.168.100.10/MAAS/settings/
Enter the host machines upstream DNS if necessary here, it should setup the bind configuration file and restart that service. After this re-test.

In addition I had to disable dnssec-validation for bind. Edit the following file:

sudo vim /etc/bind/named.conf.options

And change the following value:

dnssec-validation no;

And restart the service:

sudo service bind9 restart

Now you have a working virtual maas setup using the latest Ubuntu LTS!