Automate bare metal server provisioning using Ironic (bifrost) and the ansible deploy driver
On our team, we mostly conduct various research in OpenStack, so we use bare metal machines extensively. To make our lives somewhat easier, we've developed a set of simple scripts that enables us to backup and restore the current state of the file system on the server. It also enables us to switch between different backups very easily. The set of scripts is called multi-root (https://github.com/vnogin/multi-root).
Unfortunately, we had a problem; in order to use this tool, we had to have our servers configured in a particular way, and we faced different issues with manual provisioning:
As you can see here, we have the example of a bare metal server, which includes two physical disks. Those disks are combined using RAID1, then partitioned by the operating system. The LVM partition then gets further partitioned, with each copy of an operating system image assigned to its own partition.
This is our network diagram:
In this case we have one network to which our bare metal nodes are attached. Also attached to that network is the IRONIC server. A DHCP server assigns IP addresses for the various instances as they're provisioned on the bare metal nodes, or prior to the deployment procedure (so that we can bootstrap the destination server).
Now let's look at how to make this work.
I want to say thank you to Pavlo Shchelokovskyy and Ihor Pukha for your help and support throughout the entire process. I am very grateful to you guys.
Unfortunately, we had a problem; in order to use this tool, we had to have our servers configured in a particular way, and we faced different issues with manual provisioning:
- It is not possible to set up more than one bare metal server at a time using a Java-based IPMI application
- The Java-based IPMI application does not properly handle disconnection from the remote host due to connectivity problems (you have to start installation from the very beginning)
- The bare metal server provisioning procedure was really time consuming
- For our particular case, in order to use multi-root functionality we needed to create software RAID and make required LVM configurations prior to operating system installation
Lab structure
This is how we manage disk partitions and how we use software RAID on our machines:
As you can see here, we have the example of a bare metal server, which includes two physical disks. Those disks are combined using RAID1, then partitioned by the operating system. The LVM partition then gets further partitioned, with each copy of an operating system image assigned to its own partition.
This is our network diagram:
In this case we have one network to which our bare metal nodes are attached. Also attached to that network is the IRONIC server. A DHCP server assigns IP addresses for the various instances as they're provisioned on the bare metal nodes, or prior to the deployment procedure (so that we can bootstrap the destination server).
Now let's look at how to make this work.
How to set up bifrost with ironic-ansible-driver
So let's get started.- First, add the following line to the /root/.bashrc file:
# export LC_ALL="en_US.UTF-8" - Ensure the operating system is up to date:
# apt-get -y update && apt-get -y upgrade - To avoid issues related to MySQL, we decided to ins tall it prior to bifrost and set the MySQL password to "secret":
# apt-get install git python-setuptools mysql-server -y - Using the following guideline, install and configure bifrost:
# mkdir -p /opt/stack # cd /opt/stack # git clone https://git.openstack.org/openstack/bifrost.git # cd bifrost
- We need to configure a few parameters related to localhost prior to the bifrost installation. Below, you can find an example of an /opt/stack/bifrost/playbooks/inventory/group_vars/localhost file:
echo "--- ironic_url: "http://localhost:6385/" network_interface: "p1p1" ironic_db_password: aSecretPassword473z mysql_username: root mysql_password: secret ssh_public_key_path: "/root/.ssh/id_rsa.pub" deploy_image_filename: "user_image.qcow2" create_image_via_dib: false transform_boot_image: false create_ipa_image: false dnsmasq_dns_servers: 8.8.8.8,8.8.4.4 dnsmasq_router: 172.16.166.14 dhcp_pool_start: 172.16.166.20 dhcp_pool_end: 172.16.166.50 dhcp_lease_time: 12h dhcp_static_mask: 255.255.255.0" > /opt/stack/bifrost/playbooks/inventory/group_vars/localhost
As you can see, we're telling Ansible where to find Ironic and how to access it, as well as the authentication information for the database so state information can be retrieved and saved. We're specifying the image to use, and the networking information.
Notice that there's no default gateway for DHCP in the configuration above, so I'm going to fix it manually after the install.yaml playbook execution. - Install ansible and all of bifrost's dependencies:
# bash ./scripts/env-setup.sh # source /opt/stack/bifrost/env-vars # source /opt/stack/ansible/hacking/env-setup # cd playbooks
- After that, let's install all packages that we need for bifrost (Ironic, MySQL, rabbitmq, and so on) ...
# ansible-playbook -v -i inventory/localhost install.yaml - ... and the Ironic staging drivers with already merged patches for enabling Ironic ansible driver functionality:
# cd /opt/stack/ # git clone git://git.openstack.org/openstack/ironic-staging-drivers # cd ironic-staging-drivers/
- Now you're ready to do the actual installation.
# pip install -e . # pip install "ansible>=2.1.0"
You should see typical "installation" output. - In the /etc/ironic/ironic.conf configuration file, add the "pxe_ipmitool_ansible" value to the list of enabled drivers. In our case, it's the only driver we need, so let's remove the other drivers:
# sed -i '/enabled_drivers =*/c\enabled_drivers = pxe_ipmitool_ansible' /etc/ironic/ironic.conf - If you want to enable cleaning and disable disk shredding during the cleaning procedure, add these options to /etc/ironic/ironic.conf:
automated_clean = true erase_devices_priority = 0
- Finally, restart the Ironic conductor service:
# service ironic-conductor restart - To check that everything was installed properly, execute the following command:
# ironic driver-list | grep ansible | pxe_ipmitool_ansible | test |
You should see the pxe_ipmitool_ansible driver in the output. - Finally, add the default gateway to /etc/dnsmasq.conf (be sure to use the IP address for your own gateway).
# sed -i '/dhcp-option=3,*/c\dhcp-option=3,172.16.166.1' /etc/dnsmasq.conf
How to use ironic-ansible-driver to provision bare-metal servers with custom configurations
Now let's look at actually provisioning the servers. Normally, we'd use a custom ansible deployment role that satisfies Ansible's requirements regarding idempotency to prevent issues that can arise if a role is executed more than once, but because this is essentially a spike solution for us to use in the lab, we've relaxed that requirement. (We've also hard-coded a number of values that you certainly wouldn't in production.) Still, by walking through the process you can see how it works.- Download the custom ansible deployment role:
curl -Lk https://github.com/vnogin/Ansible-role-for-baremetal-node-provision/archive/master.tar.gz | tar xz -C /opt/stack/ironic-staging-drivers/ironic_staging_drivers/ansible/playbooks/ --strip-components 1 - Next, create an inventory file for the bare metal server(s) that need to be provisioned:
# echo "--- server1: ipa_kernel_url: "http://172.16.166.14:8080/ansible_ubuntu.vmlinuz" ipa_ramdisk_url: "http://172.16.166.14:8080/ansible_ubuntu.initramfs" uuid: 00000000-0000-0000-0000-000000000001 driver_info: power: ipmi_username: IPMI_USERNAME ipmi_address: IPMI_IP_ADDRESS ipmi_password: IPMI_PASSWORD ansible_deploy_playbook: deploy_custom.yaml nics: - mac: 00:25:90:a6:13:ea driver: pxe_ipmitool_ansible ipv4_address: 172.16.166.22 properties: cpu_arch: x86_64 ram: 16000 disk_size: 60 cpus: 8 name: server1 instance_info: image_source: "http://172.16.166.14:8080/user_image.qcow2"" > /opt/stack/bifrost/playbooks/inventory/baremetal.yml
As you can see the above we have added all required information for bare-metal node provisioning using IPMI. If needed you can add information about various number of bare-metal servers here and all of them will be enrolled and deployed later.
# export BIFROST_INVENTORY_SOURCE=/opt/stack/bifrost/playbooks/inventory/baremetal.yml - Finally, you'll need to build a ramdisk for the Ironic ansible deploy driver and create a deploy image using DIB (disk image builder). Start by creating an RSA key that will be used for connectivity from the Ironic ansible driver to the provisioning bare metal host:
# su - ironic # ssh-keygen # exit
- Next set environment variables for DIB:
# export ELEMENTS_PATH=/opt/stack/ironic-staging-drivers/imagebuild # export DIB_DEV_USER_USERNAME=ansible # export DIB_DEV_USER_AUTHORIZED_KEYS=/home/ironic/.ssh/id_rsa.pub # export DIB_DEV_USER_PASSWORD=secret # export DIB_DEV_USER_PWDLESS_SUDO=yes
- Install DIB:
# cd /opt/stack/diskimage-builder/ # pip install .
- Create the bootstrap and deployment images using DIB, and move them to the web folder:
# disk-image-create -a amd64 -t qcow2 ubuntu baremetal grub2 ironic-ansible -o ansible_ubuntu # mv ansible_ubuntu.vmlinuz ansible_ubuntu.initramfs /httpboot/ # disk-image-create -a amd64 -t qcow2 ubuntu baremetal grub2 devuser cloud-init-nocloud -o user_image # mv user_image.qcow2 /httpboot/
- Fix file permissions:
# cd /httpboot/ # chown ironic:ironic *
- Now we can enroll anddeploy our bare metal node using ansible:
# cd /opt/stack/bifrost/playbooks/ # ansible-playbook -vvvv -i inventory/bifrost_inventory.py enroll-dynamic.yaml
Wait for the provisioning state to read "available", as a bare metal server needs to cycle through a few states and could be cleared, if needed. During the enrollment procedure, the node can be cleared by the shred command. This process does take a significant amount of time time, so you can disable or fine tune it in the Ironic configuration (as you saw above where we enabled it). - Now we can start the actual deployment procedure:
# ansible-playbook -vvvv -i inventory/bifrost_inventory.py deploy-dynamic.yamlIf deployment completes properly, you will see the provisioning state for your server as "active" in the Ironic node-list.+--------------------------------------------------------------+---------+--------------------+-----------------+-------------------------+------------------+ | UUID | Name | Instance UUID | Power State | Provisioning State | Maintenance | +--------------------------------------------------------------+---------+--------------------+-----------------+-------------------------+------------------+ | 00000000-0000-0000-0000-000000000001 | server1| None | power on | active | False | +--------------------------------------------------------------+---------+--------------------+-----------------+-------------------------+------------------+
Conclusion
As you can see, bare metal server provisioning isn't such a complicated procedure. Using the Ironic standalone server (bifrost) with the Ironic ansible driver, you can easily develop a custom ansible role for your specific deployment case and simultaneously deploy any number of bare metal servers in automation mode.I want to say thank you to Pavlo Shchelokovskyy and Ihor Pukha for your help and support throughout the entire process. I am very grateful to you guys.
