Docker and persistent Open vStorage volumes

Docker, the open-source container platform, is currently one of the hottest projects in the IT infrastructure business. With support of some of the world’s leading companies such as PayPal, Ebay, General Electric and many more, it is quickly becoming a cornerstone of any large deployment. Next, it also introduces a paradigm shift in how administrators see servers and applications.

Pets vs. cattle

In the past servers were treated like dogs and cats or any family pet: you give it a cute name, make sure it is in optimal condition, take care of it when it is sick, … With VMs a shift already occurred: names became more general like WebServer012 but keeping the VM healthy was still a priority for administrators. With Docker, VMs are decomposed into a sprawl of individual, clearly, well-defined applications. Sometimes there can even be multiple instances of the same application running at the same time. With thousands of containerized applications running on a single platform, it becomes impossible to treat these applications as pets but instead they are treated as cattle: they get an ID, when having issues they are taken off-line, terminated, and replaced.

Docker Storage

The original idea behind Docker was that containers would be stateless and hence didn’t need persistent storage. But over the years the insight has grown that also some applications and hence containers require persistent storage. Since the Docker platforms at large companies are housing thousands of containers, the required storage is also significant. Typically these platforms also span multiple locations or even clouds. Storage across locations and clouds is the sweet spot of the Open vStorage feature set. In order to offer distributed, persistent storage to containers, the Open vStorage team created a Docker plugin on top of the Open vStorage Edge, our lightweight block device. Note that the Docker plugin is part of the Open vStorage Enterprise Edition.

Open vStorage and Docker

Using Open vStorage to provision volumes for Docker is easy and straightforward thanks to Docker’s volume plugin system. To show how easy it is to create a volume for a container, I will give you the steps to run Minio, a minimal , open-source object store, on top of a vDisk.

First install the Open vStorage Docker plugin and the necessary packages on the compute host running Docker:
apt-get install libovsvolumedriver-ee blktap-openvstorage-ee-utils blktap-dkms volumedriver-ee-docker-plugin

Configure the configuration of the plugin by updating /etc/volumedriver-ee-docker-plugin/config.toml

[volumedriver]
hostname="IP"
port=26203
protocol="tcp"
username="root"
password="rooter"

Change the IP and port to the IP on which the vPool is exposed on the Storage Router you want to connect to (see Storage Router detail page).

Start the plugin service
systemctl start volumedriver-ee-docker-plugin.service

Create the Minio container and attach a disk for the data (minio_export) and one for the config (minio_config)

docker run --volume-driver=ovs-volumedriver-ee -p 9000:9000 --name minio \
-v minio_export:/export \
-v minio_config:/root/.minio \
minio/minio server /export

That is it. You now have a Minio object store running which stores its data on Open vStorage.

PS. Want to see more? Check the “Docker fun across Amazon Google and Packet.net”-video

Playing with Open vStorage and Docker

I was looking for a way to play with Open vStorage on my laptop with as ultimate goal letting people easily experience Open vStorage without having to rack a whole cluster. The idea of running Open vStorage inside Docker, the open container platform, sounded pretty cool so I accepted the challenge to create a hyperconverged setup with docker images. In this blog post I will show you how to build a cluster of 2 nodes running Open vStorage on a single server or laptop. You could also use this dockerised approach if you want to deploy Open vStorage without having to reinstall the server or laptop after playing around with Open vStorage.

As I’m running Windows on my laptop, I started with creating a virtual machine in VirtualBox. The Virtual Machine I created has 4 dynamically allocated disks (OS – 8GB, Cache/DB/scrubber – 100GB and 2 backend disks of 100GB). The VM has 1 network card which is bridged. The exact details of the VM can be found below.
2016-03-31_9-58-42

The steps to run Open vStorage in a Docker container:
Install Ubuntu 14.04 (http://www.ubuntu.com/download/server) in the VM.
Next install Docker:

sudo apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
echo 'deb https://apt.dockerproject.org/repo ubuntu-trusty main' | sudo tee /etc/apt/sources.list.d/docker.list
sudo apt-get update -qq
sudo apt-get purge lxc-docker
sudo apt-get install -y docker-engine

As I wanted to build a multi-container setup, the Open vStorage containers must be able to communicate with each other over the network. To achieve this I decided to use Weave, the open-source multi-host Docker networking project.

sudo curl -L git.io/weave -o /usr/local/bin/weave
sudo chmod a+x /usr/local/bin/weave

As a next step, download the Open vStorage Docker template and turn it into a Docker image. The Docker image has the latest Open vStorage packages already preloaded for your convenience. Currently the docker image is only hosted on GitHub but it will be pushed to the Docker Hub later on.

wget https://github.com/openvstorage/docker-tools/releases/download/20160325/ovshc_unstable_img.tar.gz
gzip -dc ovshc_unstable_img.tar.gz | sudo docker load

To make things easier we have an ovscluster setup script. This script uses the Docker CLI to create an Open vStorage container based upon the Docker image, joins the container to the Open vStorage cluster and configures the default settings of the container. You can download the cluster setup script from GithHub:

wget https://github.com/openvstorage/docker-tools/raw/master/hyperconverged/ovscluster.sh
chmod +x ovscluster.sh

Get the Weave network up and running. Naturally a free IP range for the weave network is required. In this case I’m

using 10.250.0.0/16.
sudo weave launch --ipalloc-range 10.250.0.0/16

Now use the cluster setup script to create the cluster and add the first Open vStorage container. I’m using ovshc1 as hostname for the container and 10.250.0.1 as IP.

sudo ./ovscluster.sh create ovshc1 10.250.0.1/16

Once the first host of the Open vStorage cluster is created, you will get a bash shell inside the container. Do not exit the shell as otherwise the container will be removed.
2016-03-31_11-02-55
That is all it takes to get Open vStorage container up and running!

Time to create a Backend, vPool and vDisk
Since the Open vStorage container is now fully functional, it is time to do something with it: create a vDisks on it. But, fefore the container can handle vDisks, it needs a backend to store the the actual data. As backend I’ll create an ALBA backend, the native backend, in Open vStorage on top of 2 additional Virtual Machine disks.

Surf with your browser to the Open vStorage GUI at https:// and log in with the default login and password (admin/admin).

As a first step it is required to assign a read, write, DB and scrubber role to at least one of the disks of the container. In production always use an SSD for the DB and cache roles. The scrubber can be on a SATA drive. Select Storage Routers from the top menu. Select ovshc1.weave.local to see the details of the Storage Router and select the Physical disk tab. Select the gear icon of the first 100GB disk.
2016-03-31_11-06-33
Assign the read, write, DB and scrubbing role and click Finish.
2016-03-31_11-09-13
Wait until the partitions are configured (this could take upto 30 seconds).
2016-03-31_11-11-10

Once these basic roles are assigned, it is time to create a new backend and assign the 2 additional virtual machine disks as ASDs (ALBA Storage Daemon) to that backend.

Select Backends from the top menu and click Add Backend. Give the new backend a name and click Finish.
2016-03-31_11-13-01
Wait until the newly created backend becomes available (status will be green) and you see the disks of ovshc1.weave.local.
2016-03-31_11-22-00
Initialize the first 2 disks by clicking the arrow and select Initialize.
2016-03-31_11-23-59
Wait until the status of both disks turns dark blue and claim them for the backend.
2016-03-31_11-26-46
Select the presets tab and select Add preset.
2016-03-31_11-30-56
Give the new preset a name and select advanced settings and indicate you understand the risks of specifying a custom policy. Add a policy with k=1, m=0, c=1 and x=1 (store data on a single disk without any parity fragments on other disks – NEVER user this policy in production!).
2016-03-31_11-49-58
Click Next and Finish.

On top of that backend a vpool is required (like a datastore in VMware) before a vDisk can be created. Select vPools from the top menu and click Add new vPool. Give the vpool a name and click the Reload button to load the backend. Select the newly created preset and click Next.
2016-03-31_11-53-49

Leave the settings on the second screen of the wizard to the default ones and click Next. Set the read cache to 10GB and the write cache to 10GB. Click Next, Next and Finish to create the vPool.

If all goes well, on ovshc1, you will have one disk assigned for the DB/Cache/Scrubber roles for internal use, two disks assigned to ALBA, and one vPool exported for consumption by the cluster.

root@ovshc1:~# df -h
/dev/sdb1 63G 1.1G 59G 2% /mnt/hdd1
/dev/sdc1 64G 42M 64G 1% /mnt/alba-asd/IoeSU0gwIFk591fX9MqZ15CuIx8uMWV2
/dev/sdd1 64G 42M 64G 1% /mnt/alba-asd/Acxydl4BszLlppGKwvWCL2c2Jw7dTWW0
601a3b34-9426-4dc5-9c35-84fac81b42b6 64T 0 64T 0% /exports/vpool

The same vpool can be seen on the VM as follows:

$> df -h
601a3b34-9426-4dc5-9c35-84fac81b42b6 64T 0 64T 0% /mnt/ovs/vpool

Time to create that vDisk. Open a new session to the VM and goto /mnt/ovs/vpool (replace vpool by the name of your vpool). To see if the vPool is fully functioning create .raw disk, the raw disk format used by KVM, and put some load on the disk with fio.Check the Open vStorage GUI to see its perfromance!

sudo truncate -s 123G /mnt/ovs/vpool/diskname.raw
sudo apt-get install fio
fio -name=temp-fio --bs=4k --ioengine=libaio --iodepth=64 --size=1G --direct=1 --rw=randread --numjobs=12 --time_based --runtime=60 --group_reporting --filename=/mnt/ovs/vpool/diskname.raw

Adding a second host to the cluster
To add a second Open vStorage container to the cluster, create another VM with the same specs. Install Ubuntu, Docker and Weave. Just like for the first container download the Docker image and cluster script.

sudo apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
echo 'deb https://apt.dockerproject.org/repo ubuntu-trusty main' | sudo tee /etc/apt/sources.list.d/docker.list
sudo apt-get update -qq
sudo apt-get purge lxc-docker
sudo apt-get install -y docker-engine
sudo curl -L git.io/weave -o /usr/local/bin/weave
sudo chmod a+x /usr/local/bin/weave
wget https://github.com/openvstorage/docker-tools/releases/download/20160325/ovshc_unstable_img.tar.gz
gzip -dc ovshc_unstable_img.tar.gz | docker load
wget https://github.com/openvstorage/docker-tools/raw/master/hyperconverged/ovscluster.sh
chmod +x ovscluster.sh

To launch and join a second Open vStorage container execute:

sudo weave launch --ipalloc-range 10.250.0.0/16
sudo ./ovscluster.sh join ovshc2 10.250.0.2/16

2016: Cheers to the New Year!

The past year has been a remarkable one for Open vStorage. We did 2 US roadshows, attended a successful OpenStack summit in Vancouver, moved and open-sourced all of Open vStorage on GitHub and released a lot of new functionality (our own hyperconverged backend, detailed tuning & caching parameters for vDisks, a certified OpenStack Cinder plugin, remote support, CentOS7, …). The year also ended with a bang as customers were trying to beat each other’s top 4k IOPS results.

photo_2015-12-14_13-21-09

While it might look hard to beat the success of 2015, the Open vStorage Team is confident that 2016 will be even more fruitful. Feature wise some highly anticipated features will see the light in Q1: improved QEMU integration, block devices (blktap support), docker support (Flocker), iSCSI disks, replication, support for an all-flash backend, … Next to these product features, the team will open its kimono and discuss the Open vStorage internals in detail in our new GitBook documentation. In order to be more in spirit with the open source community, blueprints of upcoming features will also be published as much as possible to the GitHub repo. A first example is the move of the Open vStorage config files to etcd. Finally, based upon the projects of partners and customers in the pipeline, 2016 will be unrivaled. So far a 39 node cluster (nearly 125TB of flash alone) is being deployed in production and multiple datacenters (US, Europe, Asia) are being transformed to use Open vStorage as storage platform.

Cheers to the New Year. May it be a memorable one.