With the Fargo release we introduce a new architecture which moves the read cache from the Volume Driver to the ALBA backend. I already explained the new backend concepts in a previous blog post but I would also like to shed some light on the various reasons why we took the decision to move the read cache to ALBA. An overview:
Performance is absolutely the main reason why we decided to move the read cache layer to ALBA. It allows us to remove a big performance bottleneck: locks. When the Volume Driver was in charge of the read cache, we used a hash based upon the volume ID and the LBA to find where the data was stored on the SSD of the Storage Router. When new data was added to the cache – on every write – old data in the cache had to be overwritten. In order to evict data from the cache a linked list was used to track the LRU (Least Recently Used) data. Consequently we had to lock the whole SSD for a while. The lock was required as the hash table (volume ID + LBA) and the linked list had to be updated simultaneously. This write lock also causes delay for read requests as the lock prevents data to be safely read. Basically, in order to increase the performance we had to move towards a lockless read cache where data isn’t updated in place.
This is where ALBA comes in. The ALBA backend doesn’t update data in place but uses a log-structured approach where data is always appended. As ALBA stores chunks of the SCOs, writes are consecutive and large in size. This greatly improves the write bandwidth to the SSDs. ALBA also allows to align cores with the ASD processes and underlying SSDs. By making the whole all-flash ALBA backend core aligned, the overhead of process switching can be minimised. Basically all operations on flash are now asynchronous, core aligned and lockless. All these changes allow Open vStorage to be the fastest distributed block store.
Lower impact of an SSD failure
By moving the read cache to the ALBA backend the impact of an SSD failure is much lower. ALBA allows to perform erasure coding across all SSDs of all nodes in the rack or datacenter. This means the read cache is now distributed and the impact of an SSD failure is limited as only a fraction of the cache is lost. So in case a single SSD fails, there is no reason to go the HDD based capacity backend as the reads can still be fulfilled based upon other fragments of the data which are still cached.
Always hot cache
While Open vStorage has always been capable of supporting live migration, we noticed that with previous versions of the architecture the migrate wasn’t always successful due to the cold cache on the new host. By using the new distributed cache approach, we now have have an always hot cache even in case of (live) migrations.
We hope the above reasons proof that we took the right decision by moving the read cache to ALBA backend. Want to see how you configure the ALBA read cache, check out this GeoScale demo.
As promised in our latest release update we would do more frequent releases. Et voilà , today we release a new alpha version of the upcoming GA release. If possible, we will provide new versions from now on as an update so that you don’t have to reinstall your Open vStorage environment. As this new version is the first one with the update/upgrade functionality, there is no update possible between alpha 1 and alpha 2.
What is new in Open vStorage 2.2 alpha 2:
Upgrade functionality: Under the Administration section you can check for updates of the Framework, the Volumedriver and the Open vStorage Backend and apply them. For the moment an update might require all VM’s to be shutdown.
Support for non-identical hardware layouts: You can now mix hardware which doesn’t have the same amount of SSDs or PCI-e flash cards. When extending a vPool to a new Storage Router you can select which devices to use as cache.
The Backend policy which defines how SCOs are stored on the Backend can now be changed. The wizard is straightforward in case you want to set for example 3-way replication.
Rebalancing of the Open vStorage Backend, moving data from disks which are almost full to new disks to make sure all disks are evenly used, is now a service which can be enabled and disabled.
Audit trails are no longer stored in the model but in a log.
ASD raises time out or stops under heavy load.
Extening a vPool to a new Storage Router doesn’t require the existing vMachines on the vPool to be stopped.
FailOverCache does not exit but hangs in accept in some cases.
Remove vpool raises cluster not reachable exception.
Add logrotate entry for /var/log/arakoon/*/*.log
Error in vMachine detail (refreshVDisks is not found).
Arakoon 1.8.4 rpm doesn’t work on CentOS7.
Arakoon catchup is quite slow.
The combination Openstack with multiple vPools and live migration does not properly work in some cases.
During the summer the Open vStorage Team has worked very hard. With this new release we can proudly present:
Feature complete OpenStack Cinder Plugin: our Cinder Plugin has been improved and meets the minimum features required to be certified by the OpenStack community.
Flexible cache layout: in 1.5.0 you have the capability to easily configure multiple SSD devices for the different caching purposes. During the setup you can choose which SSD’s to partition and then later when creating a vPool you can select which caching device should be used for read, write and writecache protection. Meaning these can from now on be spread over different or consolidated into the same SSD device depending on the available hardware and needs.
User management: an admin can now create more user which have access to the GUI.
Framework performance: a lot of work has been put into improving the performance when a lot of vDisks and vMachines are created. Improvements upto 50% has been reached in some cases.
Improved API security by means of implementing OAuth2 authentication. A rate-limit has also been imposed on API calls to prevent brute force attacks.
Fixed bugs and small items:
GUI now prevents creation of vPools with a capital letter.
Implemented recommendation for a security exploit on elasticsearch 1.1.1.
Fix for validation of vPools being stuck on validating.
Protection against reusing vPool names towards the same backend.
Fix for the Storage Router online/offline detection which failed when openstack was also installed.
Next, we also took the first step towards supporting other OS than Ubuntu (RedHat/Centos). We have created an rpm version of our volumedriver and arakoon packages. These are tested on “Linux Centos7 3.10.0-123.el7.x86_64” and can be downloaded from our packages server. This completes a first important step towards getting Open vStorage RedHat/CentOS compatible.