Open vStorage opens up its API kimono

oai
With the Fargo release Open vStorage opens up its API kimono. In earlier versions of Open vStorage the API was something that was well hidden in the documentation section. As a result many of our integration partners had questions on how to use the API, what exactly was possible with the API or for example what the required parameters were to take a snapshot. It was clear for everyone that we had to give the API some more spotlight.

Why an API?

An API is especially important because it dictates how the developers of these integrators can create new apps, websites and services on top of the Open vStorage storage solution. A hosting provider has for example built an OpenStack-like GUI for its KVM + Open vStorage cluster. They create vDisks on Open vStorage directly from their GUI, take snapshots and even scrub the vDisks on demand. They are consuming every aspect of our API. During this integration it became clear that keeping our API documentation up to date was a challenge. The idea grew to make the API self-describing and browsable.

Open API

APIs come in many forms but some standards are crystallizing. Open vStorage follows the Open API specification (OAI). This specification is supported by some of the big names in the IT industry such as Google, Microsoft, IBM and PayPal. It also means some great open-source tools can be leveraged such as NSwag and Swagger UI. NSwag is a Swagger API toolchain for .NET, Web API and TypeScript (jQuery, AngularJS, Angular 2, Aurelia, KnockoutJS, and more). Swagger UI is a tool that dynamically generates beautiful documentation and a sandbox to play with straight from the browser.

Browsable API

To explore the Open vStorage API, download the Swagger UI , unzip the archive and serve the dist folder from either your file system or a web server.

Next, enter in the textbox https://[ip of the GUI]/api/swagger.json and press enter.

open-vstorage-api

You can now browse through the API. As an example you can verify which parameters are required to move a vDisk between Storage Routers.

open-vstorage-api-move-vdisk

One small, but important remark. Currently Swagger-UI doesn’t support OAuth2 yet. This means you can browse the API but you can’t execute API requests as these need to be authenticated.

Eugene Release

To start the new year with a bang, the Open vStorage Team is proud to release Eugene:

The highlights of this release are:

Policy Update
Open vStorage enables you to actively add, remove and update policies for specific ALBA backend presets. Updating active policies might result in Open vStorage to automatically rewrite data fragments.

ALBA Backend Encryption
When configuring a backend presets, AES-256 encryption algorithms can be selected.

Failure Domain
A Failure Domain is a logical grouping of Storage Routers. The Distributed Transaction Log (DTL) and MetaDataServer (MDS) for Storage Router groups can be defined in the same Failure Domain or in a Backup Failure domain. When the DTL and MDS are defined in a Backup Failure Domain, data loss in case of a non-functioning Failure Domain is prevented. Defining the DTL and MDS in a backup Failure Domain requires low latency network connections.

Distributed Scrubber
Snapshots which are out of retention period are indicated as garbage and removed by the Scrubber. With the Distributed Scrubber functionality you can now decide to run the actual scrubbing process away from the host that holds the volume. This way, hosts that are running Virtual Machines do not experience any performance hit when the snapshots of those Virtual Machines are scrubbed.

Scrubbing Parent vDisks
Open vStorage allows to create clones of vDisks. The maximal depth of the clone tree is limited to 255. When a clone is created, scrubbing is still applied to the actual parent of the clone.

New API calls
Following API’s are added:

  • vDisk templates (set and create from template)
  • Create a vDisk (name, size)
  • Clone a vMachine from a vMachine Snapshot
  • Delete a vDisk
  • Delete a vMachine snapshot

These API calls are not exposed in the GUI.

Removal of the community restrictions
The ALBA backend is no longer restricted and you are no longer required to apply for a community license to use ALBA. The cluster needs to be registered within 30 days otherwise the GUI will stop working until the cluster is registered.

Remove Node
Open vStorage allows for nodes to be removed from the Open vStorage Cluster. With this functionality you can remove any node and scale your storage cluster along with your changing storage requirements. Both active and broken nodes can be consistently removed from the cluster.

Some smaller Feature Requests were added also:

  • Removal of the GCC dependency.
  • Option to label a manual snapshot as ‘sticky’ so it doesn’t get removed by the automated snapshot cleanup.
  • Allow stealing of a volume when no Hypervisor Management Center is configured and the node rowning the volume is down.
  • Set_config_params for vDisk no longer requires the old config.
  • Automatically reconfigure the DTL when DTL is degraded.
  • Automatic triggering of a repair job when an ASD is down for 15 minutes.
  • ALBA is independent of broadcasting.
  • Encryption of the ALBA socket communication.
  • New Arakoon client (pyarakoon).

Following are the most important bug fixes in the Eugene release:

  • Fix for various issues when first node is down.
  • “An error occurred while configuring the partition” while trying to assign DB role to a partition.
  • Cached list not updated correctly.
  • Celery workers are unable to start.
  • Nvme drives are not correctly detected.
  • Volume restart fails due to failure while clearing the DTL.
  • Arakoon configs not correct on 4th node.
  • Bad MDS Slave placement.
  • DB role is required on every node running a vPool but isn’t mandatory.
  • Exception in tick crashes the ovs-scheduled-task service.
  • OVS-extensions is very chatty.
  • Voldrv python client hangs if node1 is down.
  • Bad logic to decide when to create extra alba namespace hosts.
  • Timeout of CHAINED ensure single decorator is not high enough.
  • Possibly wrong master selected during “ovs setup demote”.
  • Possible race condition when adding nodes to cluster.

Open vStorage 1.6

Just before we start the holiday season at the CloudFounder’s HQ, we wanted to give the community an early Christmas present: Open vStorage version 1.6:
So what is included in this release:
  • Audit trails: In version 1.5 we added the option to add multiple users. In the 1.6 version we keep a log of all the actions performed through the Open vStorage API for 30 days. Not only user actions are included by also volume driver actions. This will allow us to f.e. track who  creates or deletes a vPool.
  • Improved usability: With this release we wanted to create a better user experience. Some examples, when a vPool is added we no longer ask for ports and Cinder gets now automatically downloaded, the API now returns paging metadata and we revamped the GUI so it doesn’t look as empty as it used to. So basically a whole lot of small features which make life easier.
As you can see we have limited the amount of new features in this release but focused more on bug-fixing:
  • Fix for OVS workers being stuck in rare cases.
  • Fix for a bug which causes the GUI and other API calls to take a very long time to complete in case a host goes down. There was a manual workaround but the current fix automates the process.
  • Memcache can now be restarted without the need to restart the volume driver process.
  • Fix for the slow responding of the SNMP server in case of many objects.
  • The volumedriver logs are now compressed and correctly rotated.
  • Fix for issues with live migration on OpenStack.
  • Faster loading of the vMachine Detail page.
  • Fix for incorrect amount of vDisks under Storage Routers on the Dashboard.
  • Better layout for Firefox.
  • The setup script incorrectly calculates the machine_id on some OSes.
  • Fix for xforwarding being broken.
  • Fix for failing addition of a vPool on slow environments.

Enjoy the holidays!