Open vStorage featured in OpenStack Object Storage (Swift) Essentials

swiftbook Are you still looking for a nice present to put under the Christmas tree? A good suggestion is OpenStack Object Storage (Swift) Essentials as it features Open vStorage as one of the use cases of OpenStack Swift. The book has been written by Swift experts Amar Kapadia (Sr. Director Product Marketing at Mirantis), Kris Rajana (CEO at Biarca) and Sreedhar Varma (Director – Storage Technologies and Software Development at Vedams Software).

2015, the Open vStorage predictions

New Year Wallpaper.The end of 2014 is near so it is time to look forward and see what 2015 will have to offer. Some people say there’s no point in making predictions and that it’s not worth speculating because nothing is set in stone and things change all the time in storage land. Allow us to prove these people wrong by sharing our 2015 storage predictions*:

Acceleration of (hyper-)converged platforms
Converged platforms are here to stay. Converged solutions are even the fastest growing segment for large storage vendors. But it will be players like Open vStorage who will really break through in 2015. Hyperconverged solutions showed that there is an alternative to expensive SAN or all-flash solutions by adding a software based caching layer to the host. Alas, these overhyped hyperconverged solutions are even more expensive per TB storage than an already expensive all-flash array. In 2015 we will see more solutions which unite the good of the hyperconverged appliance and the converged platform but at a significantly lower cost. Storage solutions that will be extremely hot and prove to be future proof will have to have following characteristics:

  • Caching inside the (hypervisor) host: caching on SSD or PCIe flash should be done inside the host and not a couple of hops down the network.
  • Scalability: all-flash will continue to be a waste of money due to the huge cost of flash. It is better to go for a Tiered solution: Tier 1 on flash, Tier 2 on scalable, cheap (object) storage. In case the Tier 1 and Tier 2 storage are inside the same appliance (hyperconverged), your scalability and flexibility will be limited. A much better solution is to keep the 2 Tiers completely separate in a different set of appliances but managed by the same software layer.
  • Open and programmatically: storage silo’s should in 2015 be something of the past. Integration and openness should be key. Automation will be one of the hottest and most important features of a storage solution.

It should not come as a surprise that Open vStorage checks all of the above requirements.

OpenStack will dominate the cloud in 2015
This is probably the most evident prediction. During the OpenStack conference in Paris it was very clear that OpenStack will dominate the cloud the next few years. In 2014 some new kids showed support for OpenStack such as VMware (they understand that hypervisor software is now a commodity and that the data center control plane has become the high-margin battle arena). With VMware releasing their own OpenStack distribution the OpenStack distribution battlefield will be crowded in 2015. We have RedHat, Ubuntu, Mirantis, HP, VMware and many more so it is safe to say that some consolidation will happen in this area.
A new OpenStack battlefield that will emerge in 2015 will be around OpenStack storage. Currently this area is being dominated by the traditional arrays but as the software-defined storage solutions gain traction, solutions such as Open vStorage will grab a huge market share from these traditional vendors. They can compete with these SANs and all-flash arrays as they offer the same features and have the benefit of a much lower cost and TCO. While they maybe not top the total revenue achieved by the big vendors, they will definitely seize a large share of the OpenStack storage market.
If we add the fact that the Chinese government is promoting OpenStack and open-source in general, you can bet your Christmas bonus on the fact that open-source (OpenStack) storage projects (Swift, Open vStorage, …) will be booming next year. These projects will get a lot of support from Chinese companies both in adoption and development. It will be essential for traditional high-tech companies and countries not to miss the boat as once it has left the harbor it will be very hard to catch up.

New markets for object storage vendors
2015 will be the year where object storage will break out of its niche market of large video or object repositories. This has been said for many years now but 2015 will be THE year as many companies have started to realize which benefits they achieved by implementing their first object storage projects. The next target for these enterprises is to make better use of their current object storage solution. Changing all of their legacy code will not happen in 2015 as this might impact their business. Solutions where they don’t have to change their existing code base and still benefit from the cost saving of object storage will be selected. Open vStorage is one of those solutions but we are pretty sure other solutions like for example storage gateways to object storage will flourish in 2015.
Another reason why object storage vendors will enter new markets is because currently too many players are after the same customer base. This means that if they want to keep growing and provide ROI for the venture capital invested, new markets will definitely need to be addressed. The 15-25 billion dollar SAN market is a logical market to address. But entering this market will not be a bed of roses as object storage vendors have no experience in this highly-competitive market or sometimes not even the right sales competencies and knowledge. They will have to look for partnerships with companies such as CloudFounders who are seasoned in this area.

Seagate Kinetic drives
The Kinetic drives are the most exciting, fundamental change in the storage industry in several years. These drives went GA at the end of 2014 but in 2015 we will gradually see new applications and solutions who make use of this technology. With these IP drives, you will, for the first time, be able to manage storage as a scalable pool of disks. Open vStorage will support the Kinetic drives as Tier 2 backend. This means Virtual Machine will have their hot data inside the host on flash and their cold data on a pool of Kinetic drives.

* We will look back on this post at the end of 2015 to see how good we scored.

Open vStorage is hot!

Open vStorage is hot! You can find the most important video’s, podcasts and articles of the last few weeks here:

  • OpenStack Online Meetup: How does OpenStack Swift play with Open vStorage? Podcast with Swift PTL John Dickinson and our own Wim Provoost as they discuss OpenStack storage and how Swift and Open vStorage, 2 open-source projects, work together to shake the OpenStack eco-system.
  • The Register: Belgian upstart to ‘bridge gap’ between object, block storage. Or that’s the plan at least . Learn more about the latest developments in the Open vStorage project: support for Seagate Kinetic drives. Open vStorage turns a pools of these disks into a storage solutions which has the  features of a high-end SAN (performance, zero-copy snapshots, thin cloning) but is also scale-out and low-cost, like an object storage solution.
  • Open vStorage and OpenStack – the Storage Switzerland take. Podcast with George Crump, founder and lead analyst at Storage Switzerland and Wim Provoost, Product Manager of Open vStorage. They discuss the different storage  projects in OpenStack and how the Open vStorage Cinder plugin bridges the gap between Swift and Cinder.
  • OpenStack® Summit: CloudFounders Open vStorage: CLoudFounders’ CTO Stefaan Aelbrecht explains to HP what Open vStorage is about and where it comes from.

Online Meetup: How does OpenStack Swift play with Open vStorage?

On Tuesday November 25th (09:00 AM PST), you can join the OpenStack Online Meetup community for a conversation between Swift’s Program Technical Lead John Dickinson and Wim Provoost of Open vStorage as they discuss OpenStack storage and how their respective projects work together. They both had presented before and those were some of the most popular meetups. You can see Wim’s episode here. This time they will be talking with both together about how Open vStorage can provide a block interface on top of OpenStack Swift.

You can register here to be part of the session.

Open vStorage and OpenStack – the Storage Switzerland take

Open vStorage is HyperFast storage for OpenStack Virtual Machines. SSDs and PCIe Flash cards inside the hosts are used as Tier 1 storage while OpenStack Swift, Ceph, or any S3 compatible object storage solution is used as Tier 2 storage. George Crump and Charlie Hodges from Storage Switzerland and our own Wim Provoost discussed the OpenStack Open vStorage relation during a 12 minute podcast.

Some of the questions that are raised and answered in this podcast:

  • What is Open vStorage?
  • Which problems is Open vStorage trying to solve?
  • Why does Open vStorage focusses on the OpenStack market?
  • Is Open vStorage trying to replace Ceph as OpenStack storage?

OpenStack Online Meetup on Swift, VM Storage and OpenvStorage

Wim Provoost, Product Manager of Open vStorage, was asked to host a live webinar for OpenStack Online Meetup. OpenStack Online Meetup is the #1 online community for OpenStack contributors and users. During their weekly Google Hangout sessions they feature various OpenStack speakers. The sessions are intended for technical OpenStackers and tend to deep-dive into technology.

The Open vStorage talk discusses how you can use OpenStack Swift, the Object Storage project within OpenStack, as primary storage for a Virtual Machine environment. In case you missed the live session, you can watch the recorded version below.

Like maintaining a single storage platform isn’t hard enough ….

In our last blog post we discussed the Open vStorage Cinder Plugin. We had some people ask whether this means that you can now use Swift directly as primary storage for Virtual Machines. The short answer is: YES!

A traditional OpenStack setup looks like the below reference architecture.
Slide4

You have Nova which provisions the VM. Cinder is providing block storage and Glance provides the image to deploy the VM. Swift is also used but only as repository for images and backups. You can use Cinder natively but this isn’t high available and isn’t an enterprise grade solution so you need ‘something distributed’ to actually store the blocks. This can be a SAN (Dell, HP, EMC, …) or a Ceph distributed storage platform. This means you are maintaining 2 storage platforms: 1 for object storage (Swift) and 1 for block storage (SAN, Ceph, …).

Slide5

Maintaining one storage platform is already hard enough so why would you maintain two? This is where Open vStorage comes to the rescue. It allows to turn OpenStack Swift into block storage. This means you can now use Swift both for object and for block storage. The only thing you need is to install Open vStorage and configure its Cinder Plugin. When you create a volume in OpenStack, the Cinder API will call the Open vStorage API to create a disk. The same happens when a snapshot is created. On top, Open vStorage also brings VM-centric storage management to OpenStack.

Interested in seeing how the Cinder Plugin works? Check the demo video below:

The source code for the plugin can be found here.
The steps to set up an environment can be found here.

Webscale 2.0

As a Product Manager I’m very often on challenging calls with potential users of Open vStorage and one of the questions that comes back on almost every call is:

How scalable is Open vStorage?

It is a question that is easy to answer: extremely scalable. Open vStorage is built from the ground up to support environments which have 100+ hosts. It is designed to be used in large datacenters as primary storage platform for all types of Virtual Machine workload. I’m aware that the term scalable is a bit biased and can have different meanings. Did the enquirer mean storage capacity scalability or performance scalability. Well, Open vStorage scales both ways. For the storage capacity, the scalability is mostly limited to the selected backend. For example, with Swift as storage backend of a vPool, you can almost infinitely add disks or storage nodes to enlarge the storage pool. Swift is after all designed with massive scalability as main development mantra and it has shown this quality in production environments of Disney and Rackspace amongst many others.
Performance scalability is also not a problem. Adding more hosts running the Open vStorage software will linearly scale the performance. As each hosts has one or more SSDs or PCIe Flash cards on board, the addition of every host to the Open vStorage environment increases the data that can be stored in the cache.

Does that mean Open vStorage is webscale?

No, unlike other hyperconverged storage solutions, we are not webscale. We are webscale 2.0. The reason why we can call Open vStorage webscale 2.0 is because it decoupled the storage scalability from the performance scalability. This allows for asymmetric architectures. It makes no sense having to add more storage capacity in order to improve the performance of your storage solution. Open vStorage is the only solution which allows to independently scale performance and capacity at a massive scale. Not only is Open vStorage tailored to the needs of large environments with petabytes of data and a battery of compute power but it can also address the needs of a typical enterprise. Whether that typical enterprise has lots of data with limited compute power or vice versa, Open vStorage is up for the job.

White Paper – Open vStorage + Swift, a killer VM storage combo

OpenStack Swift, probably the most used Object Storage for private cloud implementations, offers features which are very appealing for building a distributed storage solution. Alas, issues such as eventual consistency and the fact that hypervisors require block storage and can’t work with Object Storage, make it unsuitable for primary Virtual Machine storage. Open vStorage is the solution to turn Swift into a block device for Virtual Machines.

The combination of Open vStorage + Swift offers great performance due to aggressive caching inside the Host, a unified namespace and many VM-centric features. Open vStorage + Swift is the right choice to build a high performance, distributed storage platform which lowers the management overhead and offers features such as zero-copy snapshots, thin provisioning, bulk provisioning and quick restores.

Download the full white paper here!

Open vStorage 1.3

July arrived, a new Open vStorage release sees the light. We are happy to announce Open vStorage 1.3 which is a milestone release for Open vStorage: it is the first release where all content within a vPool is stored on the Storage Backend. On top, we now support the most used object store in the world, Swift.

  • The File Driver: In earlier version we requested the user to set up a distributed file system (which is very complex) to store the non-volume data or use a BitTorrent sync protocol to keep these non-volume files in sync between the different hosts. This release doesn’t require any of those workarounds anymore as we store these non-volume data ourselves in a highly available way on the storage backend.
  • Swift support: Swift is the most used object store for large public and private clouds worldwide and is implemented by PayPal , Wikipedia , MercadoLibre (Latin America’s largest online marketplace) and Disney. Now we fully support this Object Store as storage backend. This basically means we turn Swift into a block device. This is a new use case for Swift and is something nobody else can currently do.

These 2 milestones above are really important but the team also added some smaller features such as the new Arakoon release and better validation when configuring a vPool. Fixing bugs was of course also on the TODO-list. An overview of the most important fixes:

  • Fix so in use mountpoints are not listed when creating a new vPool
  • Prevent unnecessary volume stealing
  • Adding a second vPool on VMware should not stop the first vPool
  • Issue where Logstash is consuming too much CPU
  • Fixed Logrotation for OVS components
  • Fix for Arakoon not starting after a power failure in some cases
  • Fix for removing a vPool through the GUI
  • Support for /dev/vdx disks
  • Fix for sudo -s not working during package installation
  • Improved errorhandling in case of an S3 backend
  • GUI issue fixed for Firefox 30
  • Fix for issue with failing password update
  • Fix so you can remove an empty vPool while other vPools have vMachines
  • Fix for failing snapshots
  • Improved input validation when entering a vPool name