StableBit DrivePool Technical Overview

StableBit DrivePool is a Windows Home Server 2011 add-in that pools multiple physical drives into one virtual drive and provides file duplication on a folder level.

It is currently in open BETA.
(Be careful, it’s a BETA)

Benefits

At its core, StableBit DrivePool is designed to solve 2 problems:

  1. You should never have to worry about running out of disk space. If you’re running low on disk space, then just plug in another hard disk, add it to the pool, and you’re done. You don’t have to change the way you organize your data to meet the needs of the system. The system should be flexible enough to meet your needs.
  2. Important files should have an extra layer of protection. All hard drives fail, this is a fact. It’s just a matter of when. File duplication works to keep 2 copies of every important file on two separate hard drives. In case one of them is inaccessible, the other one is used to fill in the gaps. You’re in control. You get to choose which files to duplicate and which not to, on a folder by folder basis. No need to match disk sizes or to repartition anything. Once again, the system strives to meet your needs, not the other way around.

Design mantra

StableBit DrivePool was designed to be as simple as possible to solve the problems at hand, but no simpler. In this post I’ll take you through an overview of how the system works. One of the goals of designing DrivePool was simplicity and transparency of process, and storing your files in a widely available format. DrivePool does exactly that. All your files are stored on standard NTFS partitions. They can be accessed by any standard Windows system.

Disk pooling

After installing StableBit DrivePool you will get a new Dashboard tab, with two sub-tabs to manage your shared folders and pooled disks. Think of this as a pooled version of the standard Folder / Disk management interface.

At the same time, a special folder gets created in C:\ServerPool\ServerFolders. This is where the pool is mounted and where all your pooled folders and files exist. But you don’t manage them directly. Normally, you would use the Dashboard interface to add or remove disks from the pool and to create and delete shared folders.

Pool parts

The pool itself is in a self-describing folder structure. The files and folders that  it contains and duplication status is embedded in the actual folder structure and does not rely on any registry keys or configuration data. In fact, once you read this you will see how simple it is to add any drive to the pool manually by just creating a folder on a disk.

This is not to say that there isn’t external metadata. For instance, keeping track of missing disks is done using external data files. But pool participation is not determined by it. You can add a disk to the pool by just plugging it in.

To put it simply, in order for a disk to be part of a pool, it has to meet two criteria:

  1. It must be a NTFS formatted disk.
  2. It must have a folder with the following name: X:\ServerPoolPart.[GUID]

For #2, the GUID must be a unique string differentiating each pool part. This exists solely to detect missing disks, or let’s call them pool parts from now on.

Each pool part contains folders, and the folder’s name describes whether it’s duplicated or not. For example, let’s say that we want to create a duplicated Covecube folder on our drive pool.

We would create the folder like this:
C:\ServerPool\ServerFolders\Covecube.2

StableBit DrivePool will create the actual folder here:
X:\ServerPoolPart.{220b6a9a-0e61-485e-86c1-29cef19359d0}\Covecube.2

See the .2 at the end? That tells DrivePool to keep 2 copies of every file placed below this folder structure and to perform redundant I/O. In other words, for every read operation, if it fails, then it will try the same operation over the second copy before giving up.

Note that we don’t normally create folders manually. All folder creation should done through the Dashboard. And if you do make the modification manually, you would need to restart the StableBit DrivePool service in order for this new pool part and folder to be recognized. However, when you use the dashboard interface, the DrivePool service is informed of the change automatically.

How files are stored

This is very simple. Any files or folders placed under C:\ServerPool\ServerFolders\ actually get placed on one or more pool parts (i.e. they get placed on one or more physical disks).

You can think of C:\ServerPool\ServerFolders\ as a proxy that combines all of your pool parts into one. There is no SQL database involved or any such thing, so that’s one thing that will not get corrupted.

The files themselves are stored as standard NTFS files on the pool parts. There is absolutely nothing special about them. This is extremely important and is worth reiterating. In case your home server crashes, or becomes inaccessible, you can access any of your files by plugging in a disk with a pool part into any standard Windows machine. There is nothing special about duplicated files either. They are just stored twice on different pool parts.

Adding / removing disks

Adding a disk is as simple as creating the special ServerPoolPart folder. While removing a disk is a little more complicated.

To remove a disk, StableBit DrivePool will virtually disconnect the pool part on that disk from the C:\ServerPool folder. It does this by simply pre-pending “Removing.” to the beginning of the pool part’s folder name. It then begins a process where it moves each non-duplicated file from the disconnected pool part back into the pool.  For duplicated files, it re-generates each duplicated file to make sure that there are 2 copies remaining after the disk is removed.

It does all of this in a transactional way, where if a file can’t be moved or there is not enough disk space left in the pool to regenerate a duplicated file, it aborts the process by re-integrating the pool part being removed. This is as simple as removing the “Removing.” prefix from the pool part’s folder name.

Automatic balancing

Currently, there is no periodic file balancing process. Each file is placed on the correct disk at the time it’s created. That is not to say that a manual balancing process will not be introduced at a later date. You may want to manually re-bias your file distribution, and that would require a manual re-balancing process. The current BETA has no facility for this.

Performance

The current build does not focus on performance. But the plan is to offer performance on-par with non-pooled disk access (perhaps even faster). This is still in an experimental state. More information on this once it’s available.

Conclusion

StableBit DrivePool is still in early BETA and some of this design is subject to change as we deal with things like server backup and media streaming and perhaps other things. But the mantra will remain the same. It’s to have a simple and transparent system that does not place artificial limits on what you can do with your files, and to store your files in a standard widely available format.

  • Jon Smejkal

    Thanks for bringing drive pool capability to WHS 2011.  We can all finally upgrade and enjoy the benefits of 2011!   Hopefully the MS DE debacle will be over thanks to StableBit stepping in the fill the HuGe functionality gap.

  • Shane Curtis

    Hi, now that there’s C:ServerPoolServerFolders and C:ServerPoolServerFolders.Mount, could you clarify the difference between them?

  • ServerFolders.Mount is where the pooled volume is mounted. Just like you can mount any drive as a folder in the disk management snap-in, this is what DrivePool does to the pool.
    ServerFolders was introduced in order to interface with the WHS storage manager service. This adds a bit of complexity but was a necessary step in order to ensure compatibility with the WHS service. In particular, ServerFolders holds the actual share on the folder through a link.

    ServerFolders is managed by DrivePool and you should not alter it. DrivePool will re-create it if it’s missing and re-synchronize it automatically.

    If you need to work directly with the pool you should use ServerFolders.Mount. Use the dashboard to create / remove directories in ServerFolders.Mount because it will set up proper NTFS permissions and configure the media streaming service and your HomeGroup.

    It’s be perfectly ok to point, say a backup app to ServerFolders.Mount in order to backup data from the pool.

    But overall just use the shares. It’s much simpler, only point to ServerFolder.Mount if it’s absolutely necessary.

  • Shane Curtis

    It appears that one problem with using C:ServerPoolServerFolders.Mount(sharename).(duplication) rather than C:ServerPoolServerFolders(sharename) is that a change in the duplication status of the share will break the program?

  • FolderLinks are managed automatically by DrivePool. Renaming a folder / changing the duplication status or deleting it in the mount will be reflected in the FolderLinks automatically when managed using the Dashboard.

    In addition, DrivePool will synchronize FolderLinks on start-up.

    The original intention behind using a .1 or a .2 was to make the architecture transparent. I think it would be a good idea to keep those around if WHS allows.

    It’s possible to get rid of the suffix and use NTFS metadata to hold the duplication status, but then it wouldn’t be obvious to the user which folder is duplicated and which is not just by looking at the name.

  • Shane Curtis

    I agree, and I very much like DrivePool’s transparency! I apologise for being unclear – what I meant was, if non-UNC programs A, B and C get pointed at ServerFolders.MountMyShare.1ABCdata rather than ServerFoldersMyShareABCdata and later on MyShare.1 becomes a duplicated folder, then those programs A, B and C will still be looking for the old .1 path and not the new .2 path… I hope that’s clearer?

    Good thing is, programs that don’t accept UNC paths are becoming more and more rare as time goes by.

  • Ok, I see what you mean. Yes, changing the duplication status will rename the folder and so you’ll have to re-point the application.

    The alternative would be to get rid of the suffix completely, but I’d like to keep it around.