The past few weeks have seen a number of releases of our StableBit products.
Let me recap some of the new stuff that came out.
StableBit Scanner 2.2 (BETA) for Windows
Get it here:Â http://stablebit.com/scanner
StableBit Scanner 2.2 BETA is now available for Windows 7, Windows Vista, Windows 2008 (and R2), Windows 8 and Windows Server 2012. Both 32-bit and 64-bit OSes are supported.
It comes with all the features that you’ve seen in version 2.0 and a few other things.
- Computer power management is now part of the Scanner’s functionality.It will automatically wake up the computer to scan your drives and will not put it to sleep while it’s working.
This is on by default, but of course you  can turn it off in Scanner Settings.
- Automatic updating.The way this works is we deploy a cryptographically signed  update to our servers. Scanner 2.2 will check for updates every day. Once an update is found it will offer you the option to update to the latest version in the system tray.
If you choose to update, it will automatically download the  new setup package into a temporary folder, verify that the signature is authentic, ensuring that the update came from us, and then launch the setup.Once the setup launches it only takes a few clicks to update to the latest version.
StableBit DrivePool 1.1
Get it here: http://stablebit.com/drivepool
This one took a while to get out because it required some extensive testing. It has new file system locking code and some additional changes in how we buffer data. This is all under the hood stuff and so you won’t see any cosmetic changes or changes in functionality. These changes should improve interpretability with 3rd party applications such as anti-viruses that intercept and perhaps modify file I/O.
Network I/O Prioritization
After some extensive testing on the Windows Home Server 2011 and DrivePool 1.0, an issue became apparent with full length blu-ray streaming. For some reason, when a movie was being streamed from the share (not DLNA), every 30 minutes or so the video would stutter.
This made no sense since the server was not supposed to be doing much and after further investigation it turned out that DrivePool was not involved in the lag.
It turns out that the computer backup service was actually waking up periodically and consuming all the disk I/O resources on the server. Now this was odd because neither server backups nor client computer backups were set up on this particular server.
This is obviously a problem for anyone who wants to use WHS 2011 as a streaming NAS server.
But the larger issue here is that applications can wake up and interrupt your video streaming experience. In my opinion, this should not happen and is a design flaw.
I would think that the rule should be: All background services should run on a lower I/O priority level than anything else. But for some reason, Windows doesn’t follow this rule and it allows background services to come in and slow down your file transfers.
This is what Network I/O prioritization aims to fix.
What we do here is simple. We raise the I/O priority of file I/O requests, on the pool drive, that come from the network. In actuality, it’s not “all file I/O”, it’s file I/O that was not serviced by the read-ahead cache manager.
In effect, we are saying to the system, if you’re busy doing something else and can’t service these read requests in a timely manner… Stop whatever you are doing and service these network originated requests now.
After some testing this was an effective solution and remedied the stuttering that was present when streaming blu-ray images.
There is a small price to pay for this feature, in terms of CPU usage. Because we prioritize I/O in the kernel, in real-time, there is a small CPU hit to having this enabled. So if you’re sure that you don’t need this feature you can turn it off on the DrivePool Tab (under Disks).
Read Striping Improvements
As you might have guessed by now, a big part of DrivePool 1.1 was all about fine-tuning and improving the CoveFS file system. Up next is read striping.
After constructing a testing suite to examine various read-striping algorithms under different file I/O conditions, it became obvious that the read striping algorithm present in DrivePool 1.0 was not the optimal solution. It was effective at load balancing, but not as effective as it could be in actually speeding up read requests.
DrivePool 1.1 now has a special “block-based” read striping mode which is the culmination of this research.
Whenever DrivePool detects sequential file access, it will automatically switch itself into its new block-based read striping mode. This change is completely transparent to the user.
Fast Drive Removal
One interesting use case for DrivePool is to enable folder duplication on all the folders, which would protect all the files placed on the pool.
So if you do this, what happens when a drive fails?
You have 2 options,
- If the drive is completely dead, then just power down and physically disconnect it. Then, boot up and remove the missing disk in DrivePool. This takes mere seconds and is not a problem. Or you could hot-swap it if your hardware supports that.
- If the drive is still working, you might want to first remove the drive using DrivePool’s drive remove  wizard. The problem is that this can take a while, since it has to verify every file on that drive.But why should you have to wait at all?
Option #1 doesn’t make you wait. And the result is the same, all your data is still safe.
Starting with DrivePool 1.1 we now have a check box on the drive removal wizard to enable “Fast Drive Removal”.
When selected, DrivePool will only attempt to copy out of the drive any un-duplicated files, and completely skip any duplicated files for later re-duplication in the background.
This all sounds great, so why not just make this the default?
If you don’t select fast drive removal, we will warn you of any duplication problems with the other copy before you unplug the drive that you’re removing.
A New File Balancing Algorithm
DrivePool 1.0 already featured 2 separate file balancing algorithms to ensure maximum drive capacity availability for file duplication.
DrivePool 1.1 introduces a 3rd new balancing algorithm.
This algorithm is designed to prevent drive fill. It’s very simple, and it goes something like this:
If a pooled drive is more than 90% full, and it has some pooled data on it, and we have some free space on some other drive… then move the pooled data from the full drive to the drive that has more free space.
Why do we need this?
DrivePool 1.0 already prioritizes new file creation towards the drive with the most free space, so it really has no problem when drives fill up. You would still be able to copy new files to the pool, and if the file placement is obstructing duplicate file part creation, we already have algorithms to deal with that.
So what’s the problem?
Two problems actually,
- Growing existing files. If a pooled file is on that almost full drive, and some program decides to significantly extend its file size, you might run out of disk space on that drive and the operation would not succeed.
- Non-pooled file creation. Any non-pooled shared folders on that drive would be full, and you would be unable to copy any files into them.
This new algorithm addresses both points, and it only requires a single balancing pass at night, if necessary. Of course we don’t lock any files while we do this, just like our other algorithms.
As I’ve mentioned in my last post, the next release of DrivePool (1.2) will feature a brand new balancing system and UI which will be much more extensible and customizable. It won’t require any configuration out of the box, and will work exactly as this one does if you don’t want to fiddle with it.
StableBit Scanner 2.1
Get it here:Â http://stablebit.com/scanner
Scanner 2.1 for the Windows Home Server 2011 was released last week. It features some installer changes in order to make it compatible with DrivePool 1.1. It also has the new power management features that are available in Scanner 2.2 for Windows.
Overall, it features some necessary under the hood changes so that we can move forward with future versions. You will not notice anything visually different in this one.
StableBit.com Update
We’ve also simplified the front pages of both products on the StableBit web site.
Check it out: http://stablebit.com
That wraps up this update on the last 2 weeks which, as you can see, were pretty busy in terms of new releases.
We’ll be rolling out the next DrivePool BETA, as soon as it’s ready, which should be very exciting.
In addition to the balancing changes, there will eventually be some deep scanner integration as a result of the new balancing features.