StableBit

StableBit DrivePool 2.0.0.400 Release Candidate

Posted in StableBit on September 6th, 2013 by alex – 3 Comments

StableBit DrivePool 2.0 is now a Release Candidate!

Version 2.0.0.400

Version 2.0.0.400

Download here: http://stablebit.com/DrivePool/Download

This build will be pushed via. automatic updates to everyone using the BETA within 24 hours.

What Exactly is a RC?

As far as StableBit products are concerned a Release Candidate is built exactly like a Release Final. The idea behind having a RC is to test the Release Final build process. It’s simply a Release Final that has been renamed to a RC. A RC can be promoted to a Release Final with no rebuild.

This means that the trial BETA rules no longer apply to RC builds. Your days remaining on the trial will not reset when installing subsequent RC / Release Final builds.

What’s New Since the Last BETA

This build has a number of miscellaneous fixes and tweaks since the last BETA. Localizations are mostly complete for the following languages: Spanish, French, German, Polish, and Italian.

As always, you can check out the full change log for the details: http://stablebit.com/DrivePool/ChangeLog?Platform=win

StableBit.com Updates

User Manual

User Manual

User Manual

There is a new comprehensive user manual now available for StableBit DrivePool 2.x here: http://stablebit.com/Support/DrivePool/2.X/Manual

If you like reading manuals then you should definitely check it out, it took a lot of work to put together.

PayPal

The PayPal payment system was updated to use PayPal’s latest API in order to get rid of the old and somewhat clunky “PayPal for digital goods” API.

Google Checkout

Google Checkout (or Wallet) is being retired by Google in November (see: https://support.google.com/checkout/sell/answer/3080449?hl=en), so we’ll be phasing that out soon. It will probably be replaced with Amazon payments at some point in the future.

The Bundle

Product Bundle

Product Bundle

This has been requested quite often. If you’ve never purchased either the StableBit Scanner or StableBit DrivePool you can now easily buy both with the $9.95 discount without buying each product individually.

There are also links on your order page that make it easy to buy additional copies of a product with the additional unit discount.

Buy Additional Copies

Buy Additional Copies

How far We’ve Come

The first BETA of StableBit DrivePool 2.0 came out on Feb. 1 2013, and here we are in the beginning of September 2013 with a RC. All in all, I think we’ve made pretty good time considering the scope of this release.

Here’s a list of all of the supported Operating Systems that StableBit DrivePool 2.0 runs on:

  • 64-bit Consumer:
    • Windows Vista (64 bit), Windows 7 (64 bit), Windows 8 (64 bit), Windows 8.1 (64 bit)
  • 64-bit Server:
    • Windows Server 2008 (64 bit), Windows Server 2008 R2 (64 bit), Windows Server 2012, Windows Server 2012 R2.
  • 32-bit Consumer:
    • Windows Vista (32 bit), Windows 7 (32 bit) Windows 8 (32 bit), Windows 8.1 (32 bit)
  • 32-bit Server:
    • Windows Server 2008 (32 bit)
  • 64-bit Windows Server Solutions:
    • Windows Home Server 2011, Windows Small Business Server 2011 Essentials, Windows Storage Server 2008 R2 Essentials.
    • Windows Server 2012 Essentials
    • Windows Server 2012 R2 Essentials

I’ve highlighted what StableBit DrivePool 1.X supports. As you can see, that’s a lot of new Operating Systems that we now support. Plus, we have the brand new remote control system that lets you manage your pools from other computers on your LAN (without the need for a Dashboard).

The Price

StableBit DrivePool 1.0 Release Final came out on April 2 2012, almost 1 and a half years ago. It was a great release and it was priced at a very affordable $19.95. Since then we’ve added a ton of new features, including the balancing framework, multiple pool support and everything that you see in StableBit DrivePool 2.0. All of these new features come at no additional cost to our existing customers, and we’re not done yet.

We’ve added a lot of value to StableBit DrivePool since that first release, and I feel like we need to bump up the price a bit to reflect that. But don’t worry, nothing has changed yet. The price won’t be increased until a release final is published.

The future price increase does not affect our existing OEM partners. You will continue to be able to purchase client licenses at the negotiated rate.

Thank You Everyone

Thank you Christopher for offering outstanding technical support to our customers.

And thank you everyone for your great support of our products, but the best may be yet to come. Post 2.0 we may yet see some really amazing game changing features in StableBit DrivePool, if everything goes according to plan.

As always, if you like our products I ask that you spread the word. I think that StableBit DrivePool 2.0 turned out really well and I’m sure that there are many people out there who have never heard of it and may benefit from using it.

StableBit DrivePool 2.0.0.387 – Dashboard Tab

Posted in StableBit on August 22nd, 2013 by alex – Be the first to comment
WHS 2011 Dashboard Tab

WHS 2011 Dashboard Tab

Well, I guess you can say that we’ve come full circle. The latest build of StableBit DrivePool 2.0 features a WHS 2011 Dashboard tab. Not only that, it adds a Dashboard tab to any OS that has a Dashboard starting with WHS 2011, including the Windows Server 2012 R2 Essentials Preview.

Download here: http://stablebit.com/DrivePool/Download

The Dashboards

Windows Server 2012 Essentials

Windows Server 2012 Essentials

Windows Server 2012 R2 Essentials

Windows Server 2012 R2 Essentials

To get the Dashboard tab you can simply install the latest build on any OS that has a Dashboard and the installer will recognize it and install the necessary Dashboard components.

There is no separate WSSX installer for now, perhaps we can add this in the future.

Sorting Disks

This build is mostly about miscellaneous fixes as we approach the release final, but there is another small noteworthy feature. You can now sort your pooled disks in various ways from the horizontal UI.

Sort Disks

Sort Disks

This was actually a user request turned into a feature and was a pretty simple thing to implement, so there you go.

First Pool Confusion

A common support question (and sometimes followed by panic) is:

“I’ve created my first pool, but now that I look at the pool drive, it’s empty! I thought that the message said that my files would not be altered. Where are my files from the disk that I just added ?!”

Of course the answer is simple. The files are actually not altered and are still on the original disk. By adding a disk to the pool, the files on that disk are really not altered in any way (I mean really), but the disk now contributes its free space to the pool.

To alleviate the confusion, there is now a help screen that pops up after you create your first pool.

First Pool

First Pool

Don’t worry this only shows up once to help new users understand how the pool works.

Other Fixes

Some other miscellaneous fixes in this build are:

  • Microsoft NFS Server support was fixed for Windows Server 2012.
  • The ability to add a pool part from the command line.
  • Pools without a drive letter assigned would not be recognized correctly. This was actually a big problem if your “automount” setting was disabled in Windows.

    This is what you will see now:

    No Drive Letter

    No Drive Letter

  • UI settings were getting reset on each update.
  • Some flash card readers were causing the Service to constantly update the disks list, leading to unnecessary CPU load and a slowdown of the UI.
  • All UI state is now stored in “%appdata%\StableBit DrivePool” allowing for an easy reset.

You can check out the change log for a full list of fixes in this build: http://stablebit.com/DrivePool/ChangeLog?Platform=win

The Release Candidate

At this point everything needs to be tested thoroughly and hopefully we can see a RC of StableBit DrivePool 2.0 soon.

StableBit DrivePool 2.0.0.355 – Folder Duplication Improvements

Posted in StableBit on August 12th, 2013 by alex – 3 Comments

Continuing on the path to a release final for StableBit DrivePool 2.0, this build concentrates on folder duplication.

StableBit DrivePool - Folder Duplication

Folder Duplication

You can download the latest version here: http://stablebit.com/DrivePool/Download

What’s New with Folder Duplication?

In short, a lot of under the hood improvements, some UI polish and the ability to duplicate (triplicate?) your files to more than two disk from the UI.

For a list of all of the technical changes, you can check out the somewhat lengthy change log over here: http://stablebit.com/DrivePool/ChangeLog?Platform=win

Let’s talk about some of the highlights.

UI Responsiveness

In terms of UI responsiveness, this build does for folder duplication what build 345 did for adding / removing disks. Everything should be a lot smoother and more intuitive.

Let’s dive right in with a video showcasing the updated folder duplication UI.

(make sure to select 720p quality)

The video is unedited, but I used a very small pool in order to avoid long background duplication times.

Folder Duplication Stats

As you can see in the video, one new UI element is a pie chart showing you folder duplication statistics for any folder that you select.

StableBit DrivePool - Folder Duplication Statistics

Folder Duplication Statistics

The statistics are computed in the background, in real-time, as you browse the folder duplication dialog. It gives you exact folder sizes, as they are on the disk, down to the last byte. This means that you can use the statistics as a means of verification that all of your files are actually duplicated properly.

This has been a user request for some time. Well, here it is.

StableBit DrivePool - x2 Folder Duplication

x2 Folder Duplication

This is another screen showing a x2 duplicated folder. Here you can clearly see how the pie chart serves as a quick visual cue of the proper distribution of files in that folder.

Remote Control

It’s worth reiterating that the folder duplication UI, including the new folder size statistics chart, works with remote control.

This allows you to set up centralized management of one or more servers, or the other way around, multiple clients managing one system, or any combination thereof.

Multiple Duplication Counts / Duplication Inheritance

In build 355 there are some changes to the folder duplication tags shown in the UI.

Multiple Duplication Counts

Multiple Duplication Counts

As you can see in the screenshot above, there is now a little “+” next to any folder that has one or more sub-folders with a different duplication count than itself. This is the Multiple Duplication Counts flag and it serves as a hint for you to expand that folder in order to see which sub-folders are duplicated.

Multiple Duplication Counts - Expanded

Multiple Duplication Counts – Expanded

In addition, all of the blue tags are folders that are inheriting their duplication counts from their parent folder. This is the default for all new folders, unless you change a folder’s duplication level explicitly using the folder duplication dialog.

Under the Hood Changes

There are numerous and extensive under the hood changes to how StableBit DrivePool handles the Multiple Duplication Counts and Folder Duplication Inheritance flags internally. You can see the change log for a complete explanation of them.

Concurrency

As you can see in the video, you don’t have to wait for an existing background duplication pass to complete before changing the duplication level of another folder. The system is smart enough to suspend the current background duplication pass, set the duplication count on another folder and then restart the pass now including the new folder.

In fact, every task in StableBit DrivePool, such as adding / removing disks, rebalancing, or setting a duplication count uses the same system to coordinate which tasks are allowed to run and when. That’s how we are able to change folder duplication counts even while background duplication is running.

I hope that everyone enjoys the new folder duplication improvements. Until next time.

StableBit DrivePool 2.0.0.345 – Windows 8.1 Support and More

Posted in StableBit on August 1st, 2013 by alex – 1 Comment

 

The latest build of StableBit DrivePool features Windows 8.1 Preview support and a lot of fixes.

Windows 8.1 Preview Support

Windows 8.1 Preview Support

Get it here: http://stablebit.com/DrivePool/Download

Let’s talk about what’s new since build 310.

Windows 8.1 Preview Support

Well there’s really not much to say here, except that StableBit DrivePool now works correctly with Windows 8.1.

In latest builds, CoveFS for Windows 8.1 is now compiled with the development tools that Microsoft released with the Preview OS and a number of changes had to be made in the file system to make it work correctly with Windows 8.1.

If you like living on the bleeding edge, then we’ve got you covered.

UI Responsiveness

Previously, the DrivePool UI was not as responsive as it was designed to be when adding and removing disks. This is now fixed in the latest build and to demonstrate I’ve made a short video showcasing how easy it is to add and remove multiple disks to and from the pool.

https://www.youtube.com/watch?v=fwN9UU15TEw

(make sure to turn up the quality to 720p to get a clearer view)

As you can see, clicking the add / remove labels now gives you proper and instant feedback. Queuing up multiple disk operations is a snap, and you can see that StableBit DrivePool is even able to perform multiple simultaneous operations on different pools at the same time. For example, you can be creating a new pool, while adding or removing another disk on an existing pool.

Remote Control Updates

Remote control allows you to easily connect to and manage another system running StableBit DrivePool on your LAN.

Simply select which computer you want to connect to from the drop down at the top.

List Computers

List Computers

This by itself is not new, but we have 2 new remote control features in the latest build.

Reconnect to the Same Computer Used Last

This was a user feature request and is now implemented in the latest build.

If you were connected to a remote computer when you closed the UI, the next time that you open the UI it will automatically connect to that computer (provided that it’s still online).

It’s a very small change, but actually makes a big difference if you only have one DrivePool server and tend to manage it from many clients. We now save you 2 clicks every time you start the UI.

Static Peers

By default StableBit DrivePool uses multicast UDP packets to discover other peers to connect to. Sometimes, due to your network configuration, multicast packets can be dropped by your router.

Now in the latest build the Remote Control system can connect to static peers specified by you.

The installer now places a new XML file in:
C:\Program Files\StableBit\DrivePool\RemoteControl.default.xml

You can edit it and define your own DrivePool peers using IP addresses, Windows computer names or domains. For further instructions just open the XML file in a text editor like notepad.

Duplication Tag Consistency Check

When you enable per-folder duplication, StableBit DrivePool writes duplication tags to the directory entries on the disk in order to save the requested duplication level for that folder.

Having correct duplication tags is essential to the file protection that StableBit DrivePool provides.

How is This Possible?

You may be wondering, how can you get inconsistent duplication tags?

Here are a number of scenarios that can give you inconsistent duplication tags:

  • You enable folder duplication on a folder and then completely restore from backup one of the disks part of the pool to a state prior to when you had folder duplication enabled.
  • You disconnect one of the disks in a pool, connect it to another system running StableBit DrivePool and then disable folder duplication on that folder from the other system.
    Then you reconnect the same disk to the original system that had folder duplication enabled.
  • You enable folder duplication on a folder and then have a power failure a short time after.

The last case is actually not a problem with the latest builds. We now write the duplication tag through to the disk at the time that you enable it, but previously this could cause your duplication tags to get out of sync because the tag was cached.

What are the Consequences?

If a duplication tag gets out of sync on one or more pool parts, then any pooled files placed on that disk will not be duplicated. In addition, the folder might appear to be duplicated or not in the UI.

Can This be Addressed?

Yes, and it is no longer a problem in the latest build.

Every time that a pool arrives in a system, we kick off a duplication tag consistency check. The check is very fast and only traverses folders that have duplication tags set on them. The check makes sure that all the tags are valid and it checks every disk part of the pool and ensures that all the tags on each disk are in sync.

If a tag is missing or malformed an intelligent conflict resolution algorithm is run to construct a new repaired tag. The algorithm uses the available information in the conflicting tags, but if it’s in doubt it assumes that the folder was duplicated. After constructing the repaired tag it overwrites the old tags with the repaired one. It then kicks off a full duplication check over the entire pool, duplicating any files that are not duplicated.

This fixes any future such issues and also repairs pools with existing conflicting tags.

Miscellaneous Fixes

There are many other miscellaneous fixes to lockups and crashes in the latest build.

You can consult the full change log on the download page: http://stablebit.com/DrivePool/Download

At this point StableBit DrivePool 2.0 BETA is on the fast track to a release final. All outstanding issues are being addressed and everything is being finalized. Hopefully we’ll have a RC after a few more builds.

StableBit DrivePool 1.3.3 Release Final

Posted in StableBit on July 23rd, 2013 by alex – Be the first to comment

StableBit DrivePool 1.3.3.7563 is now available for download as a release final for the folks who are using the Windows Home Server 2011.

Windows Home Server 2011 - NFS Server

Windows Home Server 2011 – NFS Server

Get it here: http://stablebit.com/DrivePool/Download

Along with the usual bunch of fixes (a lot of them based on your feedback – thank you) StableBit DrivePool now supports the Microsoft NFS server out of the box.

NFS Server Support

(I know this is going to get a bit technical, but this feature has been requested enough times that I think many of you will find it useful.)

For those of you who don’t know what NFS is, it’s a different file sharing protocol typically used on UNIX and Linux machines. It’s also supported by standalone media players (such as BOXEE) and generally offers much better performance than the built in Windows SMB protocol.

In my testing I’ve seen nearly a 10x increase in throughput on BOXEE in particular.

Unfortunately there is no Dashboard integration for NFS and so it must be set up by logging into the server using Remote Desktop.

NFS Role

NFS Service

Here are the general instructions on how to enable NFS support:

  • Log into the server using Remote Desktop as Administrator.
  • Open up the Server Manager.
  • Navigate to Roles -> File Services.
  • Click Add Role Services.
  • Check Services for Network File System and complete the wizard.
NFS Share

NFS Share

Now to share a folder over NFS:

  • Open up Windows Explorer.
  • Navigate to the pool drive.
  • Right click on the folder that you wish to share over NFS and click Properties.
  • Select the NFS Sharing tab and click Manage NFS Sharing…

    • Check Share this folder.
    • (optional) Click Permissions and check Allow root access, then click OK.
    • Click OK.
  • Click Close.
NFS Share - Permissions

NFS Share – Permissions

Based on my testing with BOXEE, the optional step that I’ve included will enable BOXEE to see and access the NFS share without issues.

Open by File ID

In order to implement NFS support, CoveFS now supports the open by file ID functionality. This means that applications can request an ID for any file in the system and then open that file by using the ID instead of the path name. Open by file ID support is required by the Microsoft NFS server and that is why it didn’t work on the pool.

If you’re absolutely sure that you don’t need file ID support on the pool, you can turn it off by disabling CoveFs_OpenByFileId. For more information see: http://wiki.covecube.com/StableBit_DrivePool_Advanced_Settings

StableBit DrivePool 2.0 – Windows 8.1 Preview Support

On a completely separate note, I’d just like to mention that StableBit DrivePool 2.0 BETA is getting Windows 8.1 support in the next build coming very soon, and I’ll talk about that in my next post.

StableBit DrivePool 2.0.0.310 BETA – File System Improvements

Posted in StableBit on July 1st, 2013 by alex – 2 Comments
StableBit DrivePool Build 310

StableBit DrivePool Build 310

StableBit DrivePool is now on the road to a stable release final. The latest build doesn’t feature much visual changes but instead is focused on refining the file system component of DrivePool.

We’re still not done with tweaking the UI, it’s just not the focus of this build.

Download the latest build here: http://stablebit.com/DrivePool/Download

CoveFS Changes

CoveFS is our high performance kernel driver written specifically to pool multiple file systems together. In this build, it has many changes to ensure compatibility with 3rd party applications, in particular Enterprise grade applications such as database engines, virtual machines and essentially any applications that rely on very heavy concurrent I/O on the pool.

For example, here is what we are now able to do with this latest build out of the box:

  • StableBit DrivePool can now be checked out and built and deployed directly from the pool using SVN, Visual Studio 2012, MSBuild and some proprietary deployment scripts.
  • All of the Virtual Machines used for testing now run from the pool. This means that they can instantly benefit from the performance gains obtained through read striping, and that they are protected from drive failure by file duplication.
    • The Virtual Machine software used is VMWare Workstation 7.1.4 and Oracle’s VirtualBox 4.2.12.

A lot of work went into putting in the file system changes necessary for this particular build and I’ll talk about some of these changes below.

Report as NTFS by Default

After the many months of testing the DrivePool 2.0 BETA, it has become apparent that we can no longer report our file system as “CoveFS”. There are simply too many applications out there that check specifically for “NTFS”, and if they don’t find that string then they either refuse to run outright or assume that the file system is FAT based (which is typically limited to 2GB/4GB file sizes).

The latest example of this is VMWare Workstation 7. It needs the file system to be “NTFS” or else it assumes a 2GB file limit, and refuses to run. Windows 8 also has hard coded strings for “expected” file systems in some of the (reverse engineered) system code, and if a file system is not on that list it misbehaves in subtle ways.

So in the latest build, CoveFS now reports as “NTFS” by default. You can still revert to reporting as “CoveFS” by editing the service .config file.

Quirky File System Behavior

Some things in Windows are just wacky, and programs have been built to expect this wackiness or else they break. So we have to do our best to reproduce this wackiness in CoveFS, in order to maintain the best possible compatibility with these programs.

Here’s an example:

Normally in Windows, you can rename a file (or folder), even if that file is currently in use. Typically applications use this behavior to rename a file that can’t be deleted because something else is using it, perhaps you’ve even used this trick yourself. The Windows Home Server 2011 did this when you requested to move a folder to a different disk, but that folder was in use. I’m sure that there are countless other examples.

What’s the wacky part?

You can also tell Windows to open a file that’s in use and rename it to itself, while requesting to explicitly overwrite the destination.

Now how does that make any sense at all? How can you rename a file to itself, much less overwrite the destination (which is itself, and is in use by something else)?

In this case CoveFS used to tell the caller that the destination is in use and can’t be overwritten. However, NTFS just does this with no complaints. So now that’s what CoveFS does as well.

It’s these kinds of issues that make a huge difference in compatibility. This particular problem was causing VMWare to not function correctly from the pool.

Locking / Caching

One of the most important parts of any file system is how it interacts with the memory manager and the cache manager.

Locking and caching affects how EXE files are opened, how data is cached when read from or written to the disk, and how memory mapping works. Memory mapping is typically used by database or backup applications, word processors, or anything that needs to access random areas of very large files quickly.

The latest build drastically simplifies how DrivePool interacts with all of these core systems.

Our new model is much simpler. Eliminating complexity means eliminating potential for problems, so I think that this is a very important change.

File Performance was Broken

The idea behind file performance was a small pane to answer the question “what is going on on the pool?”. It was supposed to show you the top 5 most active file transfers in real-time.

The original back-end file performance implementation was written using ETW (the same thing that the Windows Resource Monitor uses).

Unfortunately it had some problems:

  • It was kind of slow because it was difficult to parse the required I/O in real-time.
  • Ongoing I/O that was started prior to you opening up the performance UI pane would not be shown.
  • There was a 2 second delay before we could show you the file I/O.
  • Some I/O was missed outright due to the cache manager’s lazy writer / read ahead functionality.
  • What was shown in File Performance was not in sync with what was shown in Pool Performance (I’ve alluded to this when I first introducing this feature).

Introducing Disk Performance

Well, the old code that handled File Performance was scrapped.

A completely new high performance design was created that works intimately with the CoveFS file system to give you instant file performance.

And now it’s called Disk Performance.

From the user’s point of view, the UI is similar except that it’s more accurate and timely.

But here are the functional differences:

  • Performance data is sampled every one second and is not delayed. You are seeing near real-time activity and not a historic average.
  • All I/O that goes through the pool is recorded. Cached, uncached and Fast I/O is always included. We sample everything.
  • Performance sampling is very efficient. There is no need to parse anything. The file system keeps track of all of the statistics and gives us exactly what we request.
  • There is virtually no overhead. The file system only keeps samples of file performance if it was queried for the data within the past 60 seconds.

Accuracy

We now have much greater accuracy.

Disk Performance Accuracy

Disk Performance Accuracy

For example, in the above screen you can see an ongoing throttled file copy. The file copy was throttled to 1.0 MB/s using SyncBack SE.

You can see that the number reported by DrivePool is very close to 1.0 MB/s and that the Pool Performance statistics are now completely consistent with Disk Performance.

Live Transfers

Because of the increased responsiveness of the new design, DrivePool can now actually show you whether the file transfer listed in the Disk Performance pane was ended recently.

Disk Activity

Historic Disk Activity

For example, in the above screenshot we can see that all of the disk activity is grayed out. This means that the activity is Historic. In other words, the disk activity has already ended by the time that we’ve asked for the sample. By default DrivePool samples performance data once per second (which can be tweaked in the service .config).

Windows 8.1 / Windows Server 2012 R2

The current build of DrivePool is not compiled for the Windows 8.1 kernel, so it’s not compatible with the Windows 8.1 preview that Microsoft released last week.

While I would have liked to build a Windows 8.1 compatible version of DrivePool for this release, unfortunately the development tools that Microsoft released are not functioning correctly on the build machine (well, they’re crashing).

To be fair these are “Preview” tools and I’m sure that they will fix these issues before the final.

So I’d just like to mention that Windows 8.1 support is coming to DrivePool.

Other Issues / Feature Requests

There are a number of other issues and feature requests that people have been making.

For example:

  • Reparse points are currently not supported on the pool. This means that you can’t create symbolic links on the pool. Stay tuned, this is being worked on.
  • The Microsoft NFS Server doesn’t work from the pool on Windows Server 2012. This is also being addressed. Perhaps the 2012 NFS server requires reparse point support, I don’t know at this point. But it will be addressed.

The next build of DrivePool 2.X will concentrate on fixing / refining any remaining issues in the service and the UI.

Areas that are of interest are:

  • Refining the per-folder duplication UI.
  • Testing remote control more thoroughly and fixing any remaining issues there.
  • Showing better error messages when adding / removing disks fails.

You can download the latest build of StableBit DrivePool here: http://stablebit.com/DrivePool/Download

Until next time.

StableBit Scanner 2.4 Release Final

Posted in StableBit on June 18th, 2013 by alex – Be the first to comment
StableBit Scanner 2.4

StableBit Scanner 2.4

The release final of the StableBit Scanner version 2.4 is now available for download.

Get it here: http://stablebit.com/Scanner/Download

Since I’ve already talked about some of the new features in this version in previous blog posts, I’ll just quickly go over them again without getting into too much detail.

Then we’ll talk about what’s new since my last post.

Automatic File System Checking

File System Status

File System Status

In addition to testing your disks for physical damage, the StableBit Scanner will now check the file systems on your disks for formatting errors.

Repair Volume

Repair Volume

If damage is found, you can easily repair the volume right from the StableBit Scanner’s UI.

Scanned Sector Aging

Sector Scan Age

Scanner Sectors

Disk sectors that are not being re-scanned in a timely manner will turn yellow. The more “past-due” they are for a scan, the more yellow they will get.

Better Handling of Removable Media

Settings - Scanner

Settings – Scanner

This is a new feature that has been added recently, and I haven’t blogged about it.

Whenever you insert a USB flash drive or a USB hard drive, by default, the StableBit Scanner will not start scanning it immediately. You can turn off automatic scanning of removable media permanently or you can have the StableBit Scanner start scanning the device only if it has been plugged in for some hours.

The StableBit Scanner was originally designed to be run on Servers. Because you wouldn’t typically be inserting and removing USB thumb drives or hard drives from a server, there really was no need to limit automatic scanning in any way.

In fact, the original thought behind this behavior was that it would allow you to set up a disk scanning workstation. You could simply connect a new disk whenever you would want to scan it, and the Scanner would immediately start checking that disk for problems with no user intervention.

While it is still possible to use the StableBit Scanner for this scenario, it is definitely not the typical use case. So the default settings will now never scan USB thumb drives automatically, and will begin scanning USB hard drives only if they’ve been plugged in for at least 1 hour.

Past Due Scanning

Past Due Settings

Past Due Settings

This is a very simple addition that makes a big difference.

If you’ve setup your settings to only scan within a time window, it’s possible for your scans to become “stale”. In other words, your disks are not being scanned in a timely manner.

This can happen because perhaps the computer was too busy at the scan time and the StableBit Scanner has throttled your existing scans, or if you simply have too many disks. If either of these are the case, then you could get into a situation where the StableBit Scanner is not given enough time in order to maintain the health of your disks and properly check them for health.

With this new feature, the StableBit Scanner will now be given permission to scan your disks outside of the time window, if the last scan time of any disk is “past due” by some number of days (that you specify).

Scan Progress Recovery

Here’s another small feature that is pretty useful.

If your computer crashes or loses power in the middle of a surface scan, the StableBit Scanner will now be able to recover its last scan progress on the next reboot. Before this, you would lose your scan progress and would have to rescan that disk from the beginning.

Fahrenheit

Settings - Heat

Settings – Heat

Those of you who live in countries that don’t use Celsius as a measure of temperature may be wondering, why in the world is the Scanner always reporting the drive temperatures in Celsius?

There are really 2 reasons:

  • That’s how the drive reports it.
  • In engineering fields, thermal limit specifications are typically reported in Celsius.

But due to popular demand, you can now choose your own temperature scale. We default this setting to whatever the OS prefers in your region.

Miscellaneous

There are many other miscellaneous changes and fixes, which I didn’t mention here.

You can check out the full change log on the Download page.

Thanks for reading.

StableBit DrivePool 2.0.0.256 BETA – Performance UI

Posted in StableBit on May 18th, 2013 by alex – 2 Comments

The latest BETA of StableBit DrivePool 2.0 adds some major new back-end features, improved read striping and a new Performance UI.

Performance UI

Performance UI

Get the latest BETA here: http://stablebit.com/DrivePool/Download

The Performance UI

The new performance UI lets you see exactly what’s happening on the pool, at a glance. It shows you the most important information in real time.

Of course, it works perfectly with the remote control feature, letting you connect to any other machine on your network that is running StableBit DrivePool, and see exactly how the pool is performing on that machine.

Let’s take a look at the new UI in detail, and what all the visual cues actually mean.

File Performance

File Performance

File Performance

The performance UI is accessible from the Horizontal UI (and not the Vertical UI).  To get into the horizontal UI, simply stretch the window wide.

As you can see from the screenshot above, the performance UI is split up into two categories, File Performance  and Pool Performance.

File performance is designed to show you the top 5 file operations on the pool, that are consuming the most resources.

It shows you the read and write speeds of each file operation.

File Performance Speed

File Performance Speed

But in addition to the speed, it shows you something that’s even more useful,  the disk activity percentage.

Disk Activity Bars

Disk Activity Bars

In fact, the list shows you the top 5 file transfers on the pool, with the highest disk activity. The longer the bar, the more “busy” the disk is.

Let me explain what that means. Every hard disk or SSD can only read and write so fast. In addition drives have different performance characteristics. For example, SSDs can jump around between different parts of the disk (i.e. seek) faster than hard drives.

This means that the maximum speed that any drive can read or write at, depends highly on the type of drive it is, and what file operations are being performed on it.

In short, the disk activity bar shows you how busy that disk is, considering all those factors.

Performance UI

Performance UI

So reading this performance chart we see that:

  • Disk I:\ is very busy, and DrivePool is reading from it at 96 KB /s
  • Disk F:\ is not so busy and DrivePool is reading from it at 12.2 MB/s

DrivePool happens to be reading the same file from both disks, which is called read striping, but more on this later.

Pool Performance

Pool Performance

Pool Performance

On the left of the performance UI, you will see Pool Performance.

In pool performance, you can see:

  • The total combined read and write speeds that are occurring on the pool at this time.
  • Out of all the read / write operations, how many of those are utilizing Fast I/O.
  • Out of all the read operations, how many are utilizing read striping, and what algorithm is being used to stripe the reads (more on this later).
  • And finally, out of all the read operations, how many are being prioritized by Network I/O Boost.

Let’s go over the 3 vertical bars in detail, in order to understand exactly what they mean.

Fast I/O

Fast I/O - Write

Fast I/O – Write

The notion of Fast I/O is something that Microsoft coined for a technology that is designed to bypass the file system and retrieve your data directly from the Windows cache.

The reality of the situation, is that this is not the case. In fact, Fast I/O can and mostly does interact with the filesystem and disk (albeit in a roundabout way).

A better way to think about Fast I/O is I/O that requires no additional processing and bypasses some of the overhead that is imposed on regular I/O. For example, when DrivePool is performing any kind of complex manipulation of the I/O (such as read striping or I/O prioritization) it will bypass Fast I/O.

Saving every CPU cycle was important back in the heyday of Windows NT, because computers were slow and every CPU cycle counted. But this may not be as relevant today.

Read Striping

Read striping is a feature in DrivePool that is designed to improve the performance of reading duplicated files from the pool. In essence it can read from multiple disks at the same time in order to improve (or maintain) performance.

Read Striping

Read Striping

Read striping in DrivePool actually operates in 2 different modes:

  • Bulk transfer – For sequential access of large files.
  • Optimal disk – For streaming and random access of data.

A mode is chosen when a file is first opened on the pool, based on the hints that the application gives us.

DrivePool shows you exactly what each read striping algorithm is doing in real-time, by coloring the read striping bar with 3 different colors.

Read Striping Colors

Read Striping Colors

Here’s what these mean:

  • Hold – During a bulk transfer, DrivePool has decided that it is best to hold on to the current disk and not switch over to another disk.
  • Switch – During a bulk transfer, DrivePool has decided that it is best to switch to another disk while the current disk services its outstanding requests.
  • Optimal – During random file access or media streaming, DrivePool has chosen the optimal disk based on outstanding I/O requests, and an adaptive algorithm designed to pick the fastest disk at this time (this is new).
Bulk Transfers
Read Striping

Read Striping

This is what a typical bulk transfer looks like.

Here we can see that DrivePool is reading from 2 disks at the same time (I:\ and E:\), and that 100% of the read requests are being processed by the read striping algorithm. Most I/O requests are going to the same disk, and the disk context is being switched very rarely. In addition, we can see that both disks are saturated (this is exactly what we want).

Media Streaming (or Random Access)

This part is brand new and very cool.

Read Striping Streaming

Read Striping Streaming

Let’s say for example that you’re streaming a 50 GB uncompressed Blu-Ray that is in a duplicated folder, with read striping enabled.

So far, everything is going according to plan. Read striping has switched into streaming mode and your data is being read from 2 disks at the same time (I:\ and F:\), and both disks are not busy at all.

Now, let’s say some process starts on disk I:\, something that completely saturates the disk. Normally at this point your media would start to stutter very badly. And because the media player is reading the data rather slowly, the standard bulk mode read striping algorithm would have no outstanding I/O to lock onto, in order to tell it to switch disks.

Well, here’s what the new system does in this case.

Read Striping Streaming - Switch

Read Striping Streaming – Switch

As you can see, we’re still streaming at a very good 5.75 MB /s and more importantly, DrivePool has all but stopped reading from I:\ because it’s very busy. Your media keeps playing without stuttering and you would never know that anything happened at all.

This functionality is brand new in this build of DrivePool and works by passively monitoring the speed at which each disk is able to read at, and at the first sign of a slow down, it will prefer to read from the faster disk.

I/O Boost

I/O Boost

I/O Boost

The last bar under the pool performance UI is I/O boost.

This feature is disabled by default, but can be enabled in pool options.

Starting with Windows Vista, Microsoft has added the notion of I/O priority to Microsoft Windows. I/O calls that are made with a higher priority are always serviced before any I/O calls made with priorities lower.

This is very important, because the slowest component of your computer is typically your hard disk, and prioritizing requests to the hard disk has a great potential to speed up your everyday computing experience.

DrivePool utilizes this functionality by boosting the priority on I/O request packets coming from the network. We only do this for reads and not for writes. DrivePool only does this for requests that are not being serviced by the read-ahead cache (which is typically the case, unless the disk is very busy).

The main goal of this feature is to reduce stuttering on streaming video, but it can also be used to improve the network I/O throughput on busy servers.

Performance Impact of the Performance UI

You may be wondering, with all this data collection going on and being processed, isn’t this going to slow down the pool?

Well, not really. The entire performance monitoring system shuts down 60 seconds after the last UI window has closed. So it’s only processing the performance data when you need to see it.

Technical Notes

Finally, I’d like to end by pointing out some technical differences between how Pool Performance and File Performance collect their performance data.

Pool Performance collects the data from the pool’s point of view. In other words this includes everything that goes on and comes off of the pool. While File Performance records everything that goes on and comes off of each disk part of the pool.

This is an important distinction because, for example, files that are cached will not show up under File Performance while Pool Performance will show activity on the pool. This is because those files are not being read from any disk, but rather from memory and so there really is no disk activity to show.

Conclusion

This is going to be the last major addition to DrivePool 2.0 BETA. Next, the existing features will be polished up and DrivePool 2.0 will ship as a final.

StableBit DrivePool 2.0.0.230 BETA

Posted in StableBit on May 3rd, 2013 by alex – Be the first to comment

StableBit DrivePool 2.0 build 230 is now out with some new additions to the Horizontal UI.

Horizontal UI

Horizontal UI

You can download the latest build here: http://stablebit.com/DrivePool/Download

Horizontal UI

To get into the horizontal UI mode, just stretch the window wide. In the horizontal UI mode, you’ll notice some new file distribution bars displayed under each disk. If you’ve ever used StableBit DrivePool 1.X, you may remember that we’ve had something similar to this in that version as well.

So what are these for, and what do they show?

The file distribution bars give you 3 pieces of information:

  1. How the files distributed on each disk, in terms of duplicated files vs. unduplicated files vs. unpooled files (and file system metadata).
  2. Whether the files will be redistributed on the next balancing pass, and how.
  3. Whether there are any new file placement limits, and for which disks.

Let’s go over each case with some screenshots.

File Distribution

File distribution is shown by displaying a bar below each disk. The bar shows file distribution as a percentage of the total disk size (just like Windows Explorer).

The bar is divided into a number of categories.

Unduplicated Files

Unduplicated Files

Unduplicated files are files that exist on the pool in an unduplicated state.

Duplicated Files

Duplicated Files

Duplicated files are files that exist on the pool, which are duplicated. Obviously, each duplicated file exists on 2 or more disks.

Other Files

Other Files

And then there’s “Other”. “Other” always confuses people, but it literally is everything else.

So what else is there?

  • Non-pooled files that exist on that disk. Remember that just because a disk is part of the pool doesn’t mean that you can’t continue to use that disk to store non-pooled files.
  • NTFS Metadata. For every file on a NTFS volume, there is additional metadata associated with that data stream, like the file name, file attributes, modification times, etc… These typically take very little disk space, but can add up if you have lots of files.
  • Directory entries. On NTFS, directory entries are actually stored as regular files with a “directory” attribute. But instead of a data stream, they contain a little database of index entries for each file (and subdirectory) that exists under them.
  • Slack space. Just because you have a 100GB volume, doesn’t mean that you can use all 100 Gigabytes of that volume to store data. NTFS divides your volume into equally sized chunks called clusters, which are typically 4096 bytes in size. If your file doesn’t fit neatly into these clusters then there’s going to be some space at the end of the file that’s wasted. We call this “slack space”.

Rebalance Markers

Above each file distribution bar you may have one or two rebalance markers displayed.

Rebalance Markers

Rebalance Markers

The rebalancing markers tell where DrivePool would prefer that the file distribution bars be, based on your balancing model.

In essence, these show you how DrivePool wants to reshuffle your files on the next balancing run.

There is a marker for duplicated files and unduplicated files, and if the rebalancing target is the same for both file types, they are combined into one.

You can hover your mouse over each rebalancing marker to see which file type it represents and how much data it wants to move.

Rebalancing Marker Tooltip

Rebalancing Marker Tooltip

Balancers

Balancers

Balancers

You can tell DrivePool how to organize your files by opening up balancing settings. StableBit DrivePool comes with reasonable defaults, so you don’t ever have to change anything here unless you want to.

For example, in the above screenshot I’ve told DrivePool that I don’t want any pooled files placed on drive E:\, unless all of the other disks are 90 % full.

As soon as you click save, the rebalancing markers will reflect your settings.

Pool Organization

Pool Organization

Pool Organization

The pool organization bar is situated towards the bottom of the window and tells you how “organized” your pool is. It turns from green to yellow to red, as the pool organization gets worse.

Pool organization improves when the system rebalances the pool.

It’s calculated based on how close the file distribution bars are to the rebalancing markers.

File Placement Limit Markers

File Placement Limit Markers

File Placement Limit Markers

The file placement limit markers show you whether there are any limits placed on new file creation.

There can be a limit on unduplicated file creation, on duplicated file creation or both. So it’s possible to see more than one file placement limit marker per disk.

All this tells you is where new files will be placed when they are copied to the pool.

If there are no limits set, or all the limits are violated, then the disk with the most free space wins.

Remove Options

That covers all of the rebalancing UI changes in this build. Let’s look at disk removal options.

Remove Options

Remove Options

When you click the “Remove” link to remove a disk from the pool, you will now be presented with these new disk removal options. Once again, DrivePool 1.X users will recognize these.

Force Damaged Drive Removal

Normally, StableBit DrivePool will not remove a drive from the pool, unless it can copy every single unduplicated file off of that disk onto some other disk in the pool. This of course makes sense, because you wouldn’t want any of your pooled files to disappear after removing a disk from the pool.

But this can be problematic when the disk that you’re removing is damaged, and some files on it are already unreadable. In this case, the removal process will be aborted with an error, and you will not be able to remove the disk from the pool.

However, with this new option enabled, DrivePool will continue to remove the disk, even though it may not be able to copy every single file off of that disk. The uncopied files will remain on the damaged disk in the (now no longer hidden) “PoolPart” folder.

Duplicate Files Later

This option is geared towards people who make extensive use of file duplication to protect their data.

When removing a disk from a pool that has duplicated files, for every duplicated file encountered, DrivePool will normally make a copy of that file onto some other disk in the pool. This ensures duplication consistency at all times, but can be time consuming.

With “Duplicate files later” enabled, DrivePool will simply skip all duplicated files on the disk being removed. If the disk only contains duplicated files, disk removal now becomes very fast.

After the disk is removed, DrivePool will reduplicate any unduplicated files in the background.

What’s Next?

That’s it for the new stuff in this build. We’ll probably see some real-time performance and pool usage statistics in the next UI update.

StableBit Scanner 2.4.0 BETA

Posted in StableBit on April 30th, 2013 by alex – 2 Comments

StableBit Scanner 2.4.0 BETA has now been out for a bit and it has some interesting new functionality that I think is worthy of a blog post.

Scan Age Map

Download your copy here: http://stablebit.com/Scanner/Download

File System Scan

Up until now, the StableBit Scanner has been designed to do everything that it can to ensure that the physical surface of your hard drives remains readable, and to recognize the first signs of mechanical wear, in order to warn you that your data might be in trouble.

But there is another way that your data can get corrupted, and that is through file system formatting damage.

When you have formatting damage on a hard drive, the hard drive itself  is capable of reading and writing every bit correctly, but the way that your files and folders are written on the disk is inconsistent (or corrupt).

Detecting File System Errors

File System Damaged

Starting with version 2.4.0 of the StableBit Scanner, it will now automatically detect file system errors by periodically scanning all of your volumes (or partitions), on each disk, for corruption.

Once damage is detected, you will get notified by all of the supported notification systems (tray icon, email, speech, or the Dashboard).

Tray Icon Notification

How it Works

The file system scan works by utilizing the built-in chkdsk utility, in Microsoft Windows.

But this wouldn’t be the StableBit Scanner if we didn’t add some bells and whistles on top of what chkdsk already provides.

  • The file system scan is automatic. It’s performed after the surface scan completes on a particular disk.
  • We support temperature equalization and overheat protection on the file system scan, just like with the surface scan. This means that if your disk is overheating due to a file system scan, the scan will be suspended.
  • Scan throttling using Background I/O is supported for a file system scan, if enabled in Scanner Settings.

In essence, the file system scan will not interfere with existing disk activity and will not overheat your disks. This is something that chkdsk alone does not offer.

Controlling the File System Scan

Scanner Settings

You can control the re-scan interval, or completely disable the file system scan from Scanner Settings.

Disk Settings

Or you can disable the file system scan on a per-disk basis.

File System Repair

Alright, so you’ve got file system damage, now what?

You can quickly and easily repair the file system damage from within the StableBit Scanner.

File System Status

Just select the damaged disk and click on the link in the Status panel.

Repair Volume

Then just click Repair Volume to begin the repair.

After a few minutes you will get the confirmation that the file system was repaired.

Volume Repaired

Locked Volumes

There are a couple of issues that can complicate the repair process. In order to repair a file system on a volume, Microsoft Windows requires that no files be open on that volume.

If there are file open, you have the choice of force closing them.

If this is the case, then the StableBit Scanner will present you with the following dialog:

Force Repair Volume

The dialog warns you to save all of your work before continuing, because the volume will not be accessible to other application while it is being repaired.

There is yet another possibility where Microsoft Windows is not able to force close all the files on a volume. For example, if the Operating System itself is running on the damaged volume, or there is a page file on that volume.

In this case the StableBit Scanner will automatically schedule the repair operation for the next reboot.

Force Repair on Reboot

The next time that you restart the computer, it will repair the damaged file system before Windows starts.

Sector Scan Age

Alright, that covers the new file system scan and repair feature. Let’s switch gears and talk about sector scan age.

One of the main functions of the StableBit Scanner is to re-scan the entire surface of each disk for errors. This serves 2 very important functions:

  1. It “refreshes” the entire surface of the disk, in order to prevent what is often termed as “bit rot”. Essentially it tells the drive to re-assign weak sectors before they get a change to become completely unreadable.
  2. It detects instances where the on-disk built-in error recovery system has failed to recover a damaged sector, which essentially means that some previously written data was lost. This is a great indicator that your drive is failing.

But before StableBit Scanner 2.4.0, there was no way to visually tell whether a disk’s surface is actually being re-scanned in a timely manner.

Sector Scan Age

Starting with StableBit Scanner 2.4.0 you can now easily tell when sectors on a disk have not been recently scanned.

Each sector will slowly turn from green to yellow as its last scan time becomes older.

I think that this is pretty neat new feature, so I thought that I’d mention it here. It was actually inspired by the way plants slowly turn from green to yellow if you forget to water them. Thus giving you a clear indication that something is not right.

If you have any suggestions on how to make these features better, please send them this way: https://stablebit.com/Contact

Thanks for reading.