StableBit

StableBit DrivePool 2.0.0.310 BETA – File System Improvements

Posted in StableBit on July 1st, 2013 by alex – 2 Comments
StableBit DrivePool Build 310

StableBit DrivePool Build 310

StableBit DrivePool is now on the road to a stable release final. The latest build doesn’t feature much visual changes but instead is focused on refining the file system component of DrivePool.

We’re still not done with tweaking the UI, it’s just not the focus of this build.

Download the latest build here: http://stablebit.com/DrivePool/Download

CoveFS Changes

CoveFS is our high performance kernel driver written specifically to pool multiple file systems together. In this build, it has many changes to ensure compatibility with 3rd party applications, in particular Enterprise grade applications such as database engines, virtual machines and essentially any applications that rely on very heavy concurrent I/O on the pool.

For example, here is what we are now able to do with this latest build out of the box:

  • StableBit DrivePool can now be checked out and built and deployed directly from the pool using SVN, Visual Studio 2012, MSBuild and some proprietary deployment scripts.
  • All of the Virtual Machines used for testing now run from the pool. This means that they can instantly benefit from the performance gains obtained through read striping, and that they are protected from drive failure by file duplication.
    • The Virtual Machine software used is VMWare Workstation 7.1.4 and Oracle’s VirtualBox 4.2.12.

A lot of work went into putting in the file system changes necessary for this particular build and I’ll talk about some of these changes below.

Report as NTFS by Default

After the many months of testing the DrivePool 2.0 BETA, it has become apparent that we can no longer report our file system as “CoveFS”. There are simply too many applications out there that check specifically for “NTFS”, and if they don’t find that string then they either refuse to run outright or assume that the file system is FAT based (which is typically limited to 2GB/4GB file sizes).

The latest example of this is VMWare Workstation 7. It needs the file system to be “NTFS” or else it assumes a 2GB file limit, and refuses to run. Windows 8 also has hard coded strings for “expected” file systems in some of the (reverse engineered) system code, and if a file system is not on that list it misbehaves in subtle ways.

So in the latest build, CoveFS now reports as “NTFS” by default. You can still revert to reporting as “CoveFS” by editing the service .config file.

Quirky File System Behavior

Some things in Windows are just wacky, and programs have been built to expect this wackiness or else they break. So we have to do our best to reproduce this wackiness in CoveFS, in order to maintain the best possible compatibility with these programs.

Here’s an example:

Normally in Windows, you can rename a file (or folder), even if that file is currently in use. Typically applications use this behavior to rename a file that can’t be deleted because something else is using it, perhaps you’ve even used this trick yourself. The Windows Home Server 2011 did this when you requested to move a folder to a different disk, but that folder was in use. I’m sure that there are countless other examples.

What’s the wacky part?

You can also tell Windows to open a file that’s in use and rename it to itself, while requesting to explicitly overwrite the destination.

Now how does that make any sense at all? How can you rename a file to itself, much less overwrite the destination (which is itself, and is in use by something else)?

In this case CoveFS used to tell the caller that the destination is in use and can’t be overwritten. However, NTFS just does this with no complaints. So now that’s what CoveFS does as well.

It’s these kinds of issues that make a huge difference in compatibility. This particular problem was causing VMWare to not function correctly from the pool.

Locking / Caching

One of the most important parts of any file system is how it interacts with the memory manager and the cache manager.

Locking and caching affects how EXE files are opened, how data is cached when read from or written to the disk, and how memory mapping works. Memory mapping is typically used by database or backup applications, word processors, or anything that needs to access random areas of very large files quickly.

The latest build drastically simplifies how DrivePool interacts with all of these core systems.

Our new model is much simpler. Eliminating complexity means eliminating potential for problems, so I think that this is a very important change.

File Performance was Broken

The idea behind file performance was a small pane to answer the question “what is going on on the pool?”. It was supposed to show you the top 5 most active file transfers in real-time.

The original back-end file performance implementation was written using ETW (the same thing that the Windows Resource Monitor uses).

Unfortunately it had some problems:

  • It was kind of slow because it was difficult to parse the required I/O in real-time.
  • Ongoing I/O that was started prior to you opening up the performance UI pane would not be shown.
  • There was a 2 second delay before we could show you the file I/O.
  • Some I/O was missed outright due to the cache manager’s lazy writer / read ahead functionality.
  • What was shown in File Performance was not in sync with what was shown in Pool Performance (I’ve alluded to this when I first introducing this feature).

Introducing Disk Performance

Well, the old code that handled File Performance was scrapped.

A completely new high performance design was created that works intimately with the CoveFS file system to give you instant file performance.

And now it’s called Disk Performance.

From the user’s point of view, the UI is similar except that it’s more accurate and timely.

But here are the functional differences:

  • Performance data is sampled every one second and is not delayed. You are seeing near real-time activity and not a historic average.
  • All I/O that goes through the pool is recorded. Cached, uncached and Fast I/O is always included. We sample everything.
  • Performance sampling is very efficient. There is no need to parse anything. The file system keeps track of all of the statistics and gives us exactly what we request.
  • There is virtually no overhead. The file system only keeps samples of file performance if it was queried for the data within the past 60 seconds.

Accuracy

We now have much greater accuracy.

Disk Performance Accuracy

Disk Performance Accuracy

For example, in the above screen you can see an ongoing throttled file copy. The file copy was throttled to 1.0 MB/s using SyncBack SE.

You can see that the number reported by DrivePool is very close to 1.0 MB/s and that the Pool Performance statistics are now completely consistent with Disk Performance.

Live Transfers

Because of the increased responsiveness of the new design, DrivePool can now actually show you whether the file transfer listed in the Disk Performance pane was ended recently.

Disk Activity

Historic Disk Activity

For example, in the above screenshot we can see that all of the disk activity is grayed out. This means that the activity is Historic. In other words, the disk activity has already ended by the time that we’ve asked for the sample. By default DrivePool samples performance data once per second (which can be tweaked in the service .config).

Windows 8.1 / Windows Server 2012 R2

The current build of DrivePool is not compiled for the Windows 8.1 kernel, so it’s not compatible with the Windows 8.1 preview that Microsoft released last week.

While I would have liked to build a Windows 8.1 compatible version of DrivePool for this release, unfortunately the development tools that Microsoft released are not functioning correctly on the build machine (well, they’re crashing).

To be fair these are “Preview” tools and I’m sure that they will fix these issues before the final.

So I’d just like to mention that Windows 8.1 support is coming to DrivePool.

Other Issues / Feature Requests

There are a number of other issues and feature requests that people have been making.

For example:

  • Reparse points are currently not supported on the pool. This means that you can’t create symbolic links on the pool. Stay tuned, this is being worked on.
  • The Microsoft NFS Server doesn’t work from the pool on Windows Server 2012. This is also being addressed. Perhaps the 2012 NFS server requires reparse point support, I don’t know at this point. But it will be addressed.

The next build of DrivePool 2.X will concentrate on fixing / refining any remaining issues in the service and the UI.

Areas that are of interest are:

  • Refining the per-folder duplication UI.
  • Testing remote control more thoroughly and fixing any remaining issues there.
  • Showing better error messages when adding / removing disks fails.

You can download the latest build of StableBit DrivePool here: http://stablebit.com/DrivePool/Download

Until next time.

StableBit Scanner 2.4 Release Final

Posted in StableBit on June 18th, 2013 by alex – Be the first to comment
StableBit Scanner 2.4

StableBit Scanner 2.4

The release final of the StableBit Scanner version 2.4 is now available for download.

Get it here: http://stablebit.com/Scanner/Download

Since I’ve already talked about some of the new features in this version in previous blog posts, I’ll just quickly go over them again without getting into too much detail.

Then we’ll talk about what’s new since my last post.

Automatic File System Checking

File System Status

File System Status

In addition to testing your disks for physical damage, the StableBit Scanner will now check the file systems on your disks for formatting errors.

Repair Volume

Repair Volume

If damage is found, you can easily repair the volume right from the StableBit Scanner’s UI.

Scanned Sector Aging

Sector Scan Age

Scanner Sectors

Disk sectors that are not being re-scanned in a timely manner will turn yellow. The more “past-due” they are for a scan, the more yellow they will get.

Better Handling of Removable Media

Settings - Scanner

Settings – Scanner

This is a new feature that has been added recently, and I haven’t blogged about it.

Whenever you insert a USB flash drive or a USB hard drive, by default, the StableBit Scanner will not start scanning it immediately. You can turn off automatic scanning of removable media permanently or you can have the StableBit Scanner start scanning the device only if it has been plugged in for some hours.

The StableBit Scanner was originally designed to be run on Servers. Because you wouldn’t typically be inserting and removing USB thumb drives or hard drives from a server, there really was no need to limit automatic scanning in any way.

In fact, the original thought behind this behavior was that it would allow you to set up a disk scanning workstation. You could simply connect a new disk whenever you would want to scan it, and the Scanner would immediately start checking that disk for problems with no user intervention.

While it is still possible to use the StableBit Scanner for this scenario, it is definitely not the typical use case. So the default settings will now never scan USB thumb drives automatically, and will begin scanning USB hard drives only if they’ve been plugged in for at least 1 hour.

Past Due Scanning

Past Due Settings

Past Due Settings

This is a very simple addition that makes a big difference.

If you’ve setup your settings to only scan within a time window, it’s possible for your scans to become “stale”. In other words, your disks are not being scanned in a timely manner.

This can happen because perhaps the computer was too busy at the scan time and the StableBit Scanner has throttled your existing scans, or if you simply have too many disks. If either of these are the case, then you could get into a situation where the StableBit Scanner is not given enough time in order to maintain the health of your disks and properly check them for health.

With this new feature, the StableBit Scanner will now be given permission to scan your disks outside of the time window, if the last scan time of any disk is “past due” by some number of days (that you specify).

Scan Progress Recovery

Here’s another small feature that is pretty useful.

If your computer crashes or loses power in the middle of a surface scan, the StableBit Scanner will now be able to recover its last scan progress on the next reboot. Before this, you would lose your scan progress and would have to rescan that disk from the beginning.

Fahrenheit

Settings - Heat

Settings – Heat

Those of you who live in countries that don’t use Celsius as a measure of temperature may be wondering, why in the world is the Scanner always reporting the drive temperatures in Celsius?

There are really 2 reasons:

  • That’s how the drive reports it.
  • In engineering fields, thermal limit specifications are typically reported in Celsius.

But due to popular demand, you can now choose your own temperature scale. We default this setting to whatever the OS prefers in your region.

Miscellaneous

There are many other miscellaneous changes and fixes, which I didn’t mention here.

You can check out the full change log on the Download page.

Thanks for reading.

StableBit DrivePool 2.0.0.256 BETA – Performance UI

Posted in StableBit on May 18th, 2013 by alex – 2 Comments

The latest BETA of StableBit DrivePool 2.0 adds some major new back-end features, improved read striping and a new Performance UI.

Performance UI

Performance UI

Get the latest BETA here: http://stablebit.com/DrivePool/Download

The Performance UI

The new performance UI lets you see exactly what’s happening on the pool, at a glance. It shows you the most important information in real time.

Of course, it works perfectly with the remote control feature, letting you connect to any other machine on your network that is running StableBit DrivePool, and see exactly how the pool is performing on that machine.

Let’s take a look at the new UI in detail, and what all the visual cues actually mean.

File Performance

File Performance

File Performance

The performance UI is accessible from the Horizontal UI (and not the Vertical UI).  To get into the horizontal UI, simply stretch the window wide.

As you can see from the screenshot above, the performance UI is split up into two categories, File Performance  and Pool Performance.

File performance is designed to show you the top 5 file operations on the pool, that are consuming the most resources.

It shows you the read and write speeds of each file operation.

File Performance Speed

File Performance Speed

But in addition to the speed, it shows you something that’s even more useful,  the disk activity percentage.

Disk Activity Bars

Disk Activity Bars

In fact, the list shows you the top 5 file transfers on the pool, with the highest disk activity. The longer the bar, the more “busy” the disk is.

Let me explain what that means. Every hard disk or SSD can only read and write so fast. In addition drives have different performance characteristics. For example, SSDs can jump around between different parts of the disk (i.e. seek) faster than hard drives.

This means that the maximum speed that any drive can read or write at, depends highly on the type of drive it is, and what file operations are being performed on it.

In short, the disk activity bar shows you how busy that disk is, considering all those factors.

Performance UI

Performance UI

So reading this performance chart we see that:

  • Disk I:\ is very busy, and DrivePool is reading from it at 96 KB /s
  • Disk F:\ is not so busy and DrivePool is reading from it at 12.2 MB/s

DrivePool happens to be reading the same file from both disks, which is called read striping, but more on this later.

Pool Performance

Pool Performance

Pool Performance

On the left of the performance UI, you will see Pool Performance.

In pool performance, you can see:

  • The total combined read and write speeds that are occurring on the pool at this time.
  • Out of all the read / write operations, how many of those are utilizing Fast I/O.
  • Out of all the read operations, how many are utilizing read striping, and what algorithm is being used to stripe the reads (more on this later).
  • And finally, out of all the read operations, how many are being prioritized by Network I/O Boost.

Let’s go over the 3 vertical bars in detail, in order to understand exactly what they mean.

Fast I/O

Fast I/O - Write

Fast I/O – Write

The notion of Fast I/O is something that Microsoft coined for a technology that is designed to bypass the file system and retrieve your data directly from the Windows cache.

The reality of the situation, is that this is not the case. In fact, Fast I/O can and mostly does interact with the filesystem and disk (albeit in a roundabout way).

A better way to think about Fast I/O is I/O that requires no additional processing and bypasses some of the overhead that is imposed on regular I/O. For example, when DrivePool is performing any kind of complex manipulation of the I/O (such as read striping or I/O prioritization) it will bypass Fast I/O.

Saving every CPU cycle was important back in the heyday of Windows NT, because computers were slow and every CPU cycle counted. But this may not be as relevant today.

Read Striping

Read striping is a feature in DrivePool that is designed to improve the performance of reading duplicated files from the pool. In essence it can read from multiple disks at the same time in order to improve (or maintain) performance.

Read Striping

Read Striping

Read striping in DrivePool actually operates in 2 different modes:

  • Bulk transfer – For sequential access of large files.
  • Optimal disk – For streaming and random access of data.

A mode is chosen when a file is first opened on the pool, based on the hints that the application gives us.

DrivePool shows you exactly what each read striping algorithm is doing in real-time, by coloring the read striping bar with 3 different colors.

Read Striping Colors

Read Striping Colors

Here’s what these mean:

  • Hold – During a bulk transfer, DrivePool has decided that it is best to hold on to the current disk and not switch over to another disk.
  • Switch – During a bulk transfer, DrivePool has decided that it is best to switch to another disk while the current disk services its outstanding requests.
  • Optimal – During random file access or media streaming, DrivePool has chosen the optimal disk based on outstanding I/O requests, and an adaptive algorithm designed to pick the fastest disk at this time (this is new).
Bulk Transfers
Read Striping

Read Striping

This is what a typical bulk transfer looks like.

Here we can see that DrivePool is reading from 2 disks at the same time (I:\ and E:\), and that 100% of the read requests are being processed by the read striping algorithm. Most I/O requests are going to the same disk, and the disk context is being switched very rarely. In addition, we can see that both disks are saturated (this is exactly what we want).

Media Streaming (or Random Access)

This part is brand new and very cool.

Read Striping Streaming

Read Striping Streaming

Let’s say for example that you’re streaming a 50 GB uncompressed Blu-Ray that is in a duplicated folder, with read striping enabled.

So far, everything is going according to plan. Read striping has switched into streaming mode and your data is being read from 2 disks at the same time (I:\ and F:\), and both disks are not busy at all.

Now, let’s say some process starts on disk I:\, something that completely saturates the disk. Normally at this point your media would start to stutter very badly. And because the media player is reading the data rather slowly, the standard bulk mode read striping algorithm would have no outstanding I/O to lock onto, in order to tell it to switch disks.

Well, here’s what the new system does in this case.

Read Striping Streaming - Switch

Read Striping Streaming – Switch

As you can see, we’re still streaming at a very good 5.75 MB /s and more importantly, DrivePool has all but stopped reading from I:\ because it’s very busy. Your media keeps playing without stuttering and you would never know that anything happened at all.

This functionality is brand new in this build of DrivePool and works by passively monitoring the speed at which each disk is able to read at, and at the first sign of a slow down, it will prefer to read from the faster disk.

I/O Boost

I/O Boost

I/O Boost

The last bar under the pool performance UI is I/O boost.

This feature is disabled by default, but can be enabled in pool options.

Starting with Windows Vista, Microsoft has added the notion of I/O priority to Microsoft Windows. I/O calls that are made with a higher priority are always serviced before any I/O calls made with priorities lower.

This is very important, because the slowest component of your computer is typically your hard disk, and prioritizing requests to the hard disk has a great potential to speed up your everyday computing experience.

DrivePool utilizes this functionality by boosting the priority on I/O request packets coming from the network. We only do this for reads and not for writes. DrivePool only does this for requests that are not being serviced by the read-ahead cache (which is typically the case, unless the disk is very busy).

The main goal of this feature is to reduce stuttering on streaming video, but it can also be used to improve the network I/O throughput on busy servers.

Performance Impact of the Performance UI

You may be wondering, with all this data collection going on and being processed, isn’t this going to slow down the pool?

Well, not really. The entire performance monitoring system shuts down 60 seconds after the last UI window has closed. So it’s only processing the performance data when you need to see it.

Technical Notes

Finally, I’d like to end by pointing out some technical differences between how Pool Performance and File Performance collect their performance data.

Pool Performance collects the data from the pool’s point of view. In other words this includes everything that goes on and comes off of the pool. While File Performance records everything that goes on and comes off of each disk part of the pool.

This is an important distinction because, for example, files that are cached will not show up under File Performance while Pool Performance will show activity on the pool. This is because those files are not being read from any disk, but rather from memory and so there really is no disk activity to show.

Conclusion

This is going to be the last major addition to DrivePool 2.0 BETA. Next, the existing features will be polished up and DrivePool 2.0 will ship as a final.

StableBit DrivePool 2.0.0.230 BETA

Posted in StableBit on May 3rd, 2013 by alex – Be the first to comment

StableBit DrivePool 2.0 build 230 is now out with some new additions to the Horizontal UI.

Horizontal UI

Horizontal UI

You can download the latest build here: http://stablebit.com/DrivePool/Download

Horizontal UI

To get into the horizontal UI mode, just stretch the window wide. In the horizontal UI mode, you’ll notice some new file distribution bars displayed under each disk. If you’ve ever used StableBit DrivePool 1.X, you may remember that we’ve had something similar to this in that version as well.

So what are these for, and what do they show?

The file distribution bars give you 3 pieces of information:

  1. How the files distributed on each disk, in terms of duplicated files vs. unduplicated files vs. unpooled files (and file system metadata).
  2. Whether the files will be redistributed on the next balancing pass, and how.
  3. Whether there are any new file placement limits, and for which disks.

Let’s go over each case with some screenshots.

File Distribution

File distribution is shown by displaying a bar below each disk. The bar shows file distribution as a percentage of the total disk size (just like Windows Explorer).

The bar is divided into a number of categories.

Unduplicated Files

Unduplicated Files

Unduplicated files are files that exist on the pool in an unduplicated state.

Duplicated Files

Duplicated Files

Duplicated files are files that exist on the pool, which are duplicated. Obviously, each duplicated file exists on 2 or more disks.

Other Files

Other Files

And then there’s “Other”. “Other” always confuses people, but it literally is everything else.

So what else is there?

  • Non-pooled files that exist on that disk. Remember that just because a disk is part of the pool doesn’t mean that you can’t continue to use that disk to store non-pooled files.
  • NTFS Metadata. For every file on a NTFS volume, there is additional metadata associated with that data stream, like the file name, file attributes, modification times, etc… These typically take very little disk space, but can add up if you have lots of files.
  • Directory entries. On NTFS, directory entries are actually stored as regular files with a “directory” attribute. But instead of a data stream, they contain a little database of index entries for each file (and subdirectory) that exists under them.
  • Slack space. Just because you have a 100GB volume, doesn’t mean that you can use all 100 Gigabytes of that volume to store data. NTFS divides your volume into equally sized chunks called clusters, which are typically 4096 bytes in size. If your file doesn’t fit neatly into these clusters then there’s going to be some space at the end of the file that’s wasted. We call this “slack space”.

Rebalance Markers

Above each file distribution bar you may have one or two rebalance markers displayed.

Rebalance Markers

Rebalance Markers

The rebalancing markers tell where DrivePool would prefer that the file distribution bars be, based on your balancing model.

In essence, these show you how DrivePool wants to reshuffle your files on the next balancing run.

There is a marker for duplicated files and unduplicated files, and if the rebalancing target is the same for both file types, they are combined into one.

You can hover your mouse over each rebalancing marker to see which file type it represents and how much data it wants to move.

Rebalancing Marker Tooltip

Rebalancing Marker Tooltip

Balancers

Balancers

Balancers

You can tell DrivePool how to organize your files by opening up balancing settings. StableBit DrivePool comes with reasonable defaults, so you don’t ever have to change anything here unless you want to.

For example, in the above screenshot I’ve told DrivePool that I don’t want any pooled files placed on drive E:\, unless all of the other disks are 90 % full.

As soon as you click save, the rebalancing markers will reflect your settings.

Pool Organization

Pool Organization

Pool Organization

The pool organization bar is situated towards the bottom of the window and tells you how “organized” your pool is. It turns from green to yellow to red, as the pool organization gets worse.

Pool organization improves when the system rebalances the pool.

It’s calculated based on how close the file distribution bars are to the rebalancing markers.

File Placement Limit Markers

File Placement Limit Markers

File Placement Limit Markers

The file placement limit markers show you whether there are any limits placed on new file creation.

There can be a limit on unduplicated file creation, on duplicated file creation or both. So it’s possible to see more than one file placement limit marker per disk.

All this tells you is where new files will be placed when they are copied to the pool.

If there are no limits set, or all the limits are violated, then the disk with the most free space wins.

Remove Options

That covers all of the rebalancing UI changes in this build. Let’s look at disk removal options.

Remove Options

Remove Options

When you click the “Remove” link to remove a disk from the pool, you will now be presented with these new disk removal options. Once again, DrivePool 1.X users will recognize these.

Force Damaged Drive Removal

Normally, StableBit DrivePool will not remove a drive from the pool, unless it can copy every single unduplicated file off of that disk onto some other disk in the pool. This of course makes sense, because you wouldn’t want any of your pooled files to disappear after removing a disk from the pool.

But this can be problematic when the disk that you’re removing is damaged, and some files on it are already unreadable. In this case, the removal process will be aborted with an error, and you will not be able to remove the disk from the pool.

However, with this new option enabled, DrivePool will continue to remove the disk, even though it may not be able to copy every single file off of that disk. The uncopied files will remain on the damaged disk in the (now no longer hidden) “PoolPart” folder.

Duplicate Files Later

This option is geared towards people who make extensive use of file duplication to protect their data.

When removing a disk from a pool that has duplicated files, for every duplicated file encountered, DrivePool will normally make a copy of that file onto some other disk in the pool. This ensures duplication consistency at all times, but can be time consuming.

With “Duplicate files later” enabled, DrivePool will simply skip all duplicated files on the disk being removed. If the disk only contains duplicated files, disk removal now becomes very fast.

After the disk is removed, DrivePool will reduplicate any unduplicated files in the background.

What’s Next?

That’s it for the new stuff in this build. We’ll probably see some real-time performance and pool usage statistics in the next UI update.

StableBit Scanner 2.4.0 BETA

Posted in StableBit on April 30th, 2013 by alex – 2 Comments

StableBit Scanner 2.4.0 BETA has now been out for a bit and it has some interesting new functionality that I think is worthy of a blog post.

Scan Age Map

Download your copy here: http://stablebit.com/Scanner/Download

File System Scan

Up until now, the StableBit Scanner has been designed to do everything that it can to ensure that the physical surface of your hard drives remains readable, and to recognize the first signs of mechanical wear, in order to warn you that your data might be in trouble.

But there is another way that your data can get corrupted, and that is through file system formatting damage.

When you have formatting damage on a hard drive, the hard drive itself  is capable of reading and writing every bit correctly, but the way that your files and folders are written on the disk is inconsistent (or corrupt).

Detecting File System Errors

File System Damaged

Starting with version 2.4.0 of the StableBit Scanner, it will now automatically detect file system errors by periodically scanning all of your volumes (or partitions), on each disk, for corruption.

Once damage is detected, you will get notified by all of the supported notification systems (tray icon, email, speech, or the Dashboard).

Tray Icon Notification

How it Works

The file system scan works by utilizing the built-in chkdsk utility, in Microsoft Windows.

But this wouldn’t be the StableBit Scanner if we didn’t add some bells and whistles on top of what chkdsk already provides.

  • The file system scan is automatic. It’s performed after the surface scan completes on a particular disk.
  • We support temperature equalization and overheat protection on the file system scan, just like with the surface scan. This means that if your disk is overheating due to a file system scan, the scan will be suspended.
  • Scan throttling using Background I/O is supported for a file system scan, if enabled in Scanner Settings.

In essence, the file system scan will not interfere with existing disk activity and will not overheat your disks. This is something that chkdsk alone does not offer.

Controlling the File System Scan

Scanner Settings

You can control the re-scan interval, or completely disable the file system scan from Scanner Settings.

Disk Settings

Or you can disable the file system scan on a per-disk basis.

File System Repair

Alright, so you’ve got file system damage, now what?

You can quickly and easily repair the file system damage from within the StableBit Scanner.

File System Status

Just select the damaged disk and click on the link in the Status panel.

Repair Volume

Then just click Repair Volume to begin the repair.

After a few minutes you will get the confirmation that the file system was repaired.

Volume Repaired

Locked Volumes

There are a couple of issues that can complicate the repair process. In order to repair a file system on a volume, Microsoft Windows requires that no files be open on that volume.

If there are file open, you have the choice of force closing them.

If this is the case, then the StableBit Scanner will present you with the following dialog:

Force Repair Volume

The dialog warns you to save all of your work before continuing, because the volume will not be accessible to other application while it is being repaired.

There is yet another possibility where Microsoft Windows is not able to force close all the files on a volume. For example, if the Operating System itself is running on the damaged volume, or there is a page file on that volume.

In this case the StableBit Scanner will automatically schedule the repair operation for the next reboot.

Force Repair on Reboot

The next time that you restart the computer, it will repair the damaged file system before Windows starts.

Sector Scan Age

Alright, that covers the new file system scan and repair feature. Let’s switch gears and talk about sector scan age.

One of the main functions of the StableBit Scanner is to re-scan the entire surface of each disk for errors. This serves 2 very important functions:

  1. It “refreshes” the entire surface of the disk, in order to prevent what is often termed as “bit rot”. Essentially it tells the drive to re-assign weak sectors before they get a change to become completely unreadable.
  2. It detects instances where the on-disk built-in error recovery system has failed to recover a damaged sector, which essentially means that some previously written data was lost. This is a great indicator that your drive is failing.

But before StableBit Scanner 2.4.0, there was no way to visually tell whether a disk’s surface is actually being re-scanned in a timely manner.

Sector Scan Age

Starting with StableBit Scanner 2.4.0 you can now easily tell when sectors on a disk have not been recently scanned.

Each sector will slowly turn from green to yellow as its last scan time becomes older.

I think that this is pretty neat new feature, so I thought that I’d mention it here. It was actually inspired by the way plants slowly turn from green to yellow if you forget to water them. Thus giving you a clear indication that something is not right.

If you have any suggestions on how to make these features better, please send them this way: https://stablebit.com/Contact

Thanks for reading.

Covecube Welcomes Christopher Courtney

Posted in Covecube, StableBit on April 16th, 2013 by alex – Be the first to comment

Christopher Courtney has just joined the Covecube team to help out with technical support.

He is a MVP award winner and has around 12 thousand posts on the we got served forum and is a lead moderator there, so he’s definitely not new to answering technical questions.

Our support portal for stablebit.com is at: https://stablebit.com/Contact

Welcome Christopher, glad to have you aboard.

And a big thank you to Dave McCabe and Jim Collison of the Home Server Show (http://homeservershow.com/) who made this possible.

StableBit.com – New Payment Method

Posted in StableBit on March 25th, 2013 by alex – Be the first to comment

StableBit.com – Stripe

We’ve introduced a new payment method over on StableBit.com, courtesy of stripe.com.

It’s now easier than ever to purchase our products. Just enter your name and credit card number and that’s it.

No need to create any accounts anywhere. As usual, we never see or store your credit card number here. Everything is done securely over SSL, so you don’t have to worry about bad guys intercepting your data.

StableBit DrivePool 1.3 Release Final

Posted in StableBit on March 18th, 2013 by alex – Be the first to comment

StableBit DrivePool 1.3.0.7534 Release Final is now available for download onto your Windows Home Server 2011.

It features multiple pool support and a few other things, along with some fixes and optimizations.

DrivePool – Add to a new pool

Download it here: http://www.stablebit.com/DrivePool/Download

What’s New?

The major new feature in this release is support for creating more than one pool. But there are a few other interesting things thrown in as well. Let’s take a look at what’s new.

Multiple Pool Support

You can now easily create more than one virtual pool using StableBit DrivePool.

Since I’ve already described this functionality in some detail in a previous blog post (http://blog.covecube.com/2012/11/stablebit-drivepool-1-3-beta-now-available-with-multiple-pool-support), I’ll just summarize.

  • You can now easily add any new drive to a new pool, which will spawn a new virtual drive letter.
  • Once you have multiple pools, when adding another drive to the pool, you will have the option of selecting which pool you’d like that drive added to.
  • When you remove the last disk from any pool, that pool is automatically unmounted.

Every pool has its own balancing settings, so you can set up different balancing behaviors for each pool.

That’s about it, it’s pretty simple from the user’s perspective. If you don’t need this functionality, the only UI change that you will notice is a new check box on the “add drive to pool” wizard.

Disk Identification

In DrivePool 1.3, we’ve added a better means of identifying missing and pooled disks.

DrivePool – Pool Tab

On the Pool tab, you can now enable additional columns that show you the model and serial number for each disk. These are retrieved from the StableBit Scanner, if you have that installed for added accuracy. Otherwise, they are retrieved from Windows.

DrivePool – Identify Missing Disk

When a disk goes missing, you now have an option to bring up a disk identification window. DrivePool will tell you as much information as it knew about the disk before it went missing. If the StableBit Scanner was installed at the time, you will also get the case and bay of the missing drive (if you’ve configured that in Disk Settings).

In addition, you can now tell DrivePool to “Ping” all connected disks for 1 minute. This will light up the disk access lights on the disks that are still connected, letting you find the missing disk by visual inspection.

Removing Damaged Disks

Previous to DrivePool 1.3, if one of your disks was damaged and it had one or more un-duplicated pooled files on it that was unreadable, then you could not remove that drive from the pool using the standard removal wizard.

The recommended procedure was to shut down the server and simply disconnect the damaged disk.

We have this documented in the manual: http://www.stablebit.com/Support/DrivePool/Manual/RemoveDrive

DrivePool 1.3 now has a new option, that will tell the remove drive from pool wizard to skip any unreadable files.

DrivePool – Remove Options

This gives you more flexibility when removing damaged drives from the pool.

Of course any unreadable un-duplicated files will no longer appear on the pool, but will remain on the removed disk.

Dashboard Optimizations

StableBit DrivePool 1.3 features some improvements in the Dashboard UI code.

A lot of time was spent optimizing DrivePool’s UI code for this release. It has gone through rigorous CPU, Memory, GPU testing and profiling.

We’ve gotten rid of any extra redraw cycles and even optimized all the animations for terminal services.

Conclusion

This should be a great new release for the Windows Home Server 2011 fans. I hope that you enjoy it.

Until next time.

StableBit DrivePool 2.0 BETA – Remote Control

Posted in StableBit on February 19th, 2013 by alex – Be the first to comment

The latest version of the StableBit DrivePool 2.0 BETA now has a new feature, remote control.

Remote Control

Download it here: http://stablebit.com/DrivePool/Download

This one check box fires up the remote control system and allows you to effortlessly and securely connect and manage other computers that are running StableBit DrivePool 2.0 on your local network, and vice versa.

With remote control, it’s now possible to manage your pool from multiple computers on your local network. You can even have many copies of StableBit DrivePool running on different computers and manage them all from one central location. It’s up to you to decide how you want to use this feature.

Managing a Remote Computer

We’ve tried to make things as simple as possible, so as soon as you enable remote control, StableBit DrivePool will quickly enumerate all the other computers on your local private network that have StableBit DrivePool installed (and remote control enabled).

You can then just select the computer that you want to manage from a drop down list, in the new computers bar that will appear.

List Computers

After selecting the destination computer, in a few moments the management UI will log you into the remote computer.

Managing a Remote Computer

The fact that you’re managing a remote computer is indicated by the pulsing blue bar at the top.

The connection is very fast, encrypted and authentication is done using standard Windows services (NTLM via. SSPI).

StableBit DrivePool doesn’t use terminal services for this, and it’s actually possible to connect to the same computer from more than one locations at the same time. All the connected management UIs will be completely synchronized.

Logging in as a Different User

If you already have an account on the destination computer, and that account is an Administrator, the connection just works. There is no need to enter your user name or password.

But if you don’t have an account on that computer, or your account is not a member of the built-in Administrators group then StableBit DrivePool will prompt you for a user name and password.

Log in as a Different User

Don’t worry about entering your credentials, we never store or send them in the clear and we use standard Microsoft APIs to protect your credentials.

Enter your Credentials

You should recognize this prompt, it’s a standard Windows authentication prompt.

If you don’t choose the option to remember your credentials, then you will be prompted for them again the next time that you restart StableBit DrivePool.

If you do select the Remember my credentials check box, then we will encrypt your password and store it in the Credential Manager provided by Windows. This is done using the standard Credential Management API, in order to secure your stored credentials.

Credential Manager

Limitations

When you’re connected to a remote copy of StableBit DrivePool, then you can do almost everything that you can on your local copy, including toggling Remote Control on and off. Obviously, if you turn it off on the remote computer, then you won’t be able to re-connect to it again after disconnecting.

There are a few things that you can’t do remotely, activating a license or performing an application update will require you to log in locally into that computer.

Licensing

Licensing is pretty straightforward with remote control.

In order to create a pool on a computer, you will need to activate a license on the computer that the pool is being created on.

But if you just want to use a copy of StableBit DrivePool exclusively for remote controlling other machines on your LAN, then that copy doesn’t require activation.

How it Works

Now that I’ve described how the new feature can be used, let’s get a little technical and talk a bit about how this all works in terms of networking and security.

Networking

The new remote control layer in StableBit DrivePool consists of 3 parts, in terms of networking:

  1. Discovery.
  2. Keep-alive.
  3. Remoting.
Let’s talk about each one and what they do.
Discovery

In order to discover other machines running StableBit DrivePool on the local network, DrivePool will send out multicast UDP packets to a particular multicast group / port combination. Any copies of DrivePool on the LAN (with remote control enabled) will pick these up and remember the machine that sent them as a potential candidates to connect to.

In addition, when you shut down your computer, a special shutdown message is sent to inform everyone that the computer is shutting down and it should be removed from the list of available remote control targets.

The multicast packets themselves are tiny, they only contain the DrivePool identifier, a 2 byte packet type and an optional 4 byte TCP port number that will be used later to establish a secure connection with this computer. This keeps network congestion to a minimum.

Keep-alive

Keep-alive is the process of ensuring that a remote computer is still online and is able to accept remote connections.

After discovery is complete, the keep-alive system will probe the destination for the keep-alive port. The port will typically be 27525, but can change in case that port is already in use by another application.

Once the keep-alive port is found, we verify that it belongs to StableBit DrivePool by issuing a simple ping / pong using UDP.

The keep-alive system will then periodically ping all the remote computers that we know about to make sure that they’re still online.

Remoting

Remoting is the process of connecting to the remote StableBit DrivePool service. The remote server is actually running on a secure .NET remoting  TCP channel, so we didn’t re-invent the wheel here. The TCP port is typically 27525, but it can change if that port is already in use.

The system knows the actual port from the Discovery process, as described above.

Firewall

StableBit DrivePool will automatically configure the Windows Firewall for remote control to function. It will add the appropriate inbound rules when Remote Control is enabled, and will remove them when Remote Control is disabled, or the StableBit DrivePool service is not running. The rules are limited to Private networks and will not allow remote control to function over networks designated as Public. When you connect your computer to a new network, Windows asks you whether it’s a private or public network.

If you have an external firewall, or a third party software firewall, then you may need to instruct them to allow TCP / UDP traffic from the DrivePool.Service.exe and DrivePool.UI.exe processes, typically on port 27525.

Security

StableBit DrivePool uses a secure .NET Remoting TCP channel to communicate all of its UI synchronization. This means that authentication is provided using the standard network authentication system in Windows (SSPI), and no user names / passwords are ever sent in the clear over the network.

All communications is also encrypted by .NET Remoting.

As I’ve mentioned earlier, in order to connect to a remote computer, the account that is connecting must be a user on the remote computer, and that user must be part of the built-in Administrators group.

StableBit DrivePool 1.3 BETA

That wraps up the topic of remote control, I think the system turned out to be fast, simple and secure. It’s also very easy to use.

Before I wrap up, let me just mention that the StableBit DrivePool 1.3 BETA for the Windows Home Server 2011 is going final soon. There will (most likely) be just one more public BETA released in a few days, then a final release one or two weeks after that.

This was probably the longest BETA that we’ve had for a single release, but I think it’s finally ready to be called a release final.

Until next time.

StableBit DrivePool 2.0 BETA

Posted in StableBit on February 1st, 2013 by alex – 15 Comments

StableBit DrivePool 2.0 BETA is out now and it supports 10 Windows Operating System variants.

Check out the download page for a full list and to grab a copy: http://stablebit.com/DrivePool/Download

What is it?

Since some of you who may be reading this may be new to our product, let me quickly summarize what it does.

StableBit DrivePool creates one virtual drive from one or more hard drives on your computer. If you ever run out of disk space on the pool drive letter then simply add another disk to the pool, and in seconds your virtual drive will expand in size.

You can even add disks to the pool with existing data, and that data will not be affected in any way. The drive will simply contribute the free space on it to the pool.

StableBit DrivePool stores all your pooled data in standard NTFS files and operates using a real kernel file system developed specifically for DrivePool for the utmost in performance.

What’s New?

Aside from the new Operating System support, StableBit DrivePool 2.0 is a complete rewrite of the management UI and the background Service. The new management UI in DrivePool 2.0 is designed to be simple to use and to stay out of the way.

Task Based System

In DrivePool 2.0 you can queue up multiple management tasks with a single click. For example, if you want to remove 2 disks from a pool and add a third, then just click Remove, Remove, Add, on the appropriate disks and DrivePool will queue up the requests and perform them one at a time.

The system is so simple to use, that it literally takes one click to create your first pool and to add your first disk to it.

Email Notifications

StableBit DrivePool 2.0 features email notifications for missing disks. It also features StableBit Scanner integration for the emails that it sends out.

If you have the StableBit Scanner installed then in your email you will get the Model and Serial Numbers of the missing disk, in addition to the Case and Bay location (if you’ve set those up in the Scanner).

Automatic Updates

StableBit DrivePool 2.0 features automatic updates, so installing new versions is quick and easy. Since DrivePool uses a kernel file system driver, some updates will require a reboot, but the installer is intelligent about that and will only request a reboot if necessary.

Additional Operating System Support

This is obviously the biggest new addition. StableBit DrivePool 2.0 supports almost every Windows version starting  with Windows Vista, both 32 and 64 bits.

Every variant of Windows offered its own challenges, and DrivePool had to be tweaked to work correctly with all of them.

dpcmd

StableBit DrivePool 2.0 comes with a command line utility to manage the pool. Just open up the command prompt and type dpcmd for usage.

Right now it’s able to query and set the duplication level on folders, and we can add more functionality in the future as needed.

What’s Missing?

Under the hood, StableBit DrivePool 2.0 supports everything that 1.3 does and more. But some things are not exposed in the UI yet. For the first BETA we don’t have a neat way to set duplication levels on individual folders, but that’s coming. For now you can use dpcmd to do that.

The balancing plug-ins from DrivePool 1.X will not install on 2.0 because of the different installer technology. Those will be updated as the BETA moves along.

The current StableBit DrivePool 2.0 BETA is only available in English, but we’ll localize it to all the languages supported by 1.X before the final is released.

Backwards Compatibility

StableBit DrivePool 2.0 was designed to be backwards compatible with 1.X. You can simply plug in your 1.X pooled drives into a machine running 2.X and all your pools will be mounted with zero configuration. Your per-folder duplication settings will also be maintained.

The format of the pool is exactly the same as 1.3, so the same backwards compatibility rules apply.

Upgrading from 1.X to 2.0

While StableBit DrivePool 2.0 was not meant to be a replacement for the 1.X series, some people may want to upgrade to the latest version. DrivePool 2.0 will work on the Windows Home Server 2011 Operating System, but there will be no Dashboard integration. We are working on a remote management solution for a future release.

For some more notes on upgrading see: http://stablebit.com/Support/DrivePool/Manual/Installation

Thank You

Thank you everyone for your continued support of StableBit DrivePool and StableBit Scanner in the past few years, and I hope that you enjoy the latest release.

Now we’ll go and wrap up all the loose ends with the 1.3 BETA.