As of version 2.0.0.1873 BETA, StableBit Scanner 2.0 adds file recovery.

More Information about the StableBit Scanner is available here: http://stablebit.com/Scanner

This means that after one or more bad sectors are found on a hard drive, you will have the option of running a file scan, and once that completes, you’ll be able to select one or more damaged files for recovery.

Scanner - File Scan

Users of Scanner 1.0 may remember this feature. In Scanner 2.0 it has a new UI and some improvements on the back-end.

Let’s take a look at exactly how this works in Scanner 2.0.

A Damaged Hard Drive

The StableBit Scanner will periodically check the surface of all your hard drives for damaged sectors.

This is what it looks like when the Scanner detects some damage on a hard drive:

Scanner - Surface Damage

(note that you can simulate this by using the Simulate tab in Scanner Settings.)

As you can see from the screenshot, there are 972 unreadable sectors on this disk. Note that, these sectors have actually failed a read, which is unlike a S.M.A.R.T. warning which is an indication by the hard drive of unverified damage.

Notice that there are only a few red blocks in that sector map. This is because the Scanner represents multiple sectors that are damaged in the same block, with just one red block. This makes it easier to read the map.

We can see the actual damage represented by a red block by hovering our mouse over it:

Scanner - Surface Damage (Tooltip)

This particular block represents 118 KB of damage. The Scanner uses the convention of 1,024 Bytes = 1 KB everywhere it displays sizes.

We can also switch the sector map to use smaller blocks to see more finer detail:

Scanner - Surface Damage (Tiny Blocks)

Scanning for File Damage

In order to open the File Scan interface, you can click the link at the bottom of the main window or right click on the damaged disk and select File Scan…

Scanner - File Scanning

How it Works

Let me explain a bit about what the file scan actually does and why. Unlike conventional surface scanners that are able to scan the disk surface for damage and attempt recovery of bad sectors, the StableBit Scanner takes a smarter approach. After scanning the entire disk surface and mapping all the bad blocks, the Scanner answers the question, was any data lost and which data?

We can answer this question because the Scanner knows how to read NTFS, directly. Unlike others, it doesn’t use any Windows APIs for this. This is because we want absolute full control over parsing any damaged on-disk data structures. The Scanner knows which sectors are damaged, and it understands how to read the undamaged portions in order to determine what exactly is damaged, in terms of on-disk data.

So you can think of the file scan being much more than just a scan looking for damaged files. Actually, it examines every single data structure on its way to scanning for damaged files. Starting from the master boot record or the GPT, going down to the partition boot sector then the master file table, each directory structure and index and finally each file and its data. The Scanner will even examine on-disk data structures that are not directly related to file data, such as file metadata records, backup MBR / GPT, the MFT mirror and will report on any damage found there as well. It understands how to access the various backup data structures in order to continue its scan, even if the primary structure is damaged. It is a comprehensive NTFS scan, looking for any damaged data structures.

The File Scan can actually report on 17 different types of on-disk data damage. 16 of them are extremely uncommon, and primary file data damage is the most common. This essentially means that a bad sector has fallen in the middle of your data. The other 16 indicate on disk data structures that are not directly related to your data.

The reason why this is the most common type of damage is because primary file data represents most of the data on the disk, by far. After all, it basically represents all of your data that is stored on the disk. And so the chance of a bad sector falling inside primary file data is much greater compared to it falling inside some other data structure.

The StableBit Scanner provides for automatic file recovery of damaged NTFS files.

It does not support automatic recovery from the other damage types because those indicate intricate damage to the on-disk formatting and will require an expert human to go through the damage and reconstruct the data that was there. Luckily that type of damage is very uncommon, especially if you catch the failing disk as soon as it starts to have trouble, before the damage becomes very extensive, which is the whole point of running the StableBit Scanner.

List of Damage

The time it takes to conduct the file scan depends on how many files are on the disk and how many damaged sectors there are. The more damage, the harder the Scanner has to work to scan your disk. The Scanner does not try to re-scan known damaged sectors during a file scan, but will detect any additional bad sectors that may have developed since the surface scan completed.

The time to complete a file scan can range from a few seconds to a few hours.

After the file scan completes, you will see a list of damaged on-disk data structures (if there are any):

Scanner - File Scan

You can expand one or more on-disk data structures, to get more information about the damage. Each item in the list will typically be a file, but it could be some other damaged on-disk data. Expanding it will provide you with more details on what it is and how much of it is damaged.

Scanner - Damage Details

Here you will notice a few things represented.

On the top you will see a Disk Map. This represents a map of the entire disk surface and the green blocks represent the portions of the disk that the current data structure is occupying. The red blocks will represent the damaged portions.

Hovering your mouse over the red blocks will reveal more detail:

Scanner - Damage Details (Tooltip)

Right below that you will notice the Damaged Entity Map. This is simply a representation of the entity and where the damage is.

You may wonder why we need both maps. But keep in mind that files may are not be stored contiguously on the disk. In other words, they can be chopped up into pieces and scattered all over the disk. The beginning of a file can be at the end of the disk and the end of the file can be at the beginning.

The Damaged Entity Map represents a view of where the damage is in the file. This can be extremely useful in understanding where the damage is, as you see the file in the Operating System.

For example, if you see a bad block at the end of this map, then you will not be able to read the end of that file in Windows, and the other way around.

Also, it’s interesting to note that NTFS typically reads file data in cluster sized blocks (4,096 byte blocks typically). This means that even if a single sectors (512 bytes) is damaged, a 4,096 byte block will be unreadable by Windows.

Since the Scanner doesn’t use NTFS and implements its own version of the file system, this limitation is not an issue. To the Scanner, if one sector is damages, then only 512 bytes are unreadable. These little details are very important when recovering files and the Scanner tries to be aware of all these things in order to do the best job that it can.

Attempting File Recovery

In order to recover the files you will need an external hard drive or a thumb drive, or some other destination to place the recovered files. It is not recommended to place the recovered files on the damaged drive, for obvious reasons.

At this point you can select one or more files for recovery and click Attempt file recovery…

Scanner - Select for Recovery

Now it’s time to select the destination for your recovered files and some recovery options:

Scanner - File Recovery

If you’d like, you can tell the Scanner to not attempt bad sector recovery at all. In this mode, the Scanner will write out all the known good sectors to the destination media and will skip all the bad sectors. This is useful if you have a lot of damaged data and would like to quickly get as much information out of the drive as possible. Bad sector recovery can take some time and selecting the option to not recover bad sectors will speed up the recovery process.

Once you click Start Recovery, the StableBit Scanner will go to work attempting to recover your files:

Scanner - Recovering

The Scanner first reads the known good sectors and then attempts bad sector recovery, for each file.

Each bad sector recovery attempt is driven by various head motion profiles. Some of the profiles are driven by a random number generator, which ensures a unique recovery attempt each try. The Scanner tries to read a damaged sector by altering the velocity and direction of the drive head on each attempt, in hopes of getting the drive to read the bad sector one last time.

On each attempt, for each bad sector, the Scanner tries 11 different recovery algorithms on that sector. This is what is considered to be one recovery attempt for a sector.

File Recovery Results

After the Scanner finishes its file recovery it will display the results like this:

Scanner - File Scan Recovered

Note that because the damage was simulated for this blog post, all sectors were instantly recovered. It is possible for the Scanner to recover files partially or none at all. If the Scanner did not succeed at recovering one or more of your files you can re-attempt file recovery, but increase the bad sector retry count to something greater.

We can select each recovered file to see how much data and which sectors were recovered:

Scanner - Recovered Details

You can hover your mouse over any recovered block to get more information on that block:

Scanner - Recovered Details Tooltip

At this point, your recovered files have been written to the recovery destination that you selected earlier.

Note that the original file on the damaged disk is still damaged, and the Scanner will continue to report disk surface damage and file damage for that disk. The Scanner does not write to the damaged disk during file recovery.

You should remove and replace the damaged disk at once. Continuing to use a hard disk after it has damaged one or more of your files is a really bad idea.

Conclusion

This concludes this comprehensive review of the file recovery system in Scanner 2.0. I haven’t written one of these long and detailed articles in a while now, I hope that some of you enjoyed reading it.

Next we will look at improving S.M.A.R.T. in Scanner 2.0.

If you build and sell computers and would like to distribute them with one of our products, we can now make that happen.

Just open up a contact request @ http://stablebit.com/contact for pricing details and to set up an OEM account.

Prices are very competitive.

StableBit DrivePool went out of BETA and into a production final at the end of last month (March 2012), and I’d like to take some time to talk about how it all began, how we got here, and perhaps a bit about the future.

For more information about DrivePool and to purchase a license visit:
http://stablebit.com/drivepool

I’d like to make this post a bit more personal and I’d like to tell you about the development effort from my perspective.

It’s the end of the world as we know it

In 2011, some time around January, Microsoft announced that they are dropping the drive extender feature from Windows Home Server 2011. Well… what can you say about something like that. It was a bombshell. Drive Extender was the heart and soul of the original Windows Home Server operating system, and taking it out is basically killing the OS, in my opinion. Their reason for doing so, as I understood it, was because it never worked correctly in the first place. Which of course made no sense at all because it has been working fine for so many people over the years.

Needless to say, there was a big outcry from the community and I recall that people started brainstorming about alternatives at the time.

Covecube Inc. was already invested into the Windows Home Server ecosystem with a 6 month development effort of Scanner 1.0, which was planned for a Windows Home Server 2011 port.

An interesting dilemma came to light. There were really 2 options for the company, move on to something else, or fix the problem with Windows Home Server 2011. The fix to the problem would obviously involve developing a brand new pooling system, that would work seamlessly with the existing WHS 2011 infrastructure. Also, it would have to be functional rather quickly, because everyone was expecting to see a pooling solution in WHS 2011, that was about to ship.

Given those requirements, the solution had to be simple and powerful. The real question was, can such a pooling architecture be created?

DrivePool emerges

On Feb. 11 2011 I announced DrivePool, first on the We Got Served forums and later in other places.

Here’s the original announcement:
http://forum.wegotserved.com/index.php/topic/17628-introducing-stablebit-drivepool-for-the-windows-home-server-2011/

As you can see, at this point, the screenshots show a working version of DrivePool.

I started Covecube Inc. back in Feb. 2010 with the simple idea of making good software that fulfills 3 needs. It has to have a deep technological footing, a functional and simple interface, and a great looking UI. The http://covecube.com web site explains this best.

I’ve tried to put as much of those things into DrivePool as possible.

The Architecture

The untold story behind those first screenshots was that I didn’t want to post anything until I was sure that what I posted was based on a solid architecture. You see, my feeling is that bugs can be fixed with time, but a bad architecture can’t. So to me it was extremely important to get the underlying foundation of a solid pooling solution in place, and I spent many sleepless nights before ever posting that, to make sure that what I was posting was a working prototype.

The architecture that was part of that first build which came to be known as DrivePool BETA M1 is still largely what is in use today, 6000 and some odd builds later. Although the file system was swapped from BETA M3 to BETA M4, as you will soon see.

Yes, DrivePool did start at build 1!

DrivePool BETA M1

DrivePool BETA M1 (what was just known as the DrivePool BETA at the time) was released a few weeks after the initial announcement. It was actually delayed by some time because of some very intricate issues that had to do with directory listings. Those were fixed, and luckily no one knew about them, until today that is.

DrivePool BETA M1 depended on Dokan, an open source project that was designed to make file system development easy. There are a number of architectures like this, some commercial, that let anyone make their own file system easily. Dokan was LGPL licensed, so there was no conflict shipping it as part of a commercial product. At one point I even set up a fork of Dokan for DrivePool so that I could check in fixes publicly.

Using a “file system for dummies” approach is actually a really good way to prototype a new system. It’s quick and easy to make changes. The disadvantages are mainly performance and compatibility. I’ve always known that we would have to develop our own file system eventually and move off Dokan before DrivePool goes final. But it made perfect sense to ship the early BETAs with it in order to work out the details of the architecture instead of spending time rewriting the same kernel driver 3 times.

DrivePool BETA M2

DrivePool BETA M1 was a great success. It worked surprisingly well, considering that it was just conceived a few months ago. Continuously patching bugs in Dokan also turned out to work well too. Many issues present in the first BETAs started to disappear, and it really started to look like Dokan could actually hold its own in a final release.

DrivePool BETA M2 was made public on May 18 2011. It’s main goal was to complete the pooling infrastructure. Where BETA M1 has only some basic pooling functionality, BETA M2 added a lot of disk management features regarding foreign disks and missing disks. It also introduced the extremely extensive logging infrastructure built into the Dokan kernel driver and the DrivePool service.

The new logging capabilities were a massive undertaking that most people were not aware of, but it made finding and fixing I/O bugs orders of magnitude simpler than before.

At this point it started to look like BETA M3 would become the final, and while I always hesitated to give out any release dates, June / July 2011 were always a good targets to try and hit.

DrivePool BETA M3

BETA M3 was all about integrating DrivePool into the existing WHS 2011 infrastructure. This involved a lot of research / reverse engineering and trial and error. It actually turned out to be much harder than any of the previous BETAs, including developing the pooing architecture itself.

BETA M3 eventually shipped on July 26 2011 and it was pretty much a feature complete build. I have to say, considering the complexities involved with the integration, it worked very well.

DrivePool BETA M4

BETA M4 was originally meant to be a performance enhancing build. Essentially, around BETA M2, the decision was made to stick with the Dokan file system driver for the final, but to replace pieces of its code with our own file system driver. This was actually possible because of the way everything was written.

A few of the later M3 builds started down this path and this hybrid file system actually shipped publicly, but the advantages of having a real kernel-based file system driver became painfully apparent. Most of the complexities introduced in BETA M3 would just disappear. It would be far better performing, easier to maintain and far more compatible.

It was obvious that Dokan, or any of the expensive commercial user mode file system libraries available would simply not work for a rock solid DrivePool final. In Windows, file systems were written in the kernel and that was the only way to do it.

CoveFS

Dokan was abandoned, and my original plan for building a brand new pooling file system, back from BETA M1, was put into effect, and it was called CoveFS.

CoveFS was a completely different approach that Dokan, no user mode dependencies, a real file system that just works. Another thing that had to be written was a virtual disk driver, and that ended up being called the CoveFS Disk.

Once the first internal builds of BETA M4 started coming out, it became obvious that this was the right approach. It was fast and long lists of issues from BETA M3 just vanished.

Much of the BETA M3 integration code was deprecated and BETA M4 went public on: Jan. 12 2012

DrivePool 1.0 Release

I have to say, after the long BETA testing period of DrivePool 1.0, I am completely happy with the end product. It is everything that I imagined back when the project was started, just over a year ago. It has a real native file system, it’s super fast and it’s very compatible. It has a finely tuned disk balancing algorithms, with multiple innovative approaches there, file duplication that is fast and reliable.

Personally, I’ve migrated all of my personal data from the original Windows Home Server to DrivePool, and I will continue to use it until something better comes along.

Hopefully it will be DrivePool 2.0.

The Future

DrivePool’s future is simple. I am committed to fixing any remaining issues and to support the existing product. But future development will depend on whether the money invested the 1+ years of development can be recouped. It is really as simple as that, Covecube Inc. as a business will need to survive and prosper in order for new products to be developed.

Recently, a 10,000 pound Gorilla, Microsoft, walked back into the pooling arena. As some of you may know, after denouncing their earlier pooling solution as a wasted effort, they had decided that it was not a mistake after all. They are working on developing their own pooling solution.

I can say this, there is already a plan laid out for DrivePool 2.0 and it is very cool and original.

I’m not announcing any new features at this time, but hopefully there is enough interest in such software to make a continued development effort worthwhile. I would sure love to continue innovating in this area.

Recently StableBit DrivePool has found a new home at stablebit.com/DrivePool and is now available for purchase with PayPal or Google Checkout.

DrivePool is an advanced disk pooling application available for Windows Home Server 2011, Windows Small Business Server 2011 Essentials and Windows Storage Server 2008 R2 Essentials.

Visit: stablebit.com/DrivePool for more information and to download a 30 day trial.

StableBit DrivePool

DrivePool will remain in BETA until the end of March 2012. This will serve as a transition period to smooth over any remaining issues.

Thank You BETA Testers

It has been a lot of work getting DrivePool ready for a final release and I am thankful to the many of you who have helped test it.

Qualifying BETA testers have received a discount over the retail purchase price. If you’ve submitted a bug report through stablebit.com, check you email, under the email address that you used to report the bug you should find a discount link.

PayPal Support

I should also mention that you can now purchase the StableBit Scanner using PayPal. Some people have been asking about PayPal support. Well, here it is.

Bundles / Additional Copies

If you need to purchase additional copies of the StableBit Scanner or StableBit DrivePool or if you already own one and want to purchase the other, just visit stablebit.com/Buy and enter your existing retail activation ID towards the bottom of the page and you will get discounted pricing.

There is no bundle, but you can effectively get the same discounted pricing by buying one product and then purchasing the other.

DrivePool BETA M4 has been out for a while now and is nearing a final release.

You can grab the public beta right here:
http://wiki.covecube.com/StableBit_DrivePool

In this post let’s delve into how file duplication works in DrivePool M4.

Real-time file duplication

I’ve received a number of questions regarding this topic, so let me start by laying out what file duplication accomplishes.

DrivePool M4 actually can maintain duplicate copies of files in 2 different ways, real-time and by duplicating later. Let’s talk about the real-time duplication option first, because that will be what the majority of users will be using and it’s also the default mode out of the box.

DrivePool M4 - Real-time file duplication

What does it do?

File duplication protects files placed in a duplicated folder from any single drive failure at a time. That means that if one drive fails, all the files in all the duplicated folders will remain intact.

Does file duplication constantly chew up resources in the background duplicating files?

Under normal circumstances, real-time file duplication does not require any background maintenance in order to keep your files duplicated.

Files are simply updated in parallel across multiple disks, every time something writes to them or modifies them in any way.

What about changing the level of duplication of an existing folder?

You can do that from the Dashboard for any folder on the pool.

DrivePool M4 - Enable duplication

There is a background duplication process that will kick off and creates new duplicated file parts for each file in that folder or cleans up existing duplicated file parts if you choose to disable folder duplication.

This process runs completely in the background and is a one time pass. When the process completes, all the duplicates files are in the correct state for the real-time duplication engine to take over.

You might be wondering what would happen if you changed the duplication level of another folder while the background duplication process is already running. Don’t worry, it will handle that properly and duplicate or de-duplicate all your files as needed.

How do background duplication changes affect disk performance?

All background duplication changes are processed using a Windows feature called Background I/O. This feature was introduced with Windows Vista and allows applications to perform intensive disk tasks, such as copying large files around, without impacting the overall disk performance. Windows accomplishes this by prioritizing the actual I/O operations in the kernel, so this is pretty effective. But of course in the real world you may notice a slight performance hit when background duplication is running.

After you change the duplication state of a folder, the time to complete a duplication pass depends on how busy your server is. This means that it might take a while for the background duplication pass to complete if your server is busy doing something else.

Future versions will optimize this even more.

Will background duplication lock files as it makes duplicate copies of them?

No it won’t.

This is actually pretty cool and is available as of build 5805. DrivePool’s background duplication engine does not lock any files while it’s working on them. What it does is, monitor the file for any other applications that might want to alter it and if it detects such an occurrence it will cleanly back out of whatever it was doing to that file and yield to the other application.

The other application doesn’t see anything out of the ordinary at all when it tries to, for example, write to a file that is currently being duplicated.

DrivePool accomplishes this using standard NTFS APIs.

Duplication consistency

Now that we’ve covered the basics, let’s delve a bit deeper into the real-time duplication system and talk about how DrivePool ensures that duplication consistency is maintained at all times.

Duplication consistency means a number of things. For files (or alternate streams), it means that there should be the correct number of file parts for each file in any given directory. For example, in a duplicated directory there should be exactly 2 file parts for each file in that directory, no more and no less. In a non-duplicated directory there should be exactly one file part for each file.

A file part is simply a file residing on one of the pooled disks that the file on the pool represents.

For directories DrivePool allows more than the required number of directory parts. For example, for a duplicated directory, there could be 4 directory parts representing it and that would still be consistent. However, one directory part representing a duplicated directory is not consistent, there must be at least 2.

There is no upper limit on how many directory parts can exists. This is because DrivePool can clone any directory at any time, depending on its internal file placement strategy. As an optimization, DrivePool does not clone the entire directory structure across all disks but opts to manage and clone directories in real-time as needed. But there can never be less directory parts than required.

Ensuring consistency during real-time usage

Every time you open any file, directory or alternate stream, CoveFS checks to make sure that the correct number of file parts were found for it, given the above rules. This is done in real-time and doesn’t cost anything extra, in terms of performance, because we need to find all the file parts anyway.

If an incorrect number of file parts were found, CoveFS records the file parts that were found, and how many were expected to be found. It sends this information up to the DrivePool service, which logs the issue in its human readable log and marks the pool as “needing duplication”.

At the next available opportunity, DrivePool will start a scan of the entire pool in order to make sure that each file is consistent and will correct the problem automatically, if possible. This is called the background duplication pass, and this is what causes the “Duplicating…” message to appear.

Background duplication

DrivePool M4 - Background duplication

Background duplication is a process that scans all the files on the pool for duplication problems. It can be triggered by changing the duplication level on an existing folder, reconnecting a missing disk to the pool, adding a disk from a different pool, and a number of other administrative tasks.

During normal shared folder usage, with real-time duplication enabled, background duplication does not need to run in order to maintain file duplication.

Not interfering with other I/O

One of the goals of background duplication is not to interfere with other disk I/O, pooled I/O or otherwise. DrivePool accomplishes this in 2 ways.

First, whenever DrivePool works on a file, for example, to restore a missing duplicated file part, it will never lock that file. This allows any other application to open the same file that DrivePool is working on and start modifying it. DrivePool qukcikly and cleanly aborts what it was doing to that particular file and moves on to the next one.

Second, the entire background duplication pass is conducted on a thread that is scheduled to use Background I/O, a Windows feature that schedules all I/O requests to yield to any other normal priority I/O request. This is far more effective than just lowering the CPU priority of a thread. The down side of doing this is that the duplication pass can take longer than it would otherwise, but with the first feature mentioned above, this is not much of a problem. Background I/O priority can be turned off using DrivePool’s advanced settings.

Checking directories and files

The background duplication check is more through than the real-time check. It goes through each file and directory and makes sure that they have the correct number of file parts and that each file part is the same.

We’ve already talked about the criteria that the real-time duplication check uses to determine how many file parts should exist for any given entity on the pool. The background duplication check uses the same rules.

After it verifies that it has the correct number of file parts, background duplication will check each file part to ensure that it’s the same. It does this by first looking at the size of the file, if the size differs then we have a duplication conflict, and we don’t go any further.

We then check the modification time on each file part, if the modification times are different then DrivePool starts to check the contents of all the file parts to make sure that they all contain the same data. It does this by performing a hash of each file part and comparing the result. If all the file parts contain the same exact data, then DrivePool synchronizes the modification times across all the file parts for the given file. If a mismatch is found in the file contents, then that triggers a duplication conflict.

Note that, at this point we’ve determined whether a file has the correct number of file parts or not, but we haven’t reacted to that information yet. We’ve simply established that all the file parts are the same.

Once we are sure that every file part is the same, we now do one of two things. We either clone existing file parts into more file parts, as needed, or we cleanup existing file parts if there are too many of them. Of course we never remove the last file part.

Duplication conflicts

When a duplication conflict is detected by the background duplication pass, the file name is recorded and the file is skipped. We don’t perform any automatic maintenance on files which have file parts that don’t match in terms of their data contents.

Just to clarify, this means that we never delete or duplicate file parts that have different data contents.

A warning is issued to the user in this case at the end of the background duplication run.

DrivePool M4 - Duplication conflict

This condition is rare, because it should only be caused by some application or user directly accessing and modifying the pooled files on the individual pooled disks, as opposed to going through the pool.

Resolving duplication conflicts

To resolve such a conflict, you can either tell DrivePool to delete the older file parts automatically, or you can find the file parts yourself and delete the ones that are not correct.

File modification times

If you’re going to modify file parts directly, or will be running applications that modify file times, keep in mind that DrivePool uses the file modification times of the file parts contained in the pool part folders to ensure duplication consistency. If you modify these times directly (not through the pool drive), it won’t break duplication, but the next time that background duplication needs to run, it will need to re-hash every duplicated file in order to make sure that the contents haven’t changed. This can take a while.

Duplicating files later

DrivePool M4 - Duplicate later

Now that we’ve talked about real-time file duplication and how it works together with background duplication, let’s talk about what it means to duplicate files later.

Duplicating files later does 3 things.

First, new files are never written to 2 disks at the same time. All new files are always written to a single disk.

Second, duplicating files later turns off the real-time duplication check. In other words, if a file is found to have an incorrect number of file parts, nothing out of the ordinary happens.

Third, a background duplication pass is scheduled to run every night in order to duplicate files that need duplicating. This background duplication pass is identical to the one that runs in the real-time duplication scenario.

Why duplicate file later?

When duplicating files in real-time, every write is issued to two file parts, in parallel. If you want to avoid the second write for performance reasons, this is another option available to you.

This might be a good option if you tend to write a lot of temporary files to duplicated folders that then get quickly deleted and so there is no real need to duplicate them.

Overall, most people should stick to the real-time duplication option as it offers instant file protection as soon as any file is copied into a duplicated folder.

Real world usage

Overall, DrivePool M4′s duplication system is designed to be as maintenance free as possible. Many tasks are automated, including recovery of missing duplicated file parts.

Anything that requires your attention will raise an alert in the Dashboard letting you know of the problem, and offer some advice on how to go about resolving it.

You’re not really required to understand its inner workings in order to make use of folder duplication, but I thought I’d share with you what is going on behind the scenes so that you can have some idea of how the system works.

It has been a while since I posed anything, but all this time there has been a lot of work going into DrivePool, and BETA M4 is finally nearing completion. DrivePool BETA M4 should be released to the public at the end of this month or in January. BETA M4 will be the final BETA and will turn into the 1.0 final once it’s tested.

There’s a lot to get excited about in M4 so let’s take a look at what’s new and what’s changed.

Keep in mind that the screen shots are based on a development version and might change before release.

A new look

Let’s take a look at the UI. The UI is more than a new look actually. DrivePool M4 is deeply integrated into the Dashboard.

DrivePool M4 - Overview

The UI of M4 has been extended to be more customizable. There are 2 distinct modes that you can run DrivePool in, integrated and standalone. In addition, there is a new overview tab that is pictured above.

The standalone mode is mostly like DrivePool M3, with the addition of the overview tab and the UI refreshed.

DrivePool M4 - Standalone UI tabs

The integrated mode is brand new and will be the default mode that DrivePool will run in. In this mode, DrivePool will extend the standard UI available in the Server Folders and Hard Drives tab.

Let’s take a look at what the standard Server Folders tab looks like with DrivePool installed.

DrivePool - M4 Integrated UI - Folders

If you look at the screenshot above you’ll notice a number of things that have been added to the original Dashboard UI.

• The list now has 2 new columns indicating pooling status and duplication for every folder.
• The details pane has been completely revamped to host the DrivePool folder details UI.
• There are 2 new tasks available on the right that let you manage the pool.

Foder Details

DrivePool M4 features full integration with Microsoft’s Search API. For one thing, this means that all your pooled folders will now be indexed, you will be able to search through them instantly using Windows Search, and you will be able to add them to libraries on client computers.

Because WHS automatically indexes all shared folders it made sense to integrate with the index in order to show an overview of the types of files in every shared folder. Because all your folders are already indexed, this new functionality comes at no additional cost in terms of performance.

In addition, DrivePool queries the Windows indexing service and shows you when your folders are being indexed with an animated tag right in the folder details pane.

One other neat thing is that now folder details are available for all folders, not just pooled folders. Here’s what it looks like when you select a non-pooled folder.

DrivePool M4 - Non-pooled folder

Because this utilizes the built-in file types in Windows Search, the list of types is the same as those that you would see in Windows Explorer.

Disk Usage

On the right of folder details you will see the disk usage. To the user this appears to work the same way as it did in M3. As you place files in a folder or delete files from a folder you will see these bars show you how your files are being distributed among all the pooled disks.

DrivePool M4 - Disk usage

This works in real-time with very little overhead. This is because the new file system in DrivePool M4 has a specific interface for monitoring file size changes. So when a file size changes on the pool, a message is queued up for the DrivePool service letting it know of the change. M3 has a similar system, but it has been extended and made even more efficient in M4.

You’ll also notice that disk usage is shown for non-pooled folders as well. How is this possible you may ask, if those files are stored on NTFS which doesn’t have the extended file size change notifications? Well, we utilize Windows Search to quickly query the size statistics for those folders.

In the end, you get full folder usage statistics for all folders, pooled and non-pooled.

Pool Condition

DrivePool M3 had a neat feature where, using some fancy math, it was able to calculate the pool balance ratio in real-time. The pool balance ratio was used to determine if balancing the pool was necessary. Generally, a single balance pass was all that was needed to balance the pool.

In M4, in addition to having a pool balance ratio we now have duplication consistency status and we now also need to display some additional status about tasks running in the background. Rest assured that the background tasks are designed to utilize minimal resources and run only when necessary.

Instead of creating a more complicated UI to show all these things, M4 tries to keep it simple and focused. The balance ratio from M3 is now known as the Pool condition percentage.

DrivePool M4 - Pool condition

For example, one of the tasks that can be running in the background is file duplication (if you’ve changed the duplication setting for a folder).

DrivePool M4 - Pool duplicating

Managing Folders on the Pool

When you wanted to add a folder to the pool or move folders to / from the pool, in DrivePool M3 you had to use custom built wizards from the DrivePool UI that accomplished the tasks for you. M4 tries very hard to re-use as much of the existing WHS infrastructure as possible.

Moving Folders

In M4, moving a folder to the pool is accomplished by using the standard move folder wizard. You just select the pool, which in this case happens to be I:\.

DrivePool M4 - Move a folder to the pool

Default folders, including the client computer backups folder, can be moved the same way without a problem.

Adding a Folder

To add a folder to the pool, you can use the standard add folder wizard and just put the folder on the pool drive. WHS will select a default disk for a new folder that’s on a non-removable disk and has the most free space.

DrivePool adds a new task to simplify the add folder process by modifying the default wizard to always select the pooled drive. This is what it looks like when you click “Add a folder to the pool”.

DrivePool M4 - Add a folder to the pool

Visually, it’s the exact same wizard but without the browse button.

Removing or un-sharing a folder works exactly the same way as it does for a non-pooled folder.

Folder Properties

In DrivePool M3 folder properties were re-done with a custom dialog that included duplication settings among other things. This is gone in M4. M4 uses the standard WHS folder properties dialog for pooled and non-pooled folders.

DrivePool M4 - Standard folder properties

Folder Duplication

One of the complaints from users of DrivePool M3 was that changing duplication took a while and you couldn’t exit the dialog until duplication was complete. In DrivePool M4, file duplication now happens in the background. I should be clear on this, DrivePool still maintains file duplication in real-time across 2 disks as you read  / write files, but any changes to whether a folder is duplicated or not, now happen in the background.

On a side note, DrivePool M4 also gets rid of numerical suffix that was used in M3 to indicate duplication level for folders (such as Pictures.2) and implements a more flexible approach that lets the service toggle duplication on any folder on the pool. It even supports duplication inheritance, but this feature is looking towards the future as it’s not exposed in the UI.

To change duplication on a folder you use the new “Change duplication” folder task.

DrivePool M4 - Change folder duplication

The change itself happens almost instantly and the files begin duplicating (or start getting cleaned up) in the background.

HomeGroup Support / Media Streaming

DrivePool M3 had a plethora of functionality to make HomeGroup support and media streaming work. Since in M4, pooled folders are just like non-pooled folders, nothing special is necessary. It just works.

Managing Hard Drives

In integrated mode, just like DrivePool extended the default folder details pane, it also extends the default hard drive details pane.

DrivePool M4 - Hard drive details

Details are shown for non-pooled drives, pooled drives and the pool itself.

In DrivePool M4, the pool is treated like any other drive, so this is what we see when we select the pool, which happens to be the I:\ drive on this server.

DrivePool M4 - Pool details

You’ll notice a cleaner UI than M3 and 4 slices in the chart. One of the confusing things about the pool details in M3 was that it wasn’t clear what un-duplicated free space was. In M4, this is now cleaned up. As you can see from the image above, some free space is listed under “Unusable for duplication”. I think this makes more sense.

How can you have space that’s not usable for duplication you may ask?

Well, if you have 2 hard drives in the pool, one 100 GB and another 200 GB, then you can at most put 100 GB of data in a duplicated folder. Remember, a duplicated file must exist on at least 2 drives, and in this example the second drive has 100 GB more than the first and so there is a duplication imbalance.

In any case, this is less of a problem as you add more than 2 drives, and DrivePool does the math for you and displays this in the chart pictured above and on the overview page. This was already available in M3, it’s just a little neater in M4.

Pool Management

Pool management is where you would add or remove drives from the pool. This is very similar to what M3 had. There is now a refreshed UI in the details pane and details are now shown for all drives, not just ones in the pool.

DrivePool M4 - Pool management

You will notice that the pool is listed as a drive in its own category.

Adding a Drive to the Pool / Removing a Drive

This looks nearly identical to what it was in M3. The underlying code is different and utilized the new CoveFs file system.

DrivePool Settings

Settings for DrivePool are now integrated into the standard settings dialog. It’s still very minimal in this build, but this is what it looks like.

DrivePool M4 - Settings

One thing you can do in this build is hide the default DrivePool tab for a completely native look and feel.

Conclusion

In DrivePool M4 the UI has been cleaned up and streamlined. Where possible, DrivePool now uses the default built-in wizards.

But this is not the whole story because DrivePool M4 features a brand new back-end, including a new file system written specifically for pooling files and maintaining maximum compatibility with existing applications that expect NTFS. But that’s a whole different story.

Once again, DrivePool BETA M4 is scheduled to be released publicly at the end of December 2011 or January 2012 and will be available for the Windows Server Solutions family of operating systems.

StableBit DrivePool BETA M3 is out and available for beta testing. It offers vital drive pooling features for the Windows Home Server 2011 and Windows Small Business Server 2011 Essentials class of operating systems.

It provides tight integration with the Windows Home Server 2011 infrastructure and supports features such as media streaming and remote web access for pooled folders.

For more information and to download a copy see: http://wiki.covecube.com/StableBit_DrivePool

It’s a BETA and we’re still ironing out any remaining issues.

What’s new in M3?

DrivePool BETA M3 is all about integrating tightly with the Windows Home Server 2011, but there are some other cool features that are unrelated, like efficient file balancing, which are part of it as well.

Let’s dive in and take a look.

Pooled Folders

In DrivePool BETA M2 and prior folders created on the pool were not visible in the standard Windows Home Server Server Folders and Hard Drives tab. Instead, they were only visible in the DrivePool tab. This made the folders special in that they didn’t work with any of the existing Windows Home Server features such as media streaming and remote web access.

Starting with BETA M3, DrivePool folders are now visible universally in the Dashboard. You can even use the standard Windows Home Server interface to manage permissions on the pooled folders.

In order to make this possible, the way DrivePool delegates permissions has changed in M3. DrivePool now supports full NTFS-like security on the pool, which means that the old share level permission scheme is gone. All folder permissions are now stored on the pool itself, as part of the folder, and not on the hosting computer where the share is defined.

When upgrading from M2 to M3 your folder permissions will be converted to the new scheme and it should all work seamlessly.

Data Visualization

In DrivePool BETA M3, if you want to see what types of files are stored in a folder, or how much of your pool is currently in use, just take a glance at the built-in charts.

One neat thing about data visualization is how it’s collected. DrivePool scans all your folders for their contents once and keeps track of any changes to them, in real-time. Don’t worry the real-time tracking won’t slow down your access to the pool, it’s all done in memory. We only process the changes once every 30 seconds on a background thread totally separate from the I/O that you do on the pool, and even then, the processing is minimal. In addition, the initial full scan is carried out using background I/O, which is a new feature in Windows that throttles the scan in order to not interfere with other applications accessing the hard drives.

It’s an efficient way of gathering statistics which helps us in avoiding unnecessary pool balancing, more on this later.

Remote Web Access

DrivePool M2 folders did not show up in remote web access, this has changed. All folders on the pool will now properly show up on the remote access web page, respecting the  permissions that you set up of course.

In the screenshot above, Media is a folder on the pool. You can upload and download files to Media, just as you would with a non-pooled folder.

Media Streaming Support

DrivePool BETA M3 seamlessly integrates with the built in media streaming capabilities that come with Windows Home Server 2011.

As you can see, out pooled Media folder is set for stream to any supported devices. Windows Home Server 2011 uses DLNA / UPnP for its media streaming protocol and should be compatible with devices of that class. Consult Microsoft or the built-in help system for a full list of supported devices.

HomeGroup Support

You can now assign HomeGroup permissions to a folder on the pool and that folder will show up in you HomeGroup. It makes configuring sharing permissions among a group of computers in your home a breeze.

Folder Management

DrivePool BETA M3 comes with better folder management capabilities. You can move folders around to and from the pool using a new wizard. Even the built-in default folders such as Videos, Music, Documents can be moved to the pool.

You can also turn duplication on or off from the folder properties screen of any folder on the pool.

File Balancing

Unlike some other balancing algorithms, DrivePool M3 is not concerned about how even your files are spread out on the disk. Instead DrivePool implements efficient file balancing that uses some fancy math and real-time folder size tracking in order to determine how necessary it is to re-balance your pool. In other words, DrivePool only advises you to re-balance when you would actually benefit from the re-shuffling of files across your pooled disks.

It expresses this notion in one simple balance ratio. When the ratio is at 100% then re-balancing your pool will not benefit you and so DrivePool doesn’t waste your time and resources to do it. If the ratio falls below 75% then a single balancing run is scheduled overnight to restore your pool to an optimal state. You can start a manual balancing run any time the ratio is below 100%.

I’ll post more on exactly how DrivePool decides when it’s necessary to re-balance, and on future balancing alternatives in a later post.

What’s Next?

DrivePool BETA M3 is full of integration features that seamlessly meld the product into the Windows Home Server 2011 ecosystem.

BETA M4 will be all about performance optimizations, more bug fixes and UX improvements. So look out for that next. BETA M4 will be the last Milestone build in our parade towards 1.0 final.

For those of you in the United States, it’s Independence Day.

Here at Covecube we’ve made something special for the Windows Home Server 2011 on this day.

Here’s a Fourth of July Fireworks add-in!

Download: Covecube.Fireworks_1.0.0.1.wssx (121 KB)

SHA1: c4645bb4775bd5fae994444454f58db66af6b79f

Features:

• Displays a spacial message every year on the 4th of July.
• User can enable an option to vary the Fireworks intensity based on disk activity and CPU usage.

Requirements:

• Windows Home Server 2011 or Windows Small Business Server 2011 – Essentials

Happy Fourth of July!

More info / download: See the wiki

Continuing on from my last post about StableBit DrivePool BETA M2, a disk pooling application for the Windows Home Server 2011, let’s talk about data consistency.

We all know that hard disks and SSDs go bad, it’s just a matter of time. DrivePool BETA M2 has built in mechanisms to detect read errors / write errors and data duplication errors and offer a fix in the form of a wizard for each condition.

In addition to fault detection and resolution, DrivePool’s folder duplication engine is designed to keep files in duplicated folders safe from a single hard drive failure, in real-time. In many cases you can continue reading or writing to a file even in the face of drive failure.

Read Errors

(pictured above)

Whenever a read error occurs on one of the drives in the pool, DrivePool remembers which file was being read and immediately marks that drive as Unhealthy. If the file that had the read error is a duplicated file, then DrivePool will attempt to read from the duplicated part, in order to satisfy the original read request.

If the file is not duplicated, or if both parts of a duplicated file fail the read, then and only then is the error returned to the caller.

This means that files in duplicated shares are resilient to read errors.

Wizard

To get rid of the Unhealthy status you will need to complete the read error wizard pictured above. It will try to read the damaged file in its entirety. If it can’t, then it won’t remove the Unhealthy status of the drive. In that case, you can try to recover the file (perhaps using the StableBit Scanner for Windows Home Server 2011 – in development) or delete the file.

Write Errors

When DrivePool detects a write error on one of the drives in the pool it will remember the file and mark that drive Unhealthy, just like when it encounters a read error. If the file is in a duplicated share, DrivePool will try to continue writing to the other duplicated file part and not interrupt the write process. The bad file part that couldn’t be written to is taken out of service and given a .bad extension. The file will show up in normal directory listing and it’s up to you to decide what to do with it.

Write errors on non-duplicated folders mark the drive Unhealthy and return an error to the caller of the write.

Limitations

Under some circumstances, a write error on a duplicated folder will not be able to take the damaged file part out of service (file may be locked, or MFT is damaged). In this case, the write will error out to the caller. This should be rare, and under most circumstances DrivePool will be able to recover the write. If this does happens, the write error wizard will restore proper folder duplication once completed.

A file in a duplicated folder that is missing a duplicated part, like when there’s a write error, can still be written to and read from. You can delete it, but you can’t rename it. This limitation will be lifted once you complete the write error wizard.

Wizard

Same as for read errors, to clear the Unhealthy status of a disk that has experienced a write error, you will need to complete a wizard. In this case, the wizard verifies the file, makes sure that it’s duplicated properly and that the bad part is taken out of service.

Duplication Errors

Duplication errors should not happen. DrivePool always maintains two copies of every file in a duplicated share. However, if something (or someone) has manually modified files on the pool part, then duplication status can be compromised. But that’s not a problem, DrivePool will detect this condition and offer a fix.

Whenever you open any file on a duplicated share, DrivePool checks to make sure that its duplication status is good. If there is a problem with one of the file parts, then it marks the drive Unhealthy and offers you a wizard to re-duplicate the problem file. This check does not take extra time, because DrivePool has to locate and open both file parts anyway, so it’s not expensive in terms of overhead.

Wizard

Run the duplication errors wizard to check all the files on the problem drive. Even though DrivePool might have detected a duplication error on one or more files, it will re-check the entire drive just to make sure that all the other files on it are duplicated properly.

Summary

StableBit DrivePool is resilient to read and write errors on duplicated folders. It tries very hard to satisfy the request without returning an error. It flags the appropriate drive and file on each error, so that you are aware of the problem as soon as it happens. DrivePool has wizards designed to recover from each type of error, even duplication errors.

As we’ve seen in the last post, DrivePool also keeps track of known and foreign disks. It makes sure that each file is duplicated on each foreign disk before lifting the Unhealthy status from it.

It is a clearly defined system, working to get your pool back on its feet when things go wrong.

Download: See the wiki

Managing a pool of disks that hold all of your data raises some interesting questions regarding pool consistency. Namely, what happens if one of the disks go missing? What happens if you need to re-install the OS? What happens to your data if one of the disks starts going bad, how does folder duplication protect your files in this case? These are all very good questions and I think the answers have to be very clear. Because it is a different system, and it’s not obvious how it would work in case of failures. After all, it’s not just enough to say that there are 2 copies of every file for redundancy if you can’t read or write that file in case of failure. Then the redundancy would be kind of useless.

In this post I will talk about what DrivePool BETA M2 brings to the table to ensure pool consistency.

Design mantra

Before going into specifics, I’d just like to say that you don’t need to know any of this to use DrivePool properly. DrivePool will detect special conditions that can compromise your data (missing disks, damaged disks, etc…), it will issue a Windows Home Server alert and flag the disk as Unhealthy or Missing in the Dashboard. It will also offer you a way to fix the problem by running a Wizard.

The idea is, if everything is ok then all the pooled disks will be tagged as Healthy. But if one of the disks has some sort of problem, its status will change. For example, when you accidentally unplug a pooled disk from the system, it will be flagged as Missing.

Generally, DrivePool will not automatically correct problems in the background without informing you, unless it is absolutely necessary to do it in real-time.

Missing disks

So you’ve dropped a disk by accident while is was running. After the initial shock wears off (pun), the thought occurs, what if now you can’t access your other data on the pool because one of  the disks is dead?

Rest assured, not with DrivePool. You pool remains accessible for read and write access across all folders, duplicated and not duplicated, even in the face of missing disks. However, there are some limitations in place to ensure data consistency.

1. If you don’t have enough physical disks / disk space left to store duplicated files, write access will be denied to duplicated folders.
2. Files with duplicated parts that were left on the missing disk cannot be moved or renamed until the missing disk is either re-connected or removed permanently.

In addition, DrivePool BETA M2 introduces missing disk management.

If it ever becomes impossible for you to reconnect a missing disk, you have the option of removing it from the pool permanently. This does 2 things. First, DrivePool re-duplicates any duplicated files that were on that disk. Second, DrivePool forgets it has ever seen this disk.

Un-duplicated files that were on that disk are of course lost.

The time it takes to complete this wizard depends on the number of duplicated files that were on the missing disk. It’s worth noting that DrivePool doesn’t have a record to keep track of duplicated files on each disk (using SQLite or some such engine). It’s one less thing to go wrong, and it means that you can’t loose duplication status because of database corruption.

Accessing the pool after an OS re-install or on a new system

First, let’s consider the best case scenario, a normal hard drive with a basic NTFS volume on it that is not part of any pool.

What happens to your data files on this hard drive if you ever re-install the OS? Obviously, they are never touched, so you would have no trouble accessing them after an OS re-install or from a different system.

I think this is a worthy goal for DrivePool to try to attain. The key here is of course the fact that the OS does not need to know anything extra about the hard drive in order to access the files on it. Specifically, that there is no external information in the registry or anywhere that is required for your OS to read a standard NTFS basic volume.

From the beginning DrivePool was designed with this idea in mind. No required external metadata in order to access the pool. This is still true with BETA M2.

This means that you can take all of your pooled drives, disconnect them from one server with DrivePool, connect them to another server running DrivePool and voila. DrivePool will automatically detect those drives as pooled drives and include them in the pool. Instant access to your files, it just works. This is of course in addition to the fact that your files are stored as standard NTFS files anyway, so even if you didn’t have another server with DrivePool installed, you could just access them from each pooled drive individually using any Windows machine.

What about shared folders? Any shared folders that were part of the old pool and that are not part of the new pool will be re-created for you automatically with default permissions. You can change the permissions to those folders from the Dashboard (this may change with BETA M3 because the permissions will be part of the pool).

I’ve said that there is no required metadata, but that doesn’t mean that there is no metadata at all. One of the things that DrivePool BETA M2 keeps track of about your pool is which disks are part of the current pool and which ones are new or foreign. When DrivePool sees a foreign disk, it flags it as Unhealthy in the Dashboard and gives you the option of running a wizard to verify the disk. This is because we don’t know the duplication status of the files on that new disk. For all we know, some files on it might be un-duplicated.

The time it takes to complete the foreign disk wizard depends on the number of un-duplicated files on that disk. If all the files are duplicated then it will take seconds. One interesting scenario is if you’ve added a single foreign disk to a system and the foreign disk has duplicated folders on it. In this case you will need to add another disk to the pool before the foreign disk can be validated. There are other edge cases, such as when there is not enough disk space left to duplicate the new files, and those are handled appropriately when encountered.

The bottom line is, just plug in a pooled disk to any machine with DrivePool installed and it will be automatically made part of the pool. You can even do this while DrivePool is running with USB disks and such. At the same time, DrivePool will ensure proper file duplication consistency.

Which brings us to the next point of interest. What happens if the actual physical disk is going bad, how does file duplication protect you in that case?

I’ll tackle that one in the next post.