StableBit DrivePool went out of BETA and into a production final at the end of last month (March 2012), and I’d like to take some time to talk about how it all began, how we got here, and perhaps a bit about the future.
For more information about DrivePool and to purchase a license visit:
http://stablebit.com/drivepool
I’d like to make this post a bit more personal and I’d like to tell you about the development effort from my perspective.
It’s the end of the world as we know it
In 2011, some time around January, Microsoft announced that they are dropping the drive extender feature from Windows Home Server 2011. Well… what can you say about something like that. It was a bombshell. Drive Extender was the heart and soul of the original Windows Home Server operating system, and taking it out is basically killing the OS, in my opinion. Their reason for doing so, as I understood it, was because it never worked correctly in the first place. Which of course made no sense at all because it has been working fine for so many people over the years.
Needless to say, there was a big outcry from the community and I recall that people started brainstorming about alternatives at the time.
Covecube Inc. was already invested into the Windows Home Server ecosystem with a 6 month development effort of Scanner 1.0, which was planned for a Windows Home Server 2011 port.
An interesting dilemma came to light. There were really 2 options for the company, move on to something else, or fix the problem with Windows Home Server 2011. The fix to the problem would obviously involve developing a brand new pooling system, that would work seamlessly with the existing WHS 2011 infrastructure. Also, it would have to be functional rather quickly, because everyone was expecting to see a pooling solution in WHS 2011, that was about to ship.
Given those requirements, the solution had to be simple and powerful. The real question was, can such a pooling architecture be created?
DrivePool emerges
On Feb. 11 2011 I announced DrivePool, first on the We Got Served forums and later in other places.
Here’s the original announcement:
http://forum.wegotserved.com/index.php/topic/17628-introducing-stablebit-drivepool-for-the-windows-home-server-2011/
As you can see, at this point, the screenshots show a working version of DrivePool.
I started Covecube Inc. back in Feb. 2010 with the simple idea of making good software that fulfills 3 needs. It has to have a deep technological footing, a functional and simple interface, and a great looking UI. The http://covecube.com web site explains this best.
I’ve tried to put as much of those things into DrivePool as possible.
The Architecture
The untold story behind those first screenshots was that I didn’t want to post anything until I was sure that what I posted was based on a solid architecture. You see, my feeling is that bugs can be fixed with time, but a bad architecture can’t. So to me it was extremely important to get the underlying foundation of a solid pooling solution in place, and I spent many sleepless nights before ever posting that, to make sure that what I was posting was a working prototype.
The architecture that was part of that first build which came to be known as DrivePool BETA M1 is still largely what is in use today, 6000 and some odd builds later. Although the file system was swapped from BETA M3 to BETA M4, as you will soon see.
Yes, DrivePool did start at build 1!
DrivePool BETA M1
DrivePool BETA M1 (what was just known as the DrivePool BETA at the time) was released a few weeks after the initial announcement. It was actually delayed by some time because of some very intricate issues that had to do with directory listings. Those were fixed, and luckily no one knew about them, until today that is.
DrivePool BETA M1 depended on Dokan, an open source project that was designed to make file system development easy. There are a number of architectures like this, some commercial, that let anyone make their own file system easily. Dokan was LGPL licensed, so there was no conflict shipping it as part of a commercial product. At one point I even set up a fork of Dokan for DrivePool so that I could check in fixes publicly.
Using a “file system for dummies” approach is actually a really good way to prototype a new system. It’s quick and easy to make changes. The disadvantages are mainly performance and compatibility. I’ve always known that we would have to develop our own file system eventually and move off Dokan before DrivePool goes final. But it made perfect sense to ship the early BETAs with it in order to work out the details of the architecture instead of spending time rewriting the same kernel driver 3 times.
DrivePool BETA M2
DrivePool BETA M1 was a great success. It worked surprisingly well, considering that it was just conceived a few months ago. Continuously patching bugs in Dokan also turned out to work well too. Many issues present in the first BETAs started to disappear, and it really started to look like Dokan could actually hold its own in a final release.
DrivePool BETA M2 was made public on May 18 2011. It’s main goal was to complete the pooling infrastructure. Where BETA M1 has only some basic pooling functionality, BETA M2 added a lot of disk management features regarding foreign disks and missing disks. It also introduced the extremely extensive logging infrastructure built into the Dokan kernel driver and the DrivePool service.
The new logging capabilities were a massive undertaking that most people were not aware of, but it made finding and fixing I/O bugs orders of magnitude simpler than before.
At this point it started to look like BETA M3 would become the final, and while I always hesitated to give out any release dates, June / July 2011 were always a good targets to try and hit.
DrivePool BETA M3
BETA M3 was all about integrating DrivePool into the existing WHS 2011 infrastructure. This involved a lot of research / reverse engineering and trial and error. It actually turned out to be much harder than any of the previous BETAs, including developing the pooing architecture itself.
BETA M3 eventually shipped on July 26 2011 and it was pretty much a feature complete build. I have to say, considering the complexities involved with the integration, it worked very well.
DrivePool BETA M4
BETA M4 was originally meant to be a performance enhancing build. Essentially, around BETA M2, the decision was made to stick with the Dokan file system driver for the final, but to replace pieces of its code with our own file system driver. This was actually possible because of the way everything was written.
A few of the later M3 builds started down this path and this hybrid file system actually shipped publicly, but the advantages of having a real kernel-based file system driver became painfully apparent. Most of the complexities introduced in BETA M3 would just disappear. It would be far better performing, easier to maintain and far more compatible.
It was obvious that Dokan, or any of the expensive commercial user mode file system libraries available would simply not work for a rock solid DrivePool final. In Windows, file systems were written in the kernel and that was the only way to do it.
CoveFS
Dokan was abandoned, and my original plan for building a brand new pooling file system, back from BETA M1, was put into effect, and it was called CoveFS.
CoveFS was a completely different approach that Dokan, no user mode dependencies, a real file system that just works. Another thing that had to be written was a virtual disk driver, and that ended up being called the CoveFS Disk.
Once the first internal builds of BETA M4 started coming out, it became obvious that this was the right approach. It was fast and long lists of issues from BETA M3 just vanished.
Much of the BETA M3 integration code was deprecated and BETA M4 went public on: Jan. 12 2012
DrivePool 1.0 Release
I have to say, after the long BETA testing period of DrivePool 1.0, I am completely happy with the end product. It is everything that I imagined back when the project was started, just over a year ago. It has a real native file system, it’s super fast and it’s very compatible. It has a finely tuned disk balancing algorithms, with multiple innovative approaches there, file duplication that is fast and reliable.
Personally, I’ve migrated all of my personal data from the original Windows Home Server to DrivePool, and I will continue to use it until something better comes along.
Hopefully it will be DrivePool 2.0.
The Future
DrivePool’s future is simple. I am committed to fixing any remaining issues and to support the existing product. But future development will depend on whether the money invested the 1+ years of development can be recouped. It is really as simple as that, Covecube Inc. as a business will need to survive and prosper in order for new products to be developed.
Recently, a 10,000 pound Gorilla, Microsoft, walked back into the pooling arena. As some of you may know, after denouncing their earlier pooling solution as a wasted effort, they had decided that it was not a mistake after all. They are working on developing their own pooling solution.
I can say this, there is already a plan laid out for DrivePool 2.0 and it is very cool and original.
I’m not announcing any new features at this time, but hopefully there is enough interest in such software to make a continued development effort worthwhile. I would sure love to continue innovating in this area.