StableBit DrivePool 2.1.0.432 BETA – Reparse Points

To those of you celebrating the holidays this week, happy Thanksgiving and happy Hanukkah! I have a brand new build of StableBit DrivePool to share with you today, it’s the next 2.1 BETA.

StableBit DrivePool 2.1.0.432 BETA

StableBit DrivePool 2.1.0.432 BETA

Download the latest version right here: http://stablebit.com/DrivePool/Download

This build took a while to get done (almost 2 months) and the main new feature here is support for reparse points. Reparse points are the underlying technology that Microsoft Windows uses to create links or shortcuts between files or directories, on the same drive or even among different drives.

I won’t get that technical here but I’ve written up a fairly technical explanation of the different types of reparse points over here: http://community.covecube.com/index.php?/topic/279-stablebit-drivepool-reparse-points/

Link Shell Extension

Unfortunately Windows Explorer is not very savvy when it comes to working with reparse points. I prefer to use a free shell extension called the Link Shell Extension in order to make working with reparse points easier.

You can download it for free right here: http://schinagl.priv.at/nt/hardlinkshellext/hardlinkshellext.html

Command Line

As an alternative, you can also use mklink and fsutil on the command line to work with reparse points.

A Simple Example

Reparse points can get very confusing if you try to think about how they work, so instead of doing that let’s dive right in with a simple example.

2 Folders

2 Folders

For this example, I’ve created a new pool (G:\) and have created 2 folders in the root directory of that pool, Folder1 and Folder2.

G:\Folder2\File2.txt

G:\Folder2\File2.txt

Now, in Folder2 I’ve created a new file called File2.txt. In this file I’ve put the text “This is file 2.”

What we’re going to do now is create a symbolic link in G:\Folder1 that points to G:\Folder2.

Pick Link Source

Pick Link Source

The first thing that we’re going to do is right click on G:\Folder2 and select Pick Link Source.

Drop Symbolic Link

Drop Symbolic Link

We then navigate to G:\Folder1 and right click, select Drop As… > Symbolic Link. You’ll notice that there are a lot of options here, so if you don’t know which one to pick, symbolic link is generally the best bet.

G:\Folder1\Folder2

G:\Folder1\Folder2

What we now end up with is a symbolic link G:\Folder1\Folder2 that points to G:\Folder2.

You can right click on G:\Folder1\Folder2 and select Properties > Link Properties to see some additional information about the symbolic link.

Link Properties

Link Properties

From here we can see that G:\Folder1\Folder2 actually points to ..\Folder2. You’ll notice that the link target doesn’t contain a drive letter and so this is called a relative symbolic link. If the link target were G:\Folder2, then it would be an absolute symbolic link. The Link Shell Extension will create the appropriate links for you automatically.

So let’s navigate into G:\Folder1\Folder2.

Reparse Point

Reparse Point

Notice that we see File2.txt which actually exists in G:\Folder2. Let’s open it up.

File2.txt

File2.txt

And there we have it, the same text that we placed in G:\Folder2\File2.txt in the beginning of this example.

It is important to understand that this is not a copy of File2.txt, but the original one and that G:\Folder1\Folder2 is not a copy of G:\Folder2. For example, if we place another file in G:\Folder2, it will automatically appear in G:\Folder1\Folder2, and this essentially illustrates what symbolic links (which are a type of reparse point) can do.

File Symbolic Links

In the example above I’ve shown how directory symbolic links work, but symbolic links can work with files, and there is no limit on how many symbolic links can point to the same file or directory, 1, 10 , 100, it doesn’t matter.

Multiple Symbolic Links

Multiple Symbolic Links

In the screen above I’ve created 3 separate symbolic links in G:\Folder1 to the file G:\Folder2\File2.txt. All of these essentially are the same file.

Links Across Volumes

A symbolic link can technically point to any kernel path, which includes another disk volume.

Links to Other Volumes

Links to Other Volumes

In the screen above, I’ve created a symbolic link in the G:\ pool that points to the F:\ pool. Isn’t that cool?

Other Types of Reparse Points

Symbolic links are only one type of reparse points, and in my opinion they are the most flexible. But in addition to symbolic links StableBit DrivePool supports junctions and mount points.

Other Reparse Points

Other Reparse Points

In the screen above I’ve created a mount point to C:\ and a junction to F:\ in G:\Folder1. Junctions and mount points are precursors to symbolic links, which were not implemented until Windows Vista. I suggest using symbolic links unless you have a special need not to.

Hard Links

In addition to the other types of links, the Link Shell Extension has the ability to create Hard Links. Hard Links are not implemented using reparse points and so this build of StableBit DrivePool does not support them. Hard Links are implemented directly in NTFS and can only point to a file on the same volume.

Implementation Difficulty

The reparse functionality here was notoriously difficult to implement correctly and that’s why this build took so long to get out. As a result, some functionality that I planned to include in it (such as TrueCrypt support) didn’t make it. But I just wanted to acknowledge that it’s on the way.

  • Clay White

    Hello, I switched over to DrivePool from Drivebender for this feature. I use Plex for my media server which uses symbolic links for connecting its metadata with the content. I wanted to move my Plex database off my SSD and into my drive pool so that I could create media indexes.

    At first I thought Drive pool was working fine with the symbolic links as a couple of new content seemed to have their metadata, but this has ceased to work in the same manner that DriveBender would not display my meta data either.

    Is there a fix for this or is there something I need to do to enable symbolic links besides installing the beta version of drivepool?

  • Pascal Mathis

    Would also love to hear an answer, I already scoped through the DrivePool configuration file (DrivePool.Service.exe.default.config) and it seems like ReparsePoints are enabled by default.

    Unfortunately, I’ve got the same problem as Clay White. Plex fetches the media data itself without any problems, but downloading of artwork does not work because it relies on symbolic links as said.

  • I do have Plex installed here on one of the servers but I haven’t tried moving the metadata database onto the pool itself.

    I just did a scan over the database folder and I don’t see any symbolic links on my installation. I have several movies and TV shows that I own stored in Plex and I do see artwork showing up correctly.

    I checked for symbolic links from the command line by switching to the “%LOCALAPPDATA%Plex Media Server” folder and typing “dir /A:L /S”.

    If you’d like to, you can open up a support case @ http://stablebit.com/contact and we’ll try to assist you guys further. In particular it would be useful to see the Plex log files.

  • Plex specifically relies on “Hard links” for the image metadata. Not Symbolic links. That means that they are not supported on the pool.

  • Pascal

    Plex specifically relies on “Hard links” for the image metadata. Not Symbolic links. That means that they are not supported on the pool.

  • Pascal Mathis

    Christopher

    Thanks for your answer. I currently have one big pool containing about 2TiB of data + apps&games installed on it. I’ve learnt though, that this is probably not the best way of using DrivePool. Also, some DRM don’t like DrivePool (Mafia II for example)

    I’ll probably have to reinstall my OS and reorganize everything, so that only data (movies for example) is stored on DrivePool. Now to my Plex issue: I let it run on my SSD (C:) and use Synkron to hourly backup the contents of the Plex folder to my DrivePool with redundancy. That works like a charm.

    One last question though: Are hard links ever going to be supported or isnt that even possible?

  • Well, Alex (the developer) actually runs VMs off of the pool, so running games should be fine as well. Though, if the game’s DRM prevents that, then obviously not.

    As for hard links, the problem is with how they work. hardlinks are a part of the NTFS file system, and not easily emulated. It is possible that we could do this in the future, but it will be significantly more complex than the reparse point support.

    But as for why Synkron works, I believe it’s grabbing the linked file, so it’s creating multiple copies. There is nothing wrong with that, but it does take up significantly more room.

    To be blunt here, I don’t know why Plex doesn’t do this linking in their SQLite database, instead of relying on the file system to do this. It would be more efficient and compatible, if they did.
    And they may not have a choice if Microsoft makes a serious push to ReFS, as that doesn’t support hard links either.

  • Pascal Mathis

    Christopher

    Thanks for your very detailed answer. I’m going to open up a support ticket at Plex’s Support Site and try to propose them what you just said.

    Indeed, Synkron does copy the files, so they are actually duplicated. However, these are only some album arts which do not use a lot of space.

    Still one question though: What do you recommend more to your customers?

    A) Using the pool for media files, documents and other huge files like VMs

    B) Same as A + Games and Apps (with sometimes tenthousands of small files…)

    As far as Ive noticed, theres a little performance hit when thousands of file are accessed.

  • Pascal,

    You are very welcome. We like to be very detailed, whenver possible.

    As for which scenario to use, it really depends. On what you’re comfortable doing, and the programs in question.

    There shouldn’t be an issue with either option (other than in the case of some DRM), but I’d recommend option A.
    However, if your system disk is rather limited on space (such as in the case of a SSD), then using the pool for games and apps may be a viable option.

    Sorry if it seems like I’m trying to avoid a straight answer here. I’m not. As I said, either should work file.

    As for the performance hit, well this is normal, even on normal volumes. There are a lot more queries to the file system, and that can cause it to slow down. Eg, when you copy a 4GB file, it copies much faster than 4GBs of jpgs.
    If duplication is enabled for the folders in question, this slowdown may be amplified a bit. Especially if you have “Read Striping” enabled (as it uses additional overhead).

    If speed and performance are critical though, then I’d really recommend a dedicated drive for the software instead.

  • Pascal Mathis

    Hello Christopher

    Based on your answer, I came up with the following solution and I think it is going to be the best solution for me. One SSD, containing my OS and some high performance apps. Then there are 4 disks left – 3 of them will only be used for DrivePool and contain media files, documents and so on. As currently, I will remove their driveletters, so they are completely invisible.

    The last disk which is left, will not be used by DrivePool, so that applications like Steam (and all of the games) can be installed there. This should avoid DRM issues and reduce the performance hit with small files.

    Thanks once more, your answers helped me to do decide which way I should take. And… DrivePool rocks! First used Storage Spaces, was really disappointed and switched over to your product. Can’t believe how MS was able to release a product which was lacking so many features…