Fstransform – Optimus Tux

File system conversion is not an everyday thing. For that matter, it’s not even an every year thing. But when you do need to convert from one format to another, the operation is usually long, tedious and sometimes destructive. Most often, you would copy files to a backup location, re-format the partition, then copy the data back. The notion of being able to do a seamless, live conversion sounds like a cool thing.

Fstranform is a tool designed to offer in-place file system conversions without a need for a backup. This program does its magic by mounting several loopback devices and uses them to shuffle bytes to and fro while it restructures the file system layout. The advantages – if proven successful, of course – are in that you do not need to worry about backup devices (could be many terabytes), and you could potentially save time. Sold! Let’s see how it works.

Before you begin

Of course, I must warn you. The naughty Byteman might come to bite your data. File system manipulation is never a trivial thing. You should always have data backups, whether you ever choose to convert any of your partitions or not. Backups are healthy, and they will protect you from data loss. This is doubly true when you start tinkering with a new tool that is going to significantly alter your partition structure.

Moreover, there are also some technical limitations to how fstransform works and what it does. First, there’s a list of supported file systems – ext2/3/4, jfs, xfs, and reiser. Then, there’s the list of unsupported file systems – btrfs, fat, exfat, and ntfs. The second category means you can use fstransform, but there’s no guarantee the conversion will work – or that your data won’t be irrecoverably lost.

Fstransform is also a command-line tool, so you need to be careful with the disk and partition notation. For safety reasons, fstransform will not work with the root partition or any mounted devices, and there has to be sufficient free space for the operation. The program will do all these checks for you, but still.

Lastly, ask yourselves why. Do you really need to convert your file system? This is not something most people will ever do. I can think of very few specific cases where this might be necessary, like the limitation on the number of files in a directory, the sub-directory max depth, path string length, corner cases with inodes and hard links, partition size, and possibly performance. Normally, this is data center stuff, rarely home stuff.

Now, fstransform should be available in the system repo distributions, so the setup is quite simple. Let’s see how this utility behaves, and whether you should consider it for your data.

Linux to Linux

Mandatory listening to music at this point – David Bowie’s Ashes to Ashes. So this is the easy part. You simply choose the device you want to convert (e.g. /dev/sdc4) and the filesystem to which you want to convert it (e.g. xfs). The typical syntax would be:

fstransform /dev/sdd5 ext3

My test was to transform ext4 to xfs. I let the command run and hit a snag. The program informed me that it could not find mkfs.xfs. Indeed, the default Ubuntu system (the test distro used) does not have xfsprogs installed by default. I had to add the package before proceeding.

17:40:50 fstransform: formatting loop device '/dev/loop4' with file-system type 'xfs'...
17:40:50 mkfs: failed to execute mkfs.xfs: No such file or directory

After that, fstransform does a whole set of checks – whether the partition is mounted or not, whether it’s writable, then it does a filesystem check, and a whole lot more. Then, it runs a simulation, and only then will it actually run the real conversion. You have two separate opportunities to abort the run. Finally, if the execution of the conversion process gets aborted, you will have the ability to resume manually. This should put you at ease before hitting Enter that one last time.

Fstransform completed the first few steps very quickly, including data move, simulation and all that. Then, it relocated the data without any problems, but the last step, the clearing of free space from the device took forever.

17:48:41 fsremap: analysis completed: 84.00 megabytes must be relocated
17:48:41 fsremap: allocated 84.00 megabytes RAM as memory buffer
17:48:41 fsremap: primary-storage is 11.00 megabytes, initialized and mmapped() to contiguous RAM
17:48:41 fsremap: (simulated) starting in-place remapping. this may take a LONG time ...
17:48:41 fsremap: (simulated) progress: 6.5% done, 84.0 megabytes still to relocate
17:48:41 fsremap: (simulated) clearing 14.83 gigabytes free-space from device ...
17:48:41 fsremap: (simulated) job completed.
17:48:41 fstransform: launching '/usr/sbin/fsremap' in REAL mode to perform in-place remapping.
 17:48:41 fsremap: starting job 3, persistence data and logs are in '/var/tmp/fstransform/fsremap.job.3'
 17:48:41 fsremap: if this job is interrupted, for example by a power failure,
 17:48:41 fsremap: you CAN RESUME it with: /usr/sbin/fsremap -q --resume-job=3 -- /dev/sdb1

I was not able to access the device while this command was running, and it seemed to be running awfully slow. But based on the earlier move performance, we’re talking something like 5 min per GB, so it can take a while, even for small thumb drives.

17:49:06 fsremap: starting in-place remapping. this may take a LONG time ...
17:49:08 fsremap: progress: 6.5% done, 84.0 megabytes still to relocate
17:49:34 fsremap: clearing 14.83 gigabytes free-space from device ...

Clearing free space from device

Eventually, the command completed – successfully. As I expected, it took about 63 minutes to finish, which is an awful lot for roughly 15 GB worth of empty space. Indeed, fsremap barely uses CPU when running, so I guess the move buffers can be bigger. We’re not even approaching the thumb drive write speed limit, or the USB 2.0 port speed limit.

18:51:49 fstransform: running again '/sbin/fsck' (disk check) on device '/dev/sdb1'
18:51:49 fsck: fsck from util-linux 2.31.1 
18:51:49 fsck: /sbin/fsck.xfs: XFS file system. 
18:51:49 fstransform: completed successfully. device '/dev/sdb1' now contains 'xfs' file-system

Completed successfully

The command was successful – and the data was intact. All the files were there, and nothing was missing. No strange names, links, permissions, or anything of that sort. The conversion from Ext4 to xfs worked well, although it was painfully slow. Now, let’s make this a bit more challenging.

Unsupported file systems

This is a trickier use case, but also more interesting. If anything, for home users, the incentive for using fstransform will be the conversion from Windows file systems, so to speak, like FAT and NTFS to Linux formats.

I tested this twice, once converting from Ext4 to NTFS and then the other way around. I also tried with large devices, i.e. 250GB partition on an external hard disk, and then a more modest 16GB partition on a thumb drive. Like earlier, fstransform does a lot of safety checks. However, there are some notable differences.

13:27:11 fstransform: file-system type 'fuseblk' is a placeholder name for FUSE... ignoring it
 
13:27:11 ERROR! fstransform: failed to detect device current file system type
 
13:27:11 ERROR! fstransform: this program is tested ONLY on file systems: minix ext2 ext3 ext4 reiserfs jfs xfs
 
13:27:11 ERROR! fstransform: cowardly refusing to run. you can use option '--force-untested-file-systems' if you know what you are doing (DANGEROUS, you can LOSE your DATA)

You cannot run the tool against an unsupported file system without a force flag. This is yet another safety check to deter you from doing foolish things. Anyway, Ext4 to NTFS first. At first, everything seemed to be okay, and the tool went through the checks, and then started eating CPU cycles.

13:29:44 fstransform: moving '/dev/sdb1' contents into the loop file. 
13:29:44 fstransform: this may take a long time, please be patient... 
13:29:45 fsmove: move() skipped `/tmp/fstransform.mount.9892/.fstransform.loop.9892', matches exclude list 
13:29:47 fsmove: progress: 5.0% done, 1.2 gigabytes still to move 
13:29:47 fsmove: job completed.
13:29:55 fstransform: mounting again device '/dev/sdb1' read-only
13:29:55 fstransform: launching '/usr/sbin/fsremap' in simulated mode
13:29:55 fsremap: starting job 1, persistence data and logs are in '/var/tmp/fstransform/fsremap.job.1'
13:29:55 fsremap: if this job is interrupted, for example by a power failure,
13:29:55 fsremap: you CAN RESUME it with: /usr/sbin/fsremap -n -q --resume-job=1 -- /dev/sdb1

I let the program run for about half an hour before I decided to check logs – and do some online reading to figure out how fast fstransform is. Based on some of the tickets on the official page, fstransform needs something like 24-30 hours to convert 1 TB of data. This translates into roughly 700 MB per minute, which is relatively slow. This meant my 250GB partition would take some 4-6 hours.

Top, NTFS operation

Furthermore, I noticed in the logs that a certain mapping operation wasn’t supported, and that a fallback method was used. Again, the GitHub issues page for the project reveals several problems, like the size limitation with the FIBMAP operation and the very slow performance with NTFS->Ext procedure. Strace showed really slow data write, probably something like 10-20 blocks per second.

2018-08-23 13:29:55 INFO [io/io_posix.open_dev(154)] device length is 292.97 gigabytes
2018-08-23 13:29:55 INFO [remap.run(716)] analyzing device '/dev/sdb1', this may take some minutes ...
2018-08-23 13:29:55 DEBUG [io/extent_posix.ff_linux_fiemap_ioctl(185)] ioctl(7, FS_IOC_FIEMAP, extents[0]) failed (Operation not supported), falling back on ioctl(FIBMAP) ...

I decided to abort this and try the other way around: Ext4 to NTFS. I also toned down my ambition and went with a much smaller device. Again, there was a lot of verbose output from fstransform, which may look alarming, but it is designed to protect your data.

The operation started running, and the first part is moving data to the loopback device. The fsmove part was really slow, with approx. 20 MB copied every 4-5 seconds. This translates into only about 300 MB/min speed, which is probably due to the whole ntfs-3g to fuse to ext4 translation. The limitation is highlighted in the project notes.

13:57:12 fsmove: progress: 55.1% done, 173.0 megabytes still to move, estimated 10 seconds left
13:57:14 fsmove: progress: 60.1% done, 153.8 megabytes still to move, estimated 8 seconds left
13:57:18 fsmove: progress: 65.1% done, 134.5 megabytes still to move, estimated 7 seconds left
13:57:23 fsmove: progress: 70.1% done, 115.2 megabytes still to move, estimated 7 seconds left
13:57:24 fsmove: move() skipped `/tmp/fstransform.mount.9195/.fstransform.loop.9195', matches exclude list
13:57:24 fsmove: job completed.

But then, the operation seized on filesystem check. I let the command below run for a good ten minutes before invoking the power of strace. It showed incomplete system calls. I was unable to kill all the processes, and some of them were stuck in the D state, and had to actually reboot.

13:57:24 fstransform: unmounting and running '/sbin/ntfsresize' (disk check) on loop file '/tmp/fstransform.mount.9195/.fstransform.loop.9195'

The good thing about this? All my data was still intact!

Despite my rather unorthodox approach to the file system conversion and the fact I had aborted the operation with the kill signal and the reboot thereafter, the original Ext4 partition and all its contents were preserved without a hitch. Perhaps fstransform cannot do miracles when it comes to moving things to unsupported formats, but it sure can protect your data, and it does that beautifully.

For me, this is the highest testament to the stability and robustness of this program. The implementation of drivers and filesystem is an easy thing to solve. Making sure user data remains untouched is the real magic, and I only have utmost praise. I was purposefully being violent and rough with the thumb drive, the partition and the process running in the terminal window, and yet, the fstransform survived.

Conclusion

There are two ways you must look at fstransform and judge its worth. One, the quality of the file system conversion, with its associated workflow, accuracy, speed, and results. Two, the safety of the user’s data. On the first front, fstransform didn’t deliver a stellar performance. Even with supported file systems, the operations can take a very long time. With unsupported file systems, the speed is abysmal, and you should not expect a happy ending. There’s a good reason why the project author decided to disclaim and hide certain file systems.

However, when it comes to data – awesome. It really protects your stuff until the operation has completed successfully, and if it does not, which is often the case, then it will revert back to the old state, no harm done. You may lose time but you will not lose important, valuable information. This is by far the more cardinal element of this equation. Over time, I am more than certain that fstransform will make its workflow smoother, easier, faster, and whatnot. But without the data integrity element, none of that will ever matter.

Which is why, despite the snags I’ve encountered, I can recommend you take this program for a spin and see how well it works for you. You should not throw caution to the wind, but you can expect decent and sane behavior, and your data should not disappear. Worth checking and testing, and I am looking forward to see how this utility evolves and improves. At the moment, it’s very rough, and the results are not that good, so ’tis a domain of hardcore nerds and experts. But do take a look. Take care.

Cheers.