New Pandora’s box: Install Linux and Windows onto the same NTFS partition

Linux 5.15 is shipped with a brand new driver for Microsoft’s classic NTFS filesystem, NTFS3. Unlike the decades-old open-source NTFS-3G project, which is based on FUSE and have always received criticism for breaking existing filesystems, NTFS3 is a new driver that is designed to be compatible with contemporary NTFS filesystems, while providing safer read/write operations. This makes it possible to install Linux onto NTFS (as is with most other filesystems), and opens up a whole new can of worms: run Linux alongside Windows, TOGETHER.

Sounds WEIRD to me. I’m going to do this experiment on my Proxmox VE cluster.

Preparation

Archiso

At the time of writing this article, the latest Arch Linux ISO (2021.11.01) was shipped with Kernel 5.14.15 - no new NTFS3 driver. I need to create one for myself or this won’t work.

Archiso is Arch’s official tool for creating custom ISO images. I’m not normally an Arch user, so I choose to install Arch first from an official ISO (20211101) before wiping it.

After this temporary system is set up, I just follow the Archiso guide and receive my own archlinux-2021.11.22-x86_64.iso with no trouble. It has Kernel 5.15.4 packed.

I copy the ISO onto the Proxmox VE host system, reboot the VM with this new ISO and wipe /dev/sda2 to avoid (possible) further issues with the Windows installer. I also format /dev/sda1 again to ensure I’m really starting over anew.

Install Windows

Since NTFS is developed by Microsoft and for Windows, it seems reasonable to assume Windows is best suited for NTFS. So I’ll install Windows first lest it recognizes the filesystem created by mkfs.ntfs (from the old ntfs-3g package) as “foreign” and complains anyhow.

The installation process of Windows 10 has always been as boring and mundane as it is, so I’m not going to be verbose here. Following the usual steps, except that the disk has already been partitioned, it’s easy to get Windows 10 up and ready.

Proceeding through the out-of-box experience and I get to the desktop. There’s not many things of interest here, so I just shutdown the VM and take a snapshot.

Now it’s time to get this compound monstrosity set up.

The Main Show

Swap the CD/DVD drive image for the newly created archiso and boot it up:

With the proper Linux kernel equipped, I can now mount the NTFS partition create by Windows installer. It seems NTFS is sophisticated enough to even allow Unix filesystem attibutes, like file modes (permissions) and ownership, as well as “special file types” like symbolic links and named sockets (Unix domain sockets). This may hint that bootstrapping a Linux system should not be too problematic.

fdisk -l /dev/sda  # confirm partition layout
mount -t ntfs3 /dev/sda2 /mnt
mkdir -p /mnt/boot/efi
mount /dev/sda1 /mnt/boot/efi
pacstrap /mnt base linux linux-firmware

Indeed, pacstrap goes so smoothly that I almost forget it’s on a non-native filesystem. The only thing that makes me concerned is that there’s no fsck tool for NTFS (file not found: fsck.ntfs3 in console output).

Now I can chroot into the system and set up the rest of the system.

genfstab -U /mnt >> /mnt/etc/fstab
arch-chroot /mnt
ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
vim /etc/locale.gen  # add en_US.UTF-8 UTF-8
echo monster > /etc/hostname
passwd -d root
exit  # quit chroot environment, return to archiso

Fixing the bootloader is a bit different than usual, as Linux detects NTFS partitions as ntfs, not ntfs3. In case of auto mounting, Linux will try to mount with -t ntfs, which is not available (it’s provided by ntfs-3g). Fortunately, there’s a rootfstype= kernel command-line parameter to override the “filesystem type” parameter when mounting.

Putting this into action:

arch-chroot /mnt
# configure networking
pacman -Sy grub efibootmgr
vim /etc/default/grub
# remove "quiet" from GRUB_CMDLINE_LINUX
# set GRUB_CMDLINE_LINUX_DEFAULT="rootfstype=ntfs3"
grub-install
grub-mkconfig -o /boot/grub/grub.cfg

To make things a bit more interesting, I’m adding a desktop environment:

pacman -Sy gnome
# select some items - not everything

And configure networking as well:

cd /etc/systemd/network
vim ens18.network
cd ../system
ln -s /lib/systemd/system/systemd-networkd.service multi-user.target.wants/

All set, let’s give it a try.

Usage experience

Arch Linux plays surprisingly well with the new NTFS3 filesystem driver.

To keep things simple, I didn’t install too much software. During my testing, the only issue I encountered was that ldconfig never worked. It always aborts.

A non-issue is that there’s no working fsck tool, and there’s a systemd service “Fsck at boot” that consequently fails. It’s not as useful so I just disabled it.

The pioneer from r/archlinux said the system breaks after a few reboots, which didn’t happen to me. On the contrary, my Arch Linux was considerably resistant to Windows, and survived multiple Windows Updates, one Microsoft Update, and a few more. It even survived a CHKDSK despite a bunch of files being reported for “invalid filename” because Windows dislikes colons in filenames (not that NTFS doesn’t support).

Thoughts

I must admit I’m amazed at how exquisitely NTFS is designed. It’s so mature that it hasn’t even been updated since Windows XP. One important part of NTFS is its Extended Attributes (EA) for files. Every NTFS filesystem contains a special file named $MFT located under its root directory. This is the metadata for all files, including file names, “normal attributes” and ACL, among which is the EA. Every file has an associated entry for EA, which can contain an arbitrary number of attributes (key-value pairs). In fact, the first generation of Windows Subsystem for Linux (WSL) stores Linux file modes and permissions using custom EA keys, which gets adapted by the new NTFS3 driver. Other EA keys are also used as needed, like security.capability, which is a 20-byte bitset. (Interestingly, EA was originally designed for compatibility with HPFS, which also has a similarly-extensible “Extended Attributes”.)

The new NTFS3 driver is a delighting improvement to the Linux ecosystem. Complaints about the classic NTFS-3G driver have always been around. Performance was one of the primary concerns because it not only is based on FUSE (Filesystem in USErspace), but also badly optimized. Use of FUSE means extra context switches when accessing files, which, paired with hard-coded 4 KiB read/write unit, delivers unusually slow access speeds.

While the NTFS3 driver is a bit more optimized, concerns around compatibility are still encompassing. This is mainly because it’s still built on knowledge obtained from reverse engineering than technical documentation and standard. Fortunately, stability for NTFS-3G is already at a satisfactory level, and the new driver is thought to be more reliable than the old one.

Besides, this is a perfect example of Linux’s inclusiveness. Years before the commencement of the new NTFS3 driver, attempts were made to run Linux on top of NTFS using NTFS-3G. This leads to an interesting question: Will Linux run on top of FAT32? Technical difficulties are more conspicuous and critical this time, like lack of support and extensibility for file modes and more. I’ll explore into this challenge and share my findings in a subsequent blog post. Stay tuned!

Links & Credits

• Pioneer from r/archlinux: Arch Linux on NTFS3!

• Original idea by a GitHub user: Installing Windows and Linux into the same partition