A file system is one of those implementations in an operating system that everyone uses but most are not aware of how it works.
Consider the older days when offices would keep records and files inside folders, bundle them into stacks, and put them on their respective shelves to where they belong. You could group the folders based on their registered dates or group them based on which area they refer to. So many ways to keep your files, yet each of them served a purpose, which was to ease our work by being kept in a structured manner and being found easily.
A file system is an architecture defining how files are stored and retrieved. It defines a format and logic of – if a newly created file will be saved, how will it be saved, what extra data will it be saved with, where will it be saved, and how will it be accessed from where it was saved.
File systems are defined based on where they are used. There are file systems defined for operating systems, networks, databases, and other special-purpose file systems. When talking about an OS, a file system may be defined as a hard disk, flash memory, RAM, or optical discs.
In this article, we will be focussing on the file system for hard disks on a Linux OS and discuss which type of file system is suitable. Before that, let’s get ourselves familiar with the various characteristics associated with a file system.
The architecture of a File System:
A file system mainly consists of 3 layers. From top to bottom:
- Logical file system:interacts with the user application with the help of an API to provide open, read, close etc. operations and passes requests to the layer below.
- Virtual file system:enables multiple instances of the physical file system to run concurrently.
- Physical file system:handles the physical aspect of the disk while managing and storing physical memory blocks being read and written.
Characteristics of a File System
- Space Management: how the data is stored on a storage device. Pertaining to the memory blocks and fragmentation practices applied in it.
- Filename: a file system may have certain restrictions to file names such as the name length, the use of special characters, and case sensitive-ness.
- Directory: the directories/folders may store files in a linear or hierarchical manner while maintaining an index table of all the files contained in that directory or subdirectory.
- Metadata: for each file stored, the file system stores various information about that file’s existence such as its data length, its access permissions, device type, modified date-time, and other attributes. This is called metadata.
- Utilities: file systems provide features for initializing, deleting, renaming, moving, copying, backup, recovery, and control access of files and folders.
- Design: due to their implementations, file systems have limitations on the amount of data they can store.
Some important terms:
Journaling:
Journaling file systems keep a log called the journal, that keeps a track of the changes made to a file but not yet permanently committed to the disk so that in case of a system failure the changes lost can be brought back.
Versioning:
Versioning file systems store previously saved versions of a file, i.e. the copies of a file is stored based on previous commits to the disk in a minutely or hourly manner to create a backup.
Inode:
The index node is the representation of any file or directory based on the parameters – size, permission, ownership, and location of the file and directory.
Now, we come to part where we discuss the various implementations of the file system in Linux for disk storage devices.
Linux File Systems:
Note: Cluster and distributed file systems will not be included for simplicity.
ext [Extended File System]:
Implemented in 1992, it is the first file system specifically designed for Linux. It is the first member of the ext family of file systems.
ext2:
The second ext was developed in 1993. It is a non-journaling file system that is preferred to be used with flash drives and SSDs. It solved the problems of separate timestamp for access, inode modification and data modification. Due to not being journaled, it is slow to load at boot time.
Xiafs:
Also developed in 1993, this file system was less powerful and functional than ext2 and is no longer in use anywhere.
ext3:
The third ext developed in 1999 is a journaling file system. It is reliable and unlike ext2, it prevents long delays at system boot if the file system is in an inconsistent state after an unclean shutdown. Other factors that make it better and different than ext2 is online file system growth and HTree indexing for large directories.
JFS [Journaled File System]:
First created by IBM in 1990, the original JFS was taken to open source to be implemented for Linux in 1999. JFS performs well under different kinds of load, but is not commonly used anymore due to releasing of ext4 in 2006 which gives better performance.
ReiserFS:
It is a journaling file system developed in 2001. Despite its earlier issues, it has tail packing as a scheme to reduce internal fragmentation. It uses a B+ Tree that gave less than linear time in directory lookups and updates. It was the default file system in SUSE Linux till version 6.4, until switching to ext3 in 2006 for version 10.2.
XFS:
XFS is a 64-bit journaling file system and was ported to Linux in 2001. It now acts as the default file system for many Linux distributions. It provides features like snapshots, online defragmentation, sparse files, variable block sizes, and excellent capacity. It also excels at parallel I/O operations.
SquashFS:
Developed in 2002, this file system is read-only and is used only with embedded systems where low overhead is needed.
Reiser4:
It is an incremental model to ReiserFS. It was developed in 2004. However, it is not widely adapted or supported on many Linux distributions.
ext4:
The fourth ext developed in 2006, is a journaling file system. It has backward compatibility with ext3 and ext2 and it provides several other features, some of which are persistent pre-allocation, unlimited number of subdirectories, metadata checksumming and large file size. ext4 is the default file system for many Linux distributions and also has compatibility with Windows and Macintosh.
btrfs [Better/Butter/B-tree FS]:
It was developed in 2007. It provides many features such as snapshotting, drive pooling, data scrubbing, self-healing and online defragmentation. It is the default file system for Fedora Workstation.
bcachefs:
This is a copy-on-write file system that was first announced in 2015 with the goal of performing better than btrfs and ext4. Its features include full filesystem encryption, native compression, snapshots, and 64-bit check summing.
Others: Linux also has support for file systems of operating systems such as NTFS and exFAT, but these do no support standard Unix permission settings. They are mostly used for interoperability with other operating systems.
Below is a table, listing out the criteria on which filesystems can be compared:
Please note that there are more criteria than the ones listed in the table. This table is supposed to give you an idea of how file systems have evolved.
Max. filename length [bytes] | 255 | 255 | 248 | 255 | 255 | 4032 255 characters | 255 | 3976 | 255 | 255 |
Allowable characters in directory entries [Any byte] | except NUL | except NUL, / | except NUL | except NUL or / | Any Unicode except NUL | except NUL or / | except NUL | except NUL, / | except NUL, / | except NUL, / |
| Max. pathname length | Undefined | Undefined | Undefined | Undefined | Undefined | Undefined | Undefined | Undefined | Undefined | Undefined |
| Max. file size | 2 GB | 16GB – 2TB | 64MB | 16GB – 2TB | 4PB | 8TB | 8EB | 8TB [on x86] | 16GB – 16TB | 16EB |
| Max. volume size | 2 GB | 2TB – 32TB | 2GB | 2TB – 32TB | 32PB | 16TB | 8EB | – | 1EB | 16EB |
| Max. no. of files | – | – | – | – | – | – | – | – | 2^32 | 2^64 |
Metadata only journaling | No | No | No | Yes | Yes | Yes | Yes | No | Yes | No |
| Compression | No | No | No | No | No | No | No | Yes | No | Yes |
| Block sub-allocation | No | No | No | No | Yes | Yes | No | Yes | No | Yes |
| Online grow | No | No | – | Yes | No | Yes | Yes | Yes | Yes | Yes |
| Encryption | No | No | No | No | No | No | No | Yes | Yes [experimental] | No |
| Checksum | No | No | No | No | No | No | Partial | No | Partial | Yes |
Observations
We see that XFS, ext4 and btrfs perform the best among all the other file systems. In fact, btrfs looks as if it’s almost the best. Despite that, the ext family of file systems has been the default for most Linux distributions for a long time. So what is it that made the developers choose ext4 as the default rather than btrfs or XFS? Since ext4 is so important for this discussion, lets describe it a bit more.
ext4:
Ext4 was designed to be backward compatible with ext3 and ext2, its previous generations. It’s better than the previous generations in the following ways:
- It provides a large file system as described in the table above.
- Utilizes extents that improves large file performance and reduces fragmentation.
- Provides persistent pre-allocation which guarantees space allocation and contiguous memory.
- Delayed allocation improves performance and reduces fragmentation by effectively allocating larger amounts of data at a time.
- It uses HTree indices to allow unlimited amount of subdirectories.
- Performs journal checksumming which allows the file system to realize that some of its entries are invalid or out of order after a crash.
- Support for time-of-creation timestamps and improved timestamps to induce granularity.
- Transparent encryption.
- Allows cleaning of inode tables in background which in turn speeds initialization. The process is called lazy initialization.
- Enables write barriers by default. Which ensures that file system metadata is correctly written and ordered on disk, even when write caches lose power.
There are still some features in the process of developing like metadata checksumming, first-class quota supports, and large allocation blocks.
However, ext4 has some limitations. Ext4 does not guarantee the integrity of your data, if the data is corrupted while already on disk then it has no way of detecting or repairing such corruption. The ext4 file system cannot do secure deletion of file, which is supposed to cause overwriting of files upon deletion. It results in sensitive data ending up in the file-system journal.
XFS performs highly well for large filesystems and high degrees of concurrency. So XFS is stable, yet there’s not a solid borderline that would make you choose it over ext4 since both work about the same. Unless, you want a file system that directly solves a problem of ext4 like having capacity > 50TiB.
Btrfs on the other hand, despite offering features like multiple device management, per-block checksumming, asynchronous replication and inline compression, it does not perform the best in many common use cases as compared to ext4 and XFS. Several of its features can be buggy and result in reduced performance and data loss.
Conclusion
ext4 is used as a default file system for many Linux distros and unless you want to practice your hands on other types of filesystems, ext4 should be your first choice. Other file systems are adopted where they perform better. For example, XFS is the default for Red Hat Enterprise 7 and is still used by agencies like NASA and the U.S Department of Energy, and btrfs is used as a single-disk filesystem in Synology’s storage appliances.