Returning to the world of blogging after nearly a month’s hiatus, I find myself intrigued by the concept of Overlay Filesystems. This article has piqued my curiosity, and I am convinced it will offer valuable insights for my future endeavors. Reflecting on my past experiences, I recall writing about tools such as pvmove, vgextend, and lvreduce, as well as methods for recovering deleted data from LVM partitions. These insights have proven invaluable over the years. Now, it’s time to delve into the realm of Overlay Filesystems and share my findings through a new write-up.
As the name itself says everything: layer on top of other layers… So there are these three layers in the Overlay Filesystem: Lower, Upper, and the top layer called Overlay. Let’s say you have 2 files in the lower layer: lowerlayer1.txt and lowerlayer2.txt and another set of 2 files in the upper layer: upperlayer1.txt and upperlayer2.txt. So what happened in the Overlay layer is a combination of all the 4 files that will be seen there: lowerlayer1.txt, lowerlayer2.txt, upperlayer1.txt, and upperlayer2.txt. Pretty straightforward forward right? So what happens if there is a duplicate file in the lower and the upper layers, say duplicatefile.txt presence in both the Upper and Lower layers? What is going to happen is in the Overlay layer, only the duplicatefile.txt from the upper layer will be seen in the overlay layer.
Let’s dive deeper into the layers to get the whole picture: The Lower and upper layers may be of different file systems, say EXT4 and XFS. For the purpose of illustration and proof of concept for this blog post, I am running a virtual machine on UTM on Rocky Linux 8.
1. I have created 3 directories under my /home and to make things good-looking, I’m using the tree command for a better visual.
2. Created those files as well:
3. The purpose of the Overlay is to make a union of the upperlayer and the lowerlayer using the union file system. note that there is a duplicate file in both the upperlayer and lowerlayer directories. I have also created a fourth directory called temporary.
4. Before proceeding, it’s good to know you have the overlay module loaded in the kernel.
5. Then mount the file system using the following mount command:
mount -t overlay -o lowerdir=/home/lowerlayer/,upperdir=/home/upperlayer/,workdir=/home/temporary/ none /home/Overlay/
6. It can be verified using the following command:
7. Once mounted, from a tree perspective, it can be seen that everything from the directory upperlayer and lowerlayer has been unioned into the Overlay directory:
8. Time to write some stuff in both duplicatefile.txt
9. As you can see the Overlay recognized only the upperlayer directory:
The lower layer is the read-only layer. Modifications in the lowerlayer do not affect directly the upperlayer.
10. If we add something here in that lower layer, say:
11. We should notice that this gets replicated in the upperlayer as well but not in the lowerlayer
12. Now this is very IMPORTANT to understand. As soon as something is being written to the Overlay layer, it gets replicated to the directory upperlayer. Look at part 7. Upperlayer never had the file lowerlayer1.txt
So, we can learn that when a file is created in the lowerlayer, it gets copied into the Overlay directory
And, when a file is created in the Overlay directory, it gets copied into the upperlayer directory but not in the lowerlayer directory.
FILE MODIFY in LOWERLAYER -> “COPY on Write” to UPPERLAYER (LOWERLAYER is read-only layer)
FILE MODIFY IN UPPERLAYER -> to OVERLAY directory
13. If we overwrite the duplicate file in the lowerlayer.txt, you will notice that neither the upperlayer nor the overlay layer will know about it
14. If I unmount the /home/Overlay and re-mount it, then the union re-occurs and the Overlay layer will re-make a union of both layers. So the lowerlayer is a read-only layer and it does not directly affect other layers.
15. However if we modify the same file in the Overlay layer, the change will appear in the Upperlayer. Here is a table I made to explain the consequence of what happens in any condition when a file is created and overwritten:
| LAYERS / Action | LOWERLAYER | UPPERLAYER | OVERLAY | Remarks |
|---|---|---|---|---|
| add newfile1 (lower layer) | added | nothing happened | added | file and content get added |
| Modify newfile1 (lower layer) | modified | nothing happened | nothing happened | nothing happened fs need to be remounted |
| add newfile2 (upper layer) | nothing happened | added | added | file and content get added |
| Modify newfile2 (upper layer) | nothing happened | modified | nothing happened | nothing happened fs need to be remounted |
| add newfile3 (overlay layer) | nothing hapenned | added | added | file and content get added |
| modify newfile3 (overlay layer) | nothing happened | modified | modified | file and content modified in both |
| modify newfile3 (upper layer) | nothing happened | modified | nothing happened | nothing happened fs need to be remounted |
| modify newfile1 (overlay layer) | nothing happened | nothing happened | modified | nothing happened.. changed data in overlay is preserved even after remount |
| modify newfile2 | nothing happened | overwritten | modified | any change to newfile2 in overlay gets replicated to upperlayer |
16. Another question is what happens if we start deleting? If both in lowerlayer and upperlayer, we know that the overlay will pick up only the overlay layer one on re-mount. However, if we delete the file from the Overlay, this will impact the upper layer only by showing a character file that indicates a deleted file. Here is an example of files that were deleted in overlay but used to be present in the upperlayer marked with “c”.
In Linux, “c” indicates that the device is a “character special device”. Character-driven will send one character at a time, thus you need a small load to carry but have to make many requests.
