Information Security

System & Software Security

Operating System Security · File System Security · Malware · Software Security · Host Defences · Virtualization

Fall 2026 · Amirkabir University of Technology
@1995parham

One Machine, Many Tenants

Lecture 4 asked allow(s, a, o) about a person. Here the subject is a process, the object is a file, a device or memory — and the kernel answers.

An operating system exists to let mutually distrusting programs share one machine. That is a security job before it is a convenience
Everything rests on a hardware boundary the software cannot talk its way past: user mode and kernel mode
A user-mode program cannot touch another process's memory, a device, or the page tables. It can only ask, through a system call
The system-call boundary is therefore the most attacked interface on the machine

Trusted Computing Base

The TCB is everything that has to be correct for the security policy to hold
Kernel, drivers, firmware, the bootloader, every setuid binary (lecture 4's s bit)
A bug inside the TCB defeats the policy. A bug outside it does not — that is the entire reason for drawing the line
So: keep it small. A driver from an unknown vendor is in your TCB with full kernel rights

Lecture 2 called this "trusted functionality" and told you to keep it small. This is what it looks like in practice.

In the Wild: CrowdStrike Falcon, 2024

19 July 2024: a content update to CrowdStrike's Falcon sensor crashed about 8.5 million Windows machines
The sensor runs as a kernel driver — in the TCB. Its update expected 21 input fields and got 20, and read past the end of the array — section 4's bug
In user mode that is one dead process. In the kernel, it is the whole machine
The vendor's fixes: a bounds check, and staged rollout instead of everyone at once

A security tool is in the TCB too — and so is its bug.

Source: CrowdStrike, Channel File 291 Root Cause Analysis; Microsoft, Helping our customers through the CrowdStrike outage

In the Wild: The xz Backdoor, 2024

A contributor spent two years earning maintainer rights on xz, a compression library, then shipped a backdoor in releases 5.6.0 and 5.6.1 (CVE-2024-3094)
sshd never uses compression — but on several distributions it loads libsystemd, which loads liblzma
Your TCB is not the code you wrote. It is every library the trusted process loads, and everyone who can publish one
Caught while it was mostly in pre-release distributions, because an engineer chased half a second of extra login time

Source: Andres Freund, backdoor in upstream xz/liblzma (oss-security); Russ Cox, Timeline of the xz open source attack

The xz Backdoor, Drawn

A contributor who became an xz maintainer published release tarballs whose build script, not in git, pulled a payload out of test files during distribution package builds; the backdoored liblzma is loaded by libsystemd, which a distribution patch links into sshd, so the code runs inside sshd as root before any login; Andres Freund found it because a failed login took about 0.8 seconds instead of 0.3

The Reference Monitor

The idealised access-control enforcer. Three requirements:
- Complete mediation — lecture 1's principle: every access goes through it, with no side door
- Tamper-proof — nothing it governs can modify it
- Verifiable — small enough that it can be analysed
Real kernels approximate all three and achieve none exactly, which is why the rest of this lecture exists

The Boundary, Drawn

Two user-mode processes ask the kernel through system calls; the reference monitor in kernel mode decides each request before it reaches files, devices or process memory; there is no direct path from a process to another process's memory

When the grade service opens its log, the decision is made here — not in a line of Go.

Hardening

Patch. The overwhelming majority of compromises use a vulnerability that had a fix available
Remove. Uninstall what is not used; disable the services that start by default
Separate. One service per account, none of them root
Default deny on the host firewall, inbound and outbound
Measure. A benchmark you can re-run — the CIS Benchmarks exist for exactly this

Ask it of the grade service: which account runs ./grades, and what else can that account touch? Mostly, the answer is files.

The File System Is the Database

Configuration, credentials, keys, logs and the data itself all live in one namespace governed by one mechanism
Lecture 4 gave the model. Here it is the operational detail:
- Sensitive files should be 0600, owned by the service account
- Directories need x to traverse — a permissive parent undoes a strict child
- Audit setuid binaries. Each one is a place a local user becomes another user
The mistakes that recur: a world-readable key, a backup written with the wrong umask, a config committed to git

Encryption at Rest

Full-disk encryption — FileVault, LUKS, BitLocker. Protects against the stolen laptop and the discarded disk
Be precise about what it does not do: once the machine is running and unlocked, every file is readable by anyone who is on it. It defends a powered-off device
File- or field-level encryption narrows the window further, at the cost of key management in the application
The key has to live somewhere: a TPM or secure enclave, released to the boot process — not a passphrase in a script beside the disk

Knowing When It Changed

File integrity monitoring: hash the files that should never change, and compare on a schedule
Binaries, configuration, cron jobs, startup scripts, the web root
The baseline has to be stored where the attacker cannot reach it, or they will update it along with the file — lecture 1's "Who guards the digest?" again
Tripwire and AIDE do this. So does a package manager verifying its own database, which most systems already have

The Service Can Rewrite Its Own Log

Lecture 1's grade service, first version, running as ./grades 2>> server.log:


$ curl -s -H 'X-User: 4001' -d 'id=4001&grade=20' localhost:8080/grade
4001 is now 20
$ cat server.log
2026/10/14 09:50:41 grade 4001=20 by 4001
$ sed -i '' '/by 4001/d' server.log
$ wc -l < server.log
0

Whoever runs as the service's account — through a bug, or a stolen shell — owns the log too
Lecture 2's "The Day It Happens" reads exactly this line. After the sed, there is nothing to read

Append-Only, and Who Can Lift It


$ chflags uappnd server.log
$ curl -s -H 'X-User: 4001' -d 'id=4001&grade=20' localhost:8080/grade
4001 is now 20
$ sed -i '' '/by 4001/d' server.log
sed: rename(): Operation not permitted
$ cat server.log
2026/10/14 09:50:50 grade 4001=20 by 4001
$ chflags nouappnd server.log

The service can still append. It can no longer rewrite or delete
But uappnd is the owner's flag — the last line removes it, with no error
The wall that holds is a different owner: root sets sappnd (chattr +a on Linux), or the log leaves for another host as it is written

The process that writes the record must not own it.

Every check here trusts the program that runs as you. Next: the program that does not deserve it.

Malware, Sorted

Sort by how it spreads, not by what it does:

Virus	Attaches to a host file. Needs a user to run it
Worm	Self-propagating over the network. Needs nobody
Trojan	Useful-looking, and the user installs it deliberately
Rootkit	Hides the others, usually by lying to the kernel
Bot	Waits for orders from a command-and-control server

Then, separately, the payload: ransomware, spyware, a miner, a wiper, a keylogger. Any spreader can carry any payload.

Why the Worm Is Different

Everything else moves at human speed. A worm moves at network speed
Morris, 1988 — a few thousand hosts, and it took down a noticeable fraction of the internet that existed
Slammer, 2003 — a single UDP packet, no disk footprint, doubling every 8.5 seconds
WannaCry, 2017 — ransomware payload on a worm spreader, using a leaked SMB exploit. A patch had been available for two months
The defence is not detection. It is patching and segmentation — nothing human is fast enough to respond

In the Wild: Stuxnet, 2010

A worm aimed at Siemens industrial controllers (PLCs); about 60% of the infected hosts Symantec counted were in Iran
Its targets were not on the internet. It crossed the gap on USB drives, using a Windows shortcut bug that ran code as soon as the drive was browsed
Four unpatched Windows vulnerabilities, drivers signed with two stolen certificates, and a rootkit on both Windows and the PLC
An air gap is a wall with a door: every USB stick and every contractor's laptop. A signature proves which key signed, not who was holding it

Source: Falliere, O Murchu & Chien, W32.Stuxnet Dossier v1.4 (Symantec, 2011)

Ransomware, Because It Is What You Will Meet

Encrypt the victim's files, sell the key. Purely a business model
It is good cryptography used against you — asymmetric, so the key you need was never on your machine
The modern version is double extortion: the data is exfiltrated first, so paying nothing means publication and paying anyway means trusting a criminal
The attack targets your backups first, because it knows that is the defence
Which makes the lecture 2 rule the technical control that matters here: one copy offline or immutable

In the Wild: NotPetya and Maersk, 2017

Delivered through the update server of M.E.Doc, a Ukrainian accounting package, then spread with EternalBlue and stolen passwords (Mimikatz)
It looked like ransomware and worked as a wiper. The White House put total damage at more than $10 billion
Maersk rebuilt 4,000 servers and 45,000 PCs. Its ~150 domain controllers synced with each other, so all were wiped together
The one survivor was in Ghana, offline because of a power cut. Two staffers relayed its disk to London by plane

A replica is not a backup. The offline copy has to be design, not luck.

Source: Andy Greenberg, The Untold Story of NotPetya (WIRED, 2018)

Detecting It

Signature — match known bytes. Exact, and useless against anything new
- Defeated by polymorphic malware, which re-encrypts itself per copy, and metamorphic, which rewrites its own instructions
Heuristic — suspicious structure. Catches variants, and raises false alarms
Behavioural — watch what it does when it runs. Enumerating and encrypting every document is hard to disguise, because it is the goal
Sandbox — run it somewhere disposable first. Malware checks whether it is in one, so this is an arms race

Detection waits for the malware to arrive. How does it arrive? Mostly through a bug in a program you trusted.

The Oldest Bug

C will happily write past the end of an array. Nothing checks, because checking costs a cycle
The bytes past the end are not empty. They are the next thing in memory
So the attacker's input stops being data and starts being state. Data crossed into control — keep that sentence; lecture 6 meets the same bug on the web


struct session {
  char name[16];
  int  is_admin;      /* the next field in memory */
};

Sixteen Bytes Is Sixteen Bytes


struct session s;
s.is_admin = 0;
strcpy(s.name, argv[1]);            /* no bound check */
printf("name=%-24s is_admin=%d\n", s.name, s.is_admin);


$ ./login parham
name=parham                   is_admin=0

$ ./login AAAAAAAAAAAAAAAAAAAA
name=AAAAAAAAAAAAAAAAAAAA     is_admin=1094795585

1094795585 is 0x41414141 — four of the letter A. The name overflowed straight into the flag.

And With a Little Aim


$ ./login "AAAAAAAAAAAAAAAA$(printf '\x01')"
name=AAAAAAAAAAAAAAAA        is_admin=1

Sixteen bytes of filler and one chosen byte. The program is now an administrator
Nothing was hacked. strcpy did exactly what it is documented to do
On the stack the interesting neighbour is not a flag but the return address — overwrite that and you choose where the function returns to

The Overflow, Drawn

The 20 bytes of struct session for the three runs: parham leaves is_admin at 00 00 00 00; twenty A characters fill is_admin with 41 41 41 41 and the terminating zero lands past the struct; sixteen A characters and 01 make is_admin 01 00 00 00, which is 1

The dots are bytes nobody set. strcpy never asks where name ends.

The Same Source, Compiled Defensively

Two checks the compiler adds: FORTIFY gives strcpy the buffer size; a canary is a random value beside the return address, checked before the function returns.


$ cc -O0 -o login-hardened login.c        # the defaults
$ ./login-hardened AAAAAAAAAAAAAAAAAAAA
exit=133          # SIGTRAP — __strcpy_chk knew the buffer was 16 bytes

$ cc -O0 -fstack-protector-all -o login-canary login.c
$ ./login-canary AAAA…40 bytes
exit=134          # SIGABRT — the canary was overwritten

Not one line of the program changed. The compiler did the work
Both mitigations turn a silent privilege escalation into a crash — which is a denial of service, and an enormous improvement

Layers of Mitigation

A stack frame from name and is_admin up through the canary, the saved frame pointer and the return address; FORTIFY guards name, the canary guards the return, ASLR and CFI guard the return address, NX covers the whole writable stack

Each raises the cost; none is a fix. NX produced return-oriented programming, which builds the payload out of code already in the binary

In the Wild: Heartbleed, 2014

OpenSSL's TLS heartbeat: the client sends a payload and says how long it is; the server echoes that many bytes back
OpenSSL never compared the claimed length with what arrived. Send one byte, claim 65,535, and receive up to 64 KB of server memory — keys, passwords, cookies (CVE-2014-0160)
An over-read, so nothing on the previous slide fires: no canary is touched, nothing crashes, and the logs show nothing unusual
The fix was one bounds check. Found independently by Google's Neel Mehta and by Codenomicon

Source: OpenSSL Security Advisory, 7 April 2014; Codenomicon, The Heartbleed Bug

Heartbleed, Drawn

A heartbeat request claims a length of 65535 but carries one byte, A; the server copies 65535 bytes starting at that A, so the reply holds the A followed by 65,534 bytes of whatever lay next in the heap, such as a private key, a password or a cookie; it is a read, so no canary changes and nothing crashes

The length came with the data. Lecture 6 meets the same trust, one layer up.

The Actual Fix

Use a memory-safe language where you can. Go, Rust, Java, Python — the bug class simply does not exist
Microsoft and Google have both reported that around 70% of their severe vulnerabilities were memory-safety issues. Independently, in different codebases
Where C is required: bounded functions, sanitizers in CI (-fsanitize=address,undefined), fuzzing, and review of every allocation
And the same discipline as everywhere else — validate at the boundary, and never trust a length that came with the data

Bugs will still ship. So the host must notice when one is used — and stop what it does next.

The Host Firewall

The network firewall of lecture 7 protects a zone. This one protects the machine, and it is the one that still works when the attacker is already inside the zone
Inbound: default deny, then allow the ports this host actually serves
Outbound matters more than people expect. Malware has to reach its command-and-control server; a compromised database server has no business opening connections to the internet
pf, nftables, Windows Defender Firewall — the rules differ, the discipline does not

Host-Based Intrusion Detection

A HIDS sees what the network cannot: which process, which user, which file, which syscall — and it sees it after decryption
A NIDS sees the whole segment from one place, and is invisible to the host it is watching
They are complements, not alternatives
Two detection strategies, with opposite failure modes:
- Misuse — match known-bad, the signatures of the Malware section. Precise; blind to the new
- Anomaly — model normal, alert on deviation, like behavioural detection. Catches the new; buries you in false alarms

Why "99% Accurate" Is Not Good News


events/day               1,000,000
  of them, attacks             100
detection rate               99.0%
false-alarm rate              0.1%

true  alarms/day                99
false alarms/day             1,000
total alarms/day             1,099

P(attack | alarm)            9.01%

A detector that misses almost nothing, and nine out of ten alarms are wrong.

The Base Rate, Counted

One million events split into 100 attacks and 999,900 normal events; 99 attacks raise an alarm and about 1,000 normal events do too; drawn to scale, the 99 real alarms are a thin sliver beside the false ones; 99 of 1,099 is 9.01%

The Base-Rate Fallacy

When the thing you are looking for is rare, the false-alarm rate dominates — however good the detector is
This is why analysts stop reading alerts, and why the alert that mattered was in the queue all along
What actually helps:
- Raise the base rate — watch a narrower, higher-value population rather than everything
- Correlate — require two weak signals, so the false-alarm rates multiply
- Drive the false-alarm rate down, even at the cost of detection. 0.01% would change the table above tenfold

In the Wild: Target's Alarms, 2013

Attackers stole data from 40 million payment cards, plus up to 70 million customer records
Target's FireEye system did flag the malware, with urgent alerts as each version was installed
Nobody acted, and the option to delete the malware automatically was not turned on. The alert was named only malware.binary
The Senate report's own guess: staff may have read it as a false alarm, if the system alarmed often — the base-rate fallacy, at work

Detection will miss. So limit the damage: when this host falls, what else falls with it?

Source: US Senate Commerce Committee, A "Kill Chain" Analysis of the 2013 Target Data Breach (2014)

Isolation You Can Buy

A hypervisor runs several operating systems on one machine, each believing it has the hardware
- Type 1 — on the bare metal. ESXi, Hyper-V, KVM. Small TCB, and what clouds run
- Type 2 — on top of a host OS. VirtualBox, VMware Workstation. The whole host OS joins your TCB
For security the appeal is simple: a compromised VM is a compromised VM. The blast radius has a wall around it
And snapshots make analysis cheap — run the malware, look, roll back

The Wall Has Doors

VM escape — a bug in the hypervisor, usually in an emulated device, that gets code onto the host. Rare, and total
Side channels — shared caches and shared branch predictors leak between tenants. Spectre and Meltdown were about exactly this
Sprawl — a VM that is off is not being patched, and it comes back months out of date
The management plane is the real target. Whoever holds the hypervisor console holds every guest, and the console is protected by a password

Containers Are Not Small VMs

A container is a process on one kernel. Namespaces limit what it sees; cgroups limit how much it uses. There is no second operating system
So a kernel bug is shared by every container on the host; VMs still have a hypervisor between them. Weaker isolation, far lower cost
If you run them: non-root user, read-only root filesystem, dropped capabilities (root's rights, in pieces), a seccomp filter on system calls, and never --privileged
Where the boundary must be strong, put one back: a micro-VM (Firecracker, Kata) or a user-space kernel (gVisor)

Isolation, Drawn

Three stacks: Type 2 puts a hypervisor on a full host OS; Type 1 puts the hypervisor on the hardware; containers put namespaces and cgroups on one shared kernel; the wall between two tenants is the hypervisor in the first two and the kernel itself in the third

Ask of each one: what stands between tenant A and tenant B?

Two Distributions, One Kernel


$ uname -sr
Darwin 27.0.0
$ docker run --rm debian:bookworm uname -sr
Linux 7.0.14-linuxkit
$ docker run --rm quay.io/centos/centos:stream9 uname -sr
Linux 7.0.14-linuxkit
$ docker run --rm debian:bookworm sh -c 'ls /dev | wc -l'
15
$ docker run --rm --privileged debian:bookworm sh -c 'ls /dev | wc -l'
169

Debian and CentOS, and one kernel under both — and it is not the Mac's. Docker Desktop runs a Linux VM: a hypervisor wall around the containers, and only a kernel between them
--privileged takes the container from 15 devices to 169: the VM's hardware, handed over in one flag

In the Wild: runc, 2019

runc starts the containers under Docker, containerd, CRI-O and Kubernetes. CVE-2019-5736 let a container overwrite the host's runc binary
The next time runc ran, the attacker's code ran as root on the host — and every other container on it
It needed root inside the container, usually from an untrusted image
So the first item on Containers Are Not Small VMs was the defence: a non-root user, or user namespaces. RHEL's SELinux policy blocked it too

Source: Aleksa Sarai, CVE-2019-5736: runc container breakout (oss-security); Kubernetes, Runc and CVE-2019-5736

Check Yourself

The grade service owns server.log. Why does uappnd not protect it, and what does?
Why did twenty As make is_admin 1094795585, and why is the canary's crash an improvement?
A detector is 99% accurate, but 9 in 10 of its alarms are false. What was rare, and what is one fix?
Two containers, then two VMs on a Type 1 hypervisor: what stands between the tenants in each case?

Where This Leaves Us

The host is defended by the boring things: patch, minimise, separate, log — and a log the service cannot rewrite
The bug classes that survive are the ones where data crossed into control — a buffer into a flag, a file into a privilege
Isolation limits the damage; it never prevents the bug

Next: the same failure, on the most exposed application anyone runs — the one that takes input from every stranger on the internet.