Files, Directories, and the Filesystem Tree

Context

Bytes on disk are grouped into files, files are grouped into directories, and directories nest inside other directories until the whole machine is one tree with a single starting point at the top. That tree is not a metaphor, it is the literal structure the operating system uses to find anything.

Datasets arrive shaped like this. An image dataset is commonly one directory per class with the images inside, and reading that layout tells you the label of every file before you open a single one. Your own projects take the same shape, raw data kept separate from processed data, scripts separate from outputs, configuration tucked into hidden files that do not clutter a normal listing.

If you cannot see the shape of a project you cannot tell a script where its data is, and you cannot tell whether the file you are about to overwrite is the one you spent an hour producing. The previous lesson ended with bytes sitting on disk surviving a reboot, this lesson gives those bytes names, containers, and a position in a structure you can navigate.

Goal

• Distinguish a file from a directory and state exactly what each holds on disk.
• Read any path and trace it from the root directory down to the target file.
• Identify a file extension and prove with file that it is a naming convention, not a guarantee of content.
• Reveal hidden files with ls -a and explain the dot-prefix rule that hides them.
• Construct and navigate a multi-level filesystem tree from your terminal, inside WSL on Windows, using mkdir -p, cd, and tree.
• Explain why a file survives a reboot by connecting it to persistent storage and bytes.

Prerequisites

You should already be able to open your terminal, the WSL one on Windows, and run python3 from earlier in Module 0.

The two below are optional. Skim them if you want the picture first, and skip them freely, the Theory section assumes neither.

1. YouTube - CrashCourse Computer Science #20, "Files & File Systems" (free, openly accessible) - the first seven minutes, for how raw bytes on a disk become named files.
2. LinuxCommand.org by William Shotts - "Learning the Shell", Lesson 2 "Navigation" (free) - the pwd and cd sections only, which bring in ls along the way.

Table of Contents

• Persistent Storage and the Byte
• File
• File Extension
• Directory
• Filesystem Tree
• Root Directory
• Hidden File

Theory

Setup - install this first: This lesson uses the tree command to draw the filesystem as a diagram. tree is not preinstalled on Ubuntu/WSL, so install it once before you start:

sudo apt update && sudo apt install tree  # one-time install of the tree command

sudo apt install tree downloads and installs the tree package, apt update first refreshes the package list so the install succeeds. On macOS, run brew install tree instead, or sudo port install tree on an Intel Mac.

Every block below runs from your home directory, so start there:

cd ~

Persistent Storage and the Byte

What. A byte is the smallest practical unit of stored data, 8 bits, able to hold one of 256 values (for example, the single character A). Persistent storage is hardware, an SSD, a hard drive, a USB stick, that keeps those bytes intact even after the power is cut.

Why. Every file you will ever touch, a train.csv dataset, a saved model, a Python script, is, underneath every abstraction, a long ordered sequence of bytes on persistent storage. Internalizing this kills a whole class of beginner confusion: a file is not "inside" your program, it sits on the disk, and a program only reads its bytes into memory.

When. This distinction matters the instant you save anything. It is the memory versus storage split from Lesson 0.1.2, seen from the storage side.

The mental chain is bits → bytes → files → directories → filesystem, each layer organizing the one below. A byte alone is meaningless, the filesystem is what declares "these 2048 bytes, starting here, are one file named train.csv." You can ask the filesystem for that byte count directly from the terminal.

echo "feature,label" > train.csv  # create a file and write one line into it
stat -c%s train.csv               # print the file size as a raw byte count (macOS: 'stat -f%z train.csv')

The > redirects output into a file instead of the screen. The number stat prints is the filesystem's own count, concrete proof that a file is a measured run of bytes on persistent storage.

wc -c train.csv  # count bytes a different way: -c means 'bytes'

wc -c should report the same number as stat -c%s, a count that includes the invisible newline character at the end of the line, which is itself one byte.

DON'T:

• DON'T assume one byte always equals one visible character. That holds only for plain ASCII text, image, audio, and model-weight bytes are not characters at all.

File

What. A file is a named container for an ordered sequence of bytes, stored on persistent storage and tracked by the filesystem. The name lets you find it, the bytes are the actual content.

Why. Files are the unit of permanence in computing. You do not pass raw byte addresses around, you pass names. Every dataset, script, and saved model is a file. When you later run a training script that reads train.csv, you are handing the operating system a name and asking it to stream that file's bytes into the program. If you cannot reason about files, you cannot reason about data pipelines.

When. You create a file whenever you save work, code, data, results, logs. You read a file whenever a program needs input that outlives a single run. In ML this is the whole lifecycle: raw data files in, a model file out.

A file has two parts worth separating in your head: its name (metadata the filesystem stores, like train.csv) and its contents (the bytes). Renaming changes the name without touching a single byte, editing changes the bytes without touching the name. They are independent.

touch experiment_notes.txt  # create an empty file (zero bytes)
# fill it with one line
echo "epoch=10, lr=0.001" > experiment_notes.txt
cat experiment_notes.txt    # print the file's bytes to the screen

touch creates the file only if it does not exist yet. After the terminal closes, the file is still on disk, that is the entire point of a file versus a session variable.

mv experiment_notes.txt run1_notes.txt  # rename: same bytes, new name
cat run1_notes.txt                      # identical content as before

DON'T:

• DON'T treat a file's name as its content. model_final.pkl might be empty, the name is a label you chose, nothing more.
• DON'T assume opening a file in a program moves it. Reading copies the bytes into memory, the original stays on disk untouched.

File Extension

What. A file extension is the suffix after the last dot in a filename, .csv, .py, .png, .pkl. It is a convention that signals the file's intended format to humans and programs.

Why. The extension is how tools and people decide how to interpret a file's bytes. Your shell uses it to glob (ls *.csv lists only the names ending in .csv), and you use it to know at a glance what a file holds. In ML projects, extensions keep a folder full of .csv, .json, .pkl, and .ipynb files self-documenting.

When. You assign an extension every time you name a file you create, and you read extensions constantly while scanning a project. Choosing the right one is a courtesy to every program and person that opens the file later.

The critical truth: the extension is a label, not a law. Nothing stops you naming a JSON file data.csv, the bytes inside do not change. The Linux file command proves this, it inspects the actual bytes and reports the real type, ignoring the name entirely.

echo '{"lr": 0.01}' > config.csv  # JSON content, but a .csv name on purpose
file config.csv                   # inspects bytes, reports the REAL type (JSON/text)

This is the whole lesson of extensions: the suffix describes intent, the bytes are the reality, and a tool that trusts the suffix blindly will mishandle the file.

basename train.csv .csv  # strip the .csv extension -> train
ls *.csv                 # glob: list only files ending in .csv

A filename without its extension is its stem. The * is the shell's wildcard, so ls *.csv is the everyday way to grab "all my dataset files," driven purely off the extension string.

DON'T:

• DON'T trust an extension to guarantee content. Renaming notes.txt to notes.csv does not make it valid CSV, run file if you are unsure.
• DON'T forget files can have no extension at all (like README or Dockerfile). Globs like *.csv simply skip them, handle that case deliberately.

Directory

What. A directory (commonly a folder) is a special file whose contents are a list of other files and directories. It groups related items under one name and a shared location.

Why. Without directories, every file would sit in one giant flat pile, unmanageable past a few dozen items. Directories impose structure: data/ for datasets, models/ for checkpoints, notebooks/ for experiments. This is how real projects stay navigable. A training script expects data in a specific directory, getting directory layout right is half of getting a pipeline to run.

When. You make directories whenever you organize a project, and you traverse them constantly to locate files. Any ML project of nontrivial size has a deliberate layout from day one.

A directory does not "contain" files the way a box contains objects. It contains entries, references pointing to where each file's bytes live. That is why a directory can be tiny even when the files it lists are gigabytes: the directory is an index, not a container of the actual bytes.

mkdir -p ml_project/data  # -p creates ml_project AND data in one call
ls ml_project             # list what the ml_project directory points to -> data
cd ml_project             # step INTO the directory, it becomes your location
pwd                       # print working directory: where you now stand

mkdir, ls, cd and pwd are the four commands you will use thousands of times.

DON'T:

• DON'T picture a directory as physically holding files. It holds entries (references), the file bytes live elsewhere on disk.
• DON'T forget mkdir without -p fails if a parent is missing. mkdir a/b/c errors unless a/b already exists, mkdir -p a/b/c builds the whole chain.

Filesystem Tree

What. The filesystem tree is the hierarchy formed by directories containing files and other directories, all branching from a single starting point. It is a tree in the computer-science sense: one root, branches (directories), and leaves (files).

Why. The tree is the model for where everything lives. Every file has exactly one position in it, reached by a unique sequence of directories, its path. Reasoning about paths, "where is my data relative to my script," and the No such file or directory errors you will hit all reduce to reasoning about position in this tree.

When. Every time you open, save, or move a file, you navigate the tree. When a script says data/train.csv, it describes a walk down the tree from wherever you currently stand.

A path is the address of a node. There are two flavors. An absolute path starts from the root and spells out every level (/home/goku/ml_project/data/train.csv). A relative path starts from your current directory and describes the route from there (data/train.csv). Two special names move you around: . means "the current directory" and .. means "the parent, one level up." The tree command you installed draws the whole structure at once.

cd ~                          # the block above left you INSIDE ml_project, so come back up first
mkdir -p ml_project/data/raw  # build a three-level branch
# create the leaf file
echo "feature,label" > ml_project/data/raw/dataset.csv
tree ml_project               # draw the directory as a tree diagram

tree prints ml_project with data branching into raw branching into dataset.csv, a leaf, a file with nothing below it. Seeing the branches drawn makes "the tree" literal instead of abstract.

cd ml_project/data/raw  # descend three levels into the tree
pwd                     # absolute path of where you now stand
cd ..                   # go UP one level to data/
cd ../..                # go up two more levels (note: now above ml_project)

pwd always reports your absolute position, so you can confirm exactly where in the tree a relative cd landed you.

DON'T:

• DON'T assume a relative path works from anywhere. data/train.csv only resolves if your current directory is the one containing data/. Run the same command elsewhere and it breaks with No such file or directory.
• DON'T confuse . (current directory) with .. (parent). Swapping them sends you the wrong way through the tree.

Root Directory

What. The root directory is the single topmost directory of the tree, the one node with no parent. Every file and directory descends from it. In WSL (Ubuntu/Linux) it is written /.

Why. The root is the anchor that makes absolute paths unambiguous. Because there is exactly one root, an absolute path starting from it identifies one and only one location, no matter where you currently stand. This is why robust scripts often use absolute paths, they are immune to "where am I running from" mistakes.

When. You rely on the root every time you read or write an absolute path, and you reach for absolute paths when a relative path is too fragile. You will also see it whenever a tool reports a full location.

In WSL your whole Linux filesystem hangs under one root /, /home for users, /etc for config, /usr for programs. Your Windows drives are not separate roots here, WSL mounts them under the same tree (your C: drive appears at /mnt/c), so even they live below /. The realpath command converts any relative path into the absolute path from the root.

ls /  # list the contents of the root directory
cd /  # move to the very top of the tree
pwd   # prints just /

The root has no name above it, which is why pwd prints the single character /.

cd ~                                      # the block above left you at '/', where ml_project does not exist
realpath ml_project/data/raw/dataset.csv  # relative path -> full absolute path
# e.g. /home/goku/ml_project/data/raw/dataset.csv

DON'T:

• DON'T confuse the root / with your home directory (/home/goku, shortcut ~). Home is just one branch far below the root, not the top of the tree.
• DON'T expect a bare Windows path like C:\Users\... to work inside WSL. Use the mounted Linux path /mnt/c/Users/..., which lives under the single root /.

Hidden File

What. A hidden file is a file or directory that the default ls listing and file browsers omit. On Linux/WSL, any name beginning with a dot (.gitignore, .env, .config/) is hidden by that single convention.

Why. Hidden files keep configuration and tooling out of your way so a folder shows only the content you actively work with. ML projects are full of them: .gitignore (files version control should skip), .env (settings and keys), .ipynb_checkpoints/ (auto-saved notebook backups). They are not secret or protected, just visually suppressed so a data/ folder is not cluttered with dotfiles.

When. You meet hidden files the moment you set up a project with Git or environment variables, reveal them when you need to edit configuration, and create them whenever you add tooling that expects a dot-prefixed config.

The dot-prefix rule is purely a naming convention: there is no special "hidden" bit in the bytes, the leading dot alone makes ls skip the entry. To reveal hidden files you pass the -a ("all") flag.

echo "data/" > .gitignore  # create a hidden file (leading dot)
ls                         # default listing: .gitignore is NOT shown
ls -a                      # -a reveals hidden entries, plus . and ..

.gitignore is still an ordinary text file on disk. The . and .. that ls -a adds are this directory and its parent, themselves dot-prefixed entries.

ls -a | grep '^\.'  # list ONLY the entries whose names start with a dot

The | (a pipe) sends the full listing to grep as its input, and grep keeps only the lines matching '^\.', a literal dot at the start of a line.

DON'T:

• DON'T think "hidden" means "secure" or "encrypted." A hidden file opens with cat for anyone allowed to read it, exactly as it would without the dot, which only stops it from being listed by default. Never store secrets in a hidden file and assume they are protected.
• DON'T create a hidden file by accident with a leading dot. touch .results is hidden, touch results is not, and you may waste ages wondering where your output "disappeared."

Lab Exercises

Lab Exercise 1: Inspect a File's Identity

From your terminal, create a file named train.csv containing the line feature,label. Then print four facts about it: its full name, its name with the .csv extension stripped off, its real type as detected by inspecting the bytes, and its size in bytes. This exercises the first half of the lesson: files, bytes, and extensions.

Lab Exercise 2: Build a Three-Level Tree

Build this tree from a folder called ml_project, creating all levels in one command:

ml_project/
└── data/
    └── raw/
        └── dataset.csv

Write any line of text into dataset.csv, draw the whole structure as a diagram, and print the absolute path of dataset.csv. This exercises the second half: directories, the filesystem tree, and the root.

echo "feature,label" > train.csv  # create the file with one line of content

basename train.csv                # full filename -> train.csv
basename train.csv .csv           # name without .csv -> train
file train.csv                    # real type from inspecting the bytes
stat -c%s train.csv               # byte count the filesystem reports (macOS: 'stat -f%z train.csv')

mkdir -p ml_project/data/raw              # build all three levels at once
# create the leaf file
echo "a,b" > ml_project/data/raw/dataset.csv

tree ml_project                           # draw the tree diagram
realpath ml_project/data/raw/dataset.csv  # absolute path from the root

Theory Assessment

Conceptual reinforcement, not a coding section. All four questions with their hints first, then all four answers.

Q1: What survives, and what a byte actually counts

Two parts. A classmate says "I loaded my dataset into a variable in python3, so I can shut my laptop down and it will still be there tomorrow." Explain precisely why that is wrong, using the terms persistent storage and RAM. Then: you run echo "cat" > pet.txt and stat -c%s pet.txt (stat -f%z pet.txt on macOS) reports 4 bytes even though you typed three letters. Give the most likely reason.

Hint: For the first, ask where a session variable lives against where a saved file lives, and which of the two survives the process ending. For the second, count every byte echo actually writes rather than only the letters.

Q2: The extension is a label

You rename config.json to config.csv with mv, without changing its content, and a tool that reads CSV then crashes on it. Explain why the rename did not change the file's actual content, and name the command that would have revealed the real type.

Hint: The extension and the bytes are independent. Ask which one mv touched, and which command inspects the bytes themselves.

Q3: The index, and walking the tree

Explain how a directory can occupy only a few kilobytes while the files it lists total several gigabytes. Then, given the absolute path /home/goku/ml_project/data/train.csv, identify the root directory, the leaf, and which directory you land in after running cd .. twice starting from the folder that contains train.csv.

Hint: A directory is a special file whose content is a list of references. Read the path left to right as a walk down the tree, where each / separates one level.

Q4: Hidden is not protected, and there is only one root

Linux decides who may open a file by its permissions, read, write and execute flags it keeps on every file and directory for the owner, for one group of users and for everyone else. A teammate stores an API key in .secrets and says "it's hidden, so it's safe". Explain precisely why hidden gives no security, name what actually makes the file hidden, and say which decides whether another user can read the key, that or the file's permissions. Then the second half, which trips people for the same reason: in WSL, why does your Windows C: drive appear at /mnt/c rather than as its own separate root, and why does that mean there is exactly one root / in your Linux tree?

Hint: Separate "not shown by a default ls" from "cannot be read with cat", and ask which of the two a leading dot controls. For the second half, a tree has exactly one topmost node, so ask where C: sits relative to / if it shows up underneath /mnt.

Assignment

Assignment: Extension Census with Sizes

Write a script that walks an entire project tree and produces a census: for every extension present (including a label for files with no extension, like README), report how many files have it and their combined size in bytes, sorted from largest combined size to smallest. Use awk to total the sizes per extension and sort to order the report. Save it as census.sh and run it with bash census.sh (bash, a shell, then reads its commands from the file instead of from your keyboard).

Hints:

• Drive the loop with find <dir> -type f, and for each file derive its extension ("${f##*.}") and size (stat -c%s, or stat -f%z on macOS).
• Pipe extension size lines into awk to accumulate count and total bytes per extension, then sort -k3 -nr to order by combined size descending.

#!/bin/bash
dir="${1:-census_demo}"           # tree to census, defaults to the demo built just below

# --- test setup ---
mkdir -p census_demo/data
# a 2000-byte file
head -c 2000 /dev/zero > census_demo/data/train.csv
# a 500-byte file
head -c 500  /dev/zero > census_demo/data/test.csv
# a 4096-byte file
head -c 4096 /dev/zero > census_demo/model.pkl
echo "info" > census_demo/README  # no extension

# emit "extension size" per file, then group/sum with awk, then sort by size
find "$dir" -type f | while read -r f; do
  base="$(basename "$f")"         # we store the file names in base, one by one
  # text after the last dot
  if [[ "$base" == *.* ]]; then ext="${base##*.}"
  else ext="(none)"; fi           # label for no-extension files
  echo "$ext $(stat -c%s "$f")"   # e.g. csv 2000 (macOS: 'stat -f%z')
done | awk '{ count[$1]++; bytes[$1]+=$2 } # awk groups and sums per extension
            END { for (e in count) print e, count[e], bytes[e] }' \
   | sort -k3 -nr                 # sort by 3rd column, numeric, descending

Additional Resources

Optional, and nothing later depends on either. Worth a look only if this lesson left you curious.

1. YouTube - "Linux Command Line Tutorial For Beginners 17 - file permissions, symbolic permissions and chmod" (ProgrammingKnowledge, free). Permissions are the real access-control layer that "hidden" is not.
2. Filesystem Hierarchy Standard 3.0 - section 3.2 "Requirements" (free, Linux Foundation) - what each directory every Linux system must have under / is for.

Summary

Underneath every dataset you will load and every model you will save sits the same quiet machinery: ordered bytes resting on persistent storage, given names so you can find them again after the power goes off. A file is nothing more exotic than that, a named run of bytes, and once you stop picturing it as something living inside your program, a whole class of beginner confusion simply dissolves. The variable in your python3 session and the train.csv on your disk are different kinds of things, and knowing which one survives a reboot is the difference between confidence and panic.

Directories give those files an order. They are not boxes holding content but indexes holding references, and from a single root they branch outward into the tree that every path describes. To name a file's location is to describe a walk down that tree, from the one root / that has no parent, through directory after directory, to a leaf that branches no further. Absolute paths spell out the whole walk and work from anywhere, relative paths assume you already know where you stand, and break the moment you stand somewhere else. That single distinction explains a startling fraction of the No such file or directory errors you will meet and fix in the months ahead.

The smaller conventions, an extension that suggests a format without enforcing it, a leading dot that hides a file from ls without protecting it in the slightest, are reminders that much of a filesystem is agreement, not law. The bytes are the only ground truth, everything layered above is a human convention to keep those bytes navigable. Hold onto that as you move forward. Every time you cd into a project and ls its contents, you will be quietly trusting every idea in this lesson, and when a path fails, you will already know exactly where in the tree to look.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

mkdir -p ml_project/data               # build nested folders in one call
cd ml_project                          # step into a directory
echo "feature,label" > data/train.csv  # create file, write bytes

pwd                                    # where am I (absolute current dir)
ls            ;  ls -a                 # visible entries, then hidden + . ..
tree                                   # draw the whole tree as a diagram

stat -c%s data/train.csv               # size in bytes (macOS: 'stat -f%z')
file       data/train.csv              # real type from the bytes (ignores name)
basename   data/train.csv .csv         # train  -> strip extension
realpath   data/train.csv              # absolute path from the root /

cd ..                                  # up one level, back above ml_project
find . -type f                         # every file at any depth
cd /                                   # jump to the root, run 'find' from a project dir, NEVER from here