Modular Computers the manual GitHub

Hard Disk

A hard disk keeps files. It comes in one of two formats: hfs, where the disk serves files itself, or raw, where it is bytes you read and write by offset.

Facts

| | | |—|—| | Type | storage | | In /dev | storage0, storage1… in bus order | | Address | A SATA slot path such as sata:0 | | Tiers | 1 (1 MiB), 2 (4 MiB), 3 (16 MiB) | | Interrupt | Raised while a change report waits (changes) | | Power | Tier 1 8 W, tier 2 5 W, tier 3 2 W | | Info | tier, capacity, mode |

For the item and its sockets, see Hard disk. To format, wipe or clone one, see Disk Station.

Formats

A disk in one format refuses the other format’s methods. A file system’s methods fail on a raw disk (raw disk: no filesystem), and read and write by offset fail on an hfs disk (hfs disk: format it raw to read and write it by offset).

Space counts each file’s bytes plus 512 for each file and directory, so empty files are not free. A full disk fails a write with No space left on device.

In mcos

mcos boots from an hfs disk and mounts it at /. Every other hfs disk is mounted as it joins the bus, by where it is: a disk in SATA slot 1 at /mnt/sata/1, a disk known only by its id at /mnt/id/<id>. It is unmounted when it leaves (files open on it then fail with EIO). Programs use the files there as any others (see Filesystem). A raw disk is not mounted.

/dev/storage0, /dev/storage1…

Each disk’s file in /dev is the disk’s bytes, as a Linux block device’s are: from offset 0 to its capacity, read, written and sought anywhere.

For sys.poll, a disk’s file is always ready to read and to write, and has PRI while the disk has news (its interrupt came since the file was last read or asked).

lsdev                   # every device, with its /dev name
label storage1 backups  # a label on the disk's item
df                      # each mounted disk's size and use

Its ioctls are its methods

The file’s ioctls are the disk’s methods below, by name, with their arguments, in any open mode, on either format, mounted or not. A method that answers nothing gives true; one that fails gives nil, its message and EINVAL; a disk that is gone gives ENODEV.

local disk = io.open("/dev/storage1")
local free = sys.ioctl(sys.fileno(disk), "getCapacity") - sys.ioctl(sys.fileno(disk), "getUsed")

Paths given to list, stat and the others are the disk’s own, from its root, whatever it is mounted as. setLabel (the kernel’s, on every device file) puts a label on the disk’s item.

Some methods are for code without an OS, such as an EEPROM’s, and are not for programs under mcos: open gives a handle that only the firmware can use (under mcos, open files through the filesystem), and changes is taken by the kernel, which turns a mounted disk’s reports into inotify notices. Through an ioctl, stat of a path with nothing there answers true (the method answered nothing), and so does read at the end of a raw disk.

Methods, both formats

getCapacity() → bytes

How many bytes the disk holds.

getUsed() → bytes

Bytes in use. On hfs: files and their entries, and what open files will write at close. On raw: the whole capacity.

getMode() → “hfs” or “raw”

The disk’s format.

setMode(mode) → nothing

Formats the disk as "hfs" (an empty filesystem) or "raw". Everything on it is lost and open files close. Any other mode fails with unknown mode; a read-only disk fails with read-only disk.

changes() → event or nil

The oldest change on the disk that has not been taken yet. See Events.

Methods, hfs

Every one fails on a raw disk. Writes fail with read-only disk on a read-only disk (a Disk Station’s source disk).

open(path[, mode]) → handle

Opens a file. mode is r (the default), r+, w, w+, a or a+, as C’s fopen (a b is ignored). A new file is made at once, empty. At most 16 files are open at once on a disk (too many open files). Fails with no such file: path, is a directory: path or bad mode.

The handle has its own methods, below.

list(path[, typed]) → names

The names in a directory, sorted; directories end in /. With typed true, a table of {name, type} each, where type is file, directory or link. path defaults to /. Fails with no such file when it is not a directory.

stat(path[, nofollow]) → {size, directory, modified} or nil

modified is the world’s time when the file last changed, in milliseconds (a game tick is 50). Nil if there is nothing there. With nofollow, a symbolic link is described itself, with link set to its target (and size its length).

mkdir(path) → nothing

Makes a directory. Its parent must exist. Fails with already exists: path.

remove(path) → nothing

Removes a file, a link (not what it points to), or an empty directory (directory not empty). The root cannot be removed.

rename(from, to) → nothing

Moves a file or directory. to must not exist (already exists), and a directory cannot move into itself.

symlink(target, path) → nothing

Makes a symbolic link at path. The target is any text (the operating system follows it, the disk never does), at most 1023 bytes (name too long). path must not exist. A link is kept as a CIFS XSym file of 1067 bytes.

readlink(path) → target

A link’s target. Fails with not a link.

A path that meets a link fails with the message symbolic link, a NUL, the link’s path, a NUL and its target: the operating system goes on from there.

Methods, raw

Every one fails on an hfs disk.

read(offset, n) → string or nil

Up to n bytes (at most 4096) from the byte offset, fewer at the end. Nil at the end. An offset past the capacity fails with no such offset, a negative n with a negative count.

write(offset, data) → count

Writes the bytes at the offset, as many as fit before the end, and gives how many it wrote. Fails with read-only disk on a read-only disk.

File handles

A handle never holds a host file open. It reads through a 4096 byte window, and buffers its writes. close commits them all at once: the file is all old or all new. Power lost, or the disk taken out, before close loses the writes. A handle left unclosed and forgotten is abandoned the same way. Writes reserve space as they go and fail with No space left on device when there is none.

handle.read(n) → string or nil

Up to n bytes (at most 4096), nil at the end. Fails with not open for reading. A reader at the end looks again for data another handle has added since.

handle.write(data) → nothing

Writes where the file is at (at the end, if opened to append). Takes effect at close. Fails with not open for writing, or Cannot allocate memory when the program’s memory cannot hold it.

handle.seek([whence[, offset]]) → position

Moves to offset from "set" (the start), "cur" (the default) or "end", and gives where it is. Fails with bad whence or can't seek before the start.

handle.close() → nothing

Commits what was written and closes. On failure the handle stays open with its writes kept.

Events

A disk reports every change made on it, whoever made it (this computer, another one, a Disk Station), as an event. The oldest is taken with changes(). At most 32 wait; when more come, the oldest is dropped and the next event to arrive carries lost, the number dropped. Reports are kept across a save, and dropped when no program runs.

Event Fields
modify, create, remove path
rename path (the new one), from
format path (/), mode
write (raw) offset, length

The disk’s interrupt line is up while any wait. Under mcos, the kernel takes a mounted disk’s reports and gives them to programs as inotify notices, in the paths it is mounted at (MODIFY /mnt/sata/1/log, a rename as MOVED_FROM and MOVED_TO, a format as OVERFLOW).

Examples

Back up a directory onto another disk

Copies /home onto the disk in SATA slot 1, keeping the last three copies, after asking the disk through its ioctls whether the copy fits.

-- Back up a directory onto another disk, keeping the last few copies, as a nightly backup job does.
--
-- usage: lua disk-backup.lua [directory [disk]]
--   directory  what to back up (default /home)
--   disk       the backup disk's name in /dev (default storage1): an hfs disk, which mcos mounts
--              by itself when it joins the bus (a disk in SATA slot 1 at /mnt/sata/1)
--
-- Each run copies the directory to backup/1, backup/2... on the disk, and then removes the oldest
-- copies beyond KEEP. Before it copies anything, it asks the disk itself, through ioctls on its /dev
-- file (getCapacity, getUsed), whether the copy fits, so a backup that can't fit is refused whole.
local sys = require("sys")
local fs = require("fs")

local KEEP = 3 -- copies kept on the disk
local ENTRY = 512 -- what a disk counts for each file and directory, on top of its bytes
local FOLDER = "backup" -- where the copies go, on the disk

local from, diskname = ...
from = from or "/home"
diskname = diskname or "storage1"

local function die(...)
  io.stderr:write("disk-backup: ", ...)
  io.stderr:write("\n")
  os.exit(1)
end

-- The disk, as the kernel knows it, and where it's mounted.
local device
for _, d in ipairs(sys.devices()) do
  if d.name == diskname then
    device = d
  end
end
if not device or device.type ~= "storage" then
  die(diskname, ": not a disk")
end
local mountpoint
for _, mount in ipairs(sys.mounts()) do
  if mount.source == device.path or mount.source == device.id then
    mountpoint = mount.path
  end
end
if not mountpoint then
  die(diskname, ": not mounted (a raw disk has no files: format it hfs)")
end

-- What a tree takes on a disk: its files' bytes, and an entry for each file and directory.
local function size(path)
  local stat = assert(sys.stat(path))
  if stat.type ~= "directory" then
    return ENTRY + stat.size
  end
  local total = ENTRY
  for _, entry in ipairs(assert(fs.list(path))) do
    total = total + size(fs.join(path, entry.name))
  end
  return total
end

-- Ask the disk how much room it has.
local disk = assert(io.open("/dev/" .. diskname))
local fd = sys.fileno(disk)
local capacity = assert(sys.ioctl(fd, "getCapacity"))
local used = assert(sys.ioctl(fd, "getUsed"))
disk:close()

local needed = size(from)
if needed > capacity - used then
  die(from, " needs ", needed, " bytes; ", diskname, " has ", capacity - used, " free")
end

-- The copies there now, oldest first.
local folder = fs.join(mountpoint, FOLDER)
assert(fs.mkdir(folder, true))
local copies = {}
for _, entry in ipairs(assert(fs.list(folder))) do
  local n = tonumber(entry.name)
  if n and entry.type == "directory" then
    copies[#copies + 1] = n
  end
end
table.sort(copies)

local number = (copies[#copies] or 0) + 1
local to = fs.join(folder, tostring(number))
assert(fs.copy(from, to, true))
copies[#copies + 1] = number
print("backed up " .. from .. " to " .. to .. " (" .. needed .. " bytes)")

while #copies > KEEP do
  local oldest = fs.join(folder, tostring(table.remove(copies, 1)))
  assert(fs.remove(oldest, true))
  print("removed " .. oldest)
end

Download disk-backup.lua

Keep a log on a raw disk

A raw disk has no files: this program keeps its own format on the disk’s bytes, through its /dev file, read and sought by offset.

-- A log kept on a raw disk, with no filesystem: lines are added at the end, and read back in order.
--
-- usage: lua disk-raw-log.lua add [disk]     adds each line of standard input
--        lua disk-raw-log.lua show [disk]    writes every line kept, oldest first
--        lua disk-raw-log.lua clear [disk]   empties it
-- disk is the raw disk's name in /dev (default storage1).
--
-- A raw disk's /dev file is its bytes, from offset 0 to its capacity, as a Linux block device's:
-- read, written and sought anywhere. This log keeps a 16-byte header at offset 0 saying how many
-- bytes of lines follow it ("rawlog " and eight digits), and the lines right after. A disk never
-- written is all zeros, which reads as an empty log.
local sys = require("sys")

local HEADER = 16 -- bytes of the header, at offset 0
local MAGIC = "rawlog " -- how a header starts
local CHUNK = 4096 -- the most a disk's read gives at once

local command, diskname = ...
diskname = diskname or "storage1"

local function die(...)
  io.stderr:write("disk-raw-log: ", ...)
  io.stderr:write("\n")
  os.exit(1)
end

local fd, why = sys.open("/dev/" .. diskname, "r+")
if not fd then
  die(diskname, ": ", why)
end
local mode = sys.ioctl(fd, "getMode")
if mode ~= "raw" then
  die(diskname, " is ", tostring(mode), ": wipe it raw first (a Disk Station's wipe)")
end
local capacity = assert(sys.ioctl(fd, "getCapacity"))

-- Bytes from an offset, as many as asked (a read gives at most CHUNK).
local function readat(offset, n)
  assert(sys.lseek(fd, offset, "set"))
  local parts = {}
  while n > 0 do
    local data = assert(sys.read(fd, math.min(n, CHUNK)))
    if data == "" then
      break -- the disk's end
    end
    parts[#parts + 1] = data
    n = n - #data
  end
  return table.concat(parts)
end

local function writeat(offset, data)
  assert(sys.lseek(fd, offset, "set"))
  assert(sys.write(fd, data))
end

-- How many bytes of lines there are (0 for a disk that holds no log yet).
local function length()
  local header = readat(0, HEADER)
  return tonumber(header:match("^" .. MAGIC .. "(%d+)\n")) or 0
end

local function setlength(n)
  writeat(0, string.format("%s%08d\n", MAGIC, n))
end

if command == "add" then
  local at = length()
  for line in io.lines() do
    local record = line .. "\n"
    if HEADER + at + #record > capacity then
      die(diskname, " is full")
    end
    writeat(HEADER + at, record)
    at = at + #record
    setlength(at) -- (after each line: a log cut short still holds every line it says it has)
  end
elseif command == "show" then
  local left, offset = length(), HEADER
  while left > 0 do
    local data = readat(offset, math.min(left, CHUNK))
    io.write(data)
    offset, left = offset + #data, left - #data
  end
elseif command == "clear" then
  setlength(0)
else
  die("usage: disk-raw-log.lua add|show|clear [disk]")
end
sys.close(fd)

Download disk-raw-log.lua

dmesg | lua disk-raw-log.lua add
lua disk-raw-log.lua show

Watch a disk for changes

Logs every change made on a mounted disk, by anyone, as it happens.

-- Watch a disk's directories for changes, whoever makes them (this computer, another one on the
-- same disk, a Disk Station), and log each one with the world's time.
--
-- usage: lua disk-watch.lua [log [directory...]] &
--   log        where the lines go (default /home/disk-watch.log)
--   directory  what to watch (default /mnt/sata/1, the disk in SATA slot 1)
--
-- A disk reports every change made on it, and mcos turns its reports into inotify notices, in the
-- paths it's mounted at: "CREATE /mnt/sata/1/photos", "MODIFY ...", "DELETE ...", a rename as
-- "MOVED_FROM" and "MOVED_TO", and "OVERFLOW" when notices were lost or the disk was formatted.
-- Watching a directory reports its entries too. The read blocks until a notice comes: no time is
-- used while nothing happens.
local sys = require("sys")

local args = {...}
local logname = args[1] or "/home/disk-watch.log"
local watched = {table.unpack(args, 2)}
if #watched == 0 then
  watched = {"/mnt/sata/1"}
end

local fd = assert(sys.inotify())
for _, path in ipairs(watched) do
  assert(sys.watch(fd, path))
end

local pending = "" -- a notice read only in part, waiting for its end
while true do
  local data = assert(sys.read(fd)) -- blocks until there are notices
  pending = pending .. data
  local lines = {}
  for notice in pending:gmatch("([^\n]*)\n") do
    lines[#lines + 1] = os.date("%T ") .. notice .. "\n"
  end
  pending = pending:match("[^\n]*$")
  if #lines > 0 then
    local log = assert(io.open(logname, "a")) -- (opened and closed each time: a disk's file shows
    log:write(table.concat(lines)) --          its writes at close)
    log:close()
  end
end

Download disk-watch.lua

Survey every disk

Asks each disk on the bus for its format, its use and what is at its root, through ioctls on its /dev file, whether mcos mounted it or not.

-- Survey every disk on the bus: its format, how full it is, and what's at its root, asked of each
-- disk directly through ioctls on its /dev file, mounted or not.
--
-- usage: lua disk-survey.lua
--
-- A device file's ioctls are its device's methods, by name: getMode, getCapacity and getUsed work on
-- any disk; list and stat on an hfs one (a raw disk refuses them: it has no files). Paths given to
-- them are the disk's own, from its root, whatever mcos mounts it as.
local sys = require("sys")

local KIB = 1024

-- One disk's lines.
local function survey(device)
  local disk, why = io.open("/dev/" .. device.name)
  if not disk then
    return device.name .. ": " .. why .. "\n"
  end
  local fd = sys.fileno(disk)
  local mode = assert(sys.ioctl(fd, "getMode"))
  local capacity = assert(sys.ioctl(fd, "getCapacity"))
  local used = assert(sys.ioctl(fd, "getUsed"))
  local lines = {string.format("%s %s %s %d/%d KiB used%s", device.name, device.path or device.id, mode,
    used // KIB, capacity // KIB, device.label and (" \"" .. device.label .. "\"") or "")}
  if mode == "hfs" then
    for _, entry in ipairs(assert(sys.ioctl(fd, "list", "/", true))) do
      local stat = sys.ioctl(fd, "stat", "/" .. entry.name, true) -- (true: a link itself)
      if entry.type == "directory" then
        lines[#lines + 1] = "  " .. entry.name .. "/"
      elseif entry.type == "link" then
        lines[#lines + 1] = "  " .. entry.name .. " -> " .. stat.link
      else
        lines[#lines + 1] = string.format("  %s %d", entry.name, stat.size)
      end
    end
  end
  disk:close()
  return table.concat(lines, "\n") .. "\n"
end

for _, device in ipairs(sys.devices()) do
  if device.type == "storage" then
    io.write(survey(device))
  end
end

Download disk-survey.lua