Compare commits

...

17 Commits

Author SHA1 Message Date
Ben Potter
0eb36bccc8 incus-vm: fix attic post-build hook (use login+push, not --server flag) 2026-04-29 12:47:50 +00:00
Ben Potter
edcffb0114 incus-vm: fix NIX_PATH and allowUnfree for NixOS workspaces
Two issues made nix-build fail out of the box for workspace users:

1. <nixpkgs> not found: NIX_PATH was empty. Fix by setting
   nix.nixPath in the coder.nix module to point at root's channel
   (/nix/var/nix/profiles/per-user/root/channels/nixos), which is
   always present after provisioning.

2. Unfree packages blocked: coder's nix derivation has an
   unfreeRedistributable license. Fix by:
   - Setting nixpkgs.config.allowUnfree = true in coder.nix
     (system-wide default)
   - Writing ~/.config/nixpkgs/config.nix with allowUnfree=true
     for the workspace user

Also tightened the channel check from the directory itself to the
nixos subdirectory inside it.
2026-04-29 12:33:22 +00:00
Ben Potter
5f2daa573f incus-vm: bump disk default to 50GB, max to 500GB
20GB is insufficient for NixOS builds (Go module cache alone can exceed
it). 50GB is a safer default; 500GB ceiling gives room for heavy workloads.
2026-04-27 13:57:19 +00:00
Ben Potter
d5bdf7f9f5 incus-vm: skip incus_image copy for NixOS, use alias directly
The incus provider errors with 'source and target servers must be
different' when copying within the same remote. NixOS images are
pre-imported on the incus host with a local alias (nixos/25.11), so
skip the copy entirely (count=0) and pass the alias string directly
as incus_instance.image for NixOS workspaces.
2026-04-27 13:26:47 +00:00
Ben Potter
a8e04991bf incus-vm: fix NixOS image lookup using same-remote alias resolution
source_image.remote="local" was resolving against the provisioner machine,
not the target incus host. Use local.incus_remote (same remote as
destination) so the alias nixos/25.11 is looked up on the host that
actually has the image cached.
2026-04-27 13:06:13 +00:00
Ben Potter
5260309f4f incus-vm: use local: remote for NixOS images, drop nixos/unstable option
linuxcontainers.org (images: remote) does not carry NixOS. Images are
imported locally and aliased (e.g. nixos/25.11). Use local: as the
source remote when is_nixos, and images: for Ubuntu/other distros.

Also removes the nixos/unstable option which has no corresponding
locally-imported image.
2026-04-25 20:22:22 +00:00
Ben Potter
60cb9d9bfc incus-vm: use extra-substituters/extra-trusted-public-keys for attic
Avoids duplicating cache.nixos.org which NixOS already sets as a default
substituter. Using extra-* variants appends to the NixOS defaults rather
than replacing them, keeping a single cache.nixos.org entry in nix.conf.
2026-04-24 23:08:18 +00:00
Ben Potter
478fb7806d incus-vm: replace shared 9p nix store with Attic binary cache
Instead of mounting the ThinkStation's /data/nix via 9p into each NixOS VM
and redirecting the nix store there, use an Attic binary cache server running
on the ThinkStation (port 8080) to share build outputs across VMs.

Changes:
- Remove nix-shared Incus profile from VM profiles (main.tf)
- Remove all shared-store complexity from nixos.tf:
  * nix.settings.store (local?root=/nix-host)
  * systemd.mounts bind mount of /nix-host/nix/store over /nix/store
  * system.activationScripts.nix-host-dir
  * Pre-patch of /etc/nix/nix.conf before nixos-rebuild
  * Pre-apply bind mount before nixos-rebuild
- Add Attic cache configuration to nixos.tf:
  * nix.settings.substituters includes http://10.78.3.1:8080/main
  * nix.settings.trusted-public-keys includes the attic main key
  * nix.settings.post-build-hook = /etc/nix/post-build-hook.sh
  * environment.systemPackages includes pkgs.attic-client
- Provisioner writes post-build-hook.sh and attic-token to VM
- Add attic_url/cache/pubkey/push_token locals

The Attic server is already running on ThinkStation with:
  - Cache: main (public: true)
  - Public key: main:+O2V0KSKDos1vrth+xucxa7DCW3UX05JVwc+2WKKEUw=
  - Push token scoped to pull+push on main cache
2026-04-24 22:19:35 +00:00
Ben Potter
fde4f8dbb9 feat(incus-vm/nixos): bind-mount HDD store over /nix/store for result symlinks
The VM image bakes the NixOS closure into /nix/store (ext4 ro on sda2).
The nix-shared profile mounts the ThinkStation HDD store at /nix-host/nix
via 9p. nix.settings.store redirects daemon writes to /nix-host/nix/store,
but result symlinks still resolve against /nix/store (the ext4 partition) -
causing "No such file or directory" when running built binaries.

Fix: add a systemd.mounts entry that bind-mounts /nix-host/nix/store over
/nix/store after local-fs.target (when the 9p share is up) and before
nix-daemon.service. This makes /nix/store point at the live HDD store,
so both nix daemon internals and result symlinks work correctly.

The provisioner also pre-applies the bind mount before nixos-rebuild switch
so the first rebuild (which produces a new system derivation) can activate
successfully - without the bind mount, activation aborts because it looks
for the newly built path in the stale ext4 /nix/store.
2026-04-24 21:22:29 +00:00
Ben Potter
668f776d87 fix(incus-vm): use thinkstation profile for HDD pool, set size via null_resource 2026-04-24 19:46:50 +00:00
Ben Potter
906db0a36c fix(incus-vm): remove pool from root device, ignore profiles/device in lifecycle 2026-04-24 19:39:11 +00:00
Ben Potter
3045e433b9 feat(incus-vm): use HDD storage pool for ThinkStation VMs 2026-04-24 15:29:28 +00:00
Ben Potter
4132c53acf feat(incus-vm): add configurable disk size parameter (default 20GiB) 2026-04-24 15:13:20 +00:00
Ben Potter
19caa9598c feat(incus-vm): shared host nix-daemon for NixOS VMs via nix-shared profile
- Add nix-shared Incus profile on ThinkStation (bind /data/nix -> /nix)
- Apply nix-shared profile to NixOS VMs on ThinkStation
- coder.nix: disable VM nix-daemon/socket, use host daemon socket
- coder.nix: trusted-users includes workspace user
- coder.nix: override /nix/store fstab to bind from host-mounted /nix
2026-04-24 14:12:30 +00:00
Ben Potter
a262565650 refactor(incus-vm): split NixOS provisioning into nixos.tf
Move local.is_nixos and null_resource.provision_nixos out of main.tf
into a dedicated nixos.tf to keep main.tf focused on core infrastructure.
2026-04-24 13:50:34 +00:00
Ben Potter
cd76e01fa2 fix(incus-vm): tie boot.autostart to workspace start_count
VMs were unconditionally set to boot.autostart=true, meaning after a
host reboot all VMs (including stopped workspaces) would come back up.
With 18 VMs auto-starting, dnsmasq hit its 150 concurrent DNS query
limit, cascading into incus.service failures and a watchdog reboot.

Now boot.autostart mirrors the workspace state so stopped workspaces
stay stopped across host reboots.
2026-04-24 13:46:12 +00:00
Ben Potter
c86793f43c feat(incus-vm): add NixOS VM support
- Add nixos/25.11 and nixos/unstable image options
- Skip cloud-init config for NixOS images (not supported)
- Add null_resource.provision_nixos that:
  - Waits for incus-agent readiness
  - Pushes coder agent init script and env file via incus file push
  - Generates /etc/nixos/coder.nix declaring the workspace user,
    sudoers, and coder-agent systemd service
  - Patches /etc/nixos/configuration.nix to import coder.nix
  - Runs nixos-rebuild switch (tolerates exit 4 = service warning)
  - Explicitly restarts coder-agent.service post-rebuild
- Service unit sets Environment PATH so curl/tools are findable
- Disable token_refresh resource for NixOS (not needed)
- NixOS image alias does not append arch suffix (unlike Ubuntu)
2026-04-23 23:50:59 +00:00
2 changed files with 638 additions and 0 deletions

View File

@ -0,0 +1,453 @@
terraform {
required_providers {
coder = {
source = "coder/coder"
version = ">= 2.4.0"
}
incus = {
source = "lxc/incus"
version = "~> 1.0"
}
null = {
source = "hashicorp/null"
version = "~> 3.0"
}
}
}
module "portabledesktop" {
count = data.coder_workspace.me.start_count
source = "registry.coder.com/coder/portabledesktop/coder"
version = "~> 0.1"
agent_id = coder_agent.main[0].id
}
provider "incus" {}
data "coder_workspace" "me" {}
data "coder_workspace_owner" "me" {}
data "coder_parameter" "host" {
name = "host"
display_name = "Host"
description = "Select the host to run this workspace on. **ThinkStation** is an amd64 desktop machine. **CoderPi** is an arm64 Raspberry Pi."
type = "string"
default = "ThinkStation"
mutable = false
order = 1
option {
name = "ThinkStation"
value = "ThinkStation"
icon = "/icon/desktop.svg"
}
option {
name = "CoderPi"
value = "CoderPi"
icon = "/icon/memory.svg"
}
}
data "coder_parameter" "image" {
name = "image"
display_name = "Image"
description = "The image to use. Ubuntu images use cloud-init. NixOS images are provisioned via incus exec + nixos-rebuild."
default = "ubuntu/jammy/cloud"
icon = "/icon/image.svg"
mutable = true
option {
name = "Ubuntu 22.04 LTS (Jammy)"
value = "ubuntu/jammy/cloud"
icon = "/icon/ubuntu.svg"
}
option {
name = "Ubuntu 24.04 LTS (Noble)"
value = "ubuntu/noble/cloud"
icon = "/icon/ubuntu.svg"
}
option {
name = "NixOS 25.11"
value = "nixos/25.11"
icon = "/icon/nix.svg"
}
}
data "coder_parameter" "cpu" {
name = "cpu"
display_name = "CPU"
description = "Number of CPUs to allocate."
type = "number"
form_type = "dropdown"
default = 2
icon = "https://raw.githubusercontent.com/matifali/logos/main/cpu-3.svg"
mutable = true
order = 2
dynamic "option" {
for_each = data.coder_parameter.host.value == "ThinkStation" ? [1, 2, 4, 6, 8, 12] : [0.5, 1, 1.5, 2]
content {
name = tostring(option.value)
value = option.value
}
}
}
data "coder_parameter" "memory" {
name = "memory"
display_name = "Memory (GB)"
description = "Amount of memory in GB."
type = "number"
form_type = "slider"
default = 4
icon = "/icon/memory.svg"
mutable = true
order = 4
validation {
min = 1
max = data.coder_parameter.host.value == "ThinkStation" ? 24 : 12
}
}
data "coder_parameter" "disk" {
name = "disk"
display_name = "Disk (GB)"
description = "Root disk size in GB."
type = "number"
form_type = "slider"
default = 50
icon = "/icon/database.svg"
mutable = true
order = 3
validation {
min = 10
max = 500
}
}
data "coder_parameter" "usb_passthrough" {
name = "usb_passthrough"
display_name = "USB Passthrough"
description = "Pass a USB device through to the VM. Only applicable when host is ThinkStation."
type = "string"
form_type = "dropdown"
default = "none"
mutable = true
order = 4
option {
name = "None"
value = "none"
}
option {
name = "Kindle Paperwhite (1949:0004)"
value = "kindle"
}
option {
name = "Nook Simple Touch (2080:0003)"
value = "nook"
}
option {
name = "Kindle Fire 1st Gen (1949:0006)"
value = "kindle_fire"
}
}
data "coder_parameter" "snapshot_on_stop" {
name = "snapshot_on_stop"
display_name = "Snapshot on stop"
description = "Take a snapshot of the VM when the workspace stops."
type = "bool"
form_type = "checkbox"
default = false
mutable = true
ephemeral = true
order = 5
}
data "coder_parameter" "snapshot_name" {
count = data.coder_parameter.snapshot_on_stop.value == "true" ? 1 : 0
name = "snapshot_name"
display_name = "Snapshot name"
description = "Name for the snapshot."
type = "string"
default = "snap-${formatdate("YYYYMMDD-hhmmss", timestamp())}"
mutable = true
ephemeral = true
order = 6
}
resource "coder_agent" "main" {
count = data.coder_workspace.me.start_count
arch = data.coder_parameter.host.value == "ThinkStation" ? "amd64" : "arm64"
os = "linux"
dir = "/home/${local.workspace_user}"
metadata {
display_name = "CPU Usage"
key = "0_cpu_usage"
script = "coder stat cpu"
interval = 10
timeout = 1
}
metadata {
display_name = "RAM Usage"
key = "1_ram_usage"
script = "coder stat mem"
interval = 10
timeout = 1
}
metadata {
display_name = "Home Disk"
key = "3_home_disk"
script = "coder stat disk --path /home/${local.workspace_user}"
interval = 60
timeout = 1
}
}
# For non-NixOS images, copy from the public images: simplestreams remote.
# NixOS is not on linuxcontainers.org pre-imported on the incus host and
# aliased (e.g. nixos/25.11). The provider cannot copy same-remote, so we
# skip this resource for NixOS and reference the alias directly below.
resource "incus_image" "image" {
count = local.is_nixos ? 0 : 1
remote = local.incus_remote
source_image = {
remote = "images"
name = "${data.coder_parameter.image.value}/${data.coder_parameter.host.value == "ThinkStation" ? "amd64" : "arm64"}"
type = "virtual-machine"
architecture = data.coder_parameter.host.value == "ThinkStation" ? "x86_64" : "aarch64"
}
}
resource "incus_instance" "dev" {
remote = local.incus_remote
running = data.coder_workspace.me.start_count == 1
name = "coder-${lower(data.coder_workspace_owner.me.name)}-${lower(data.coder_workspace.me.name)}"
# NixOS: reference pre-imported alias directly; Ubuntu: use copied fingerprint.
image = local.is_nixos ? data.coder_parameter.image.value : incus_image.image[0].fingerprint
type = "virtual-machine"
profiles = data.coder_parameter.host.value == "ThinkStation" ? ["thinkstation"] : ["default"]
dynamic "device" {
for_each = local.usb_device != null ? [local.usb_device] : []
content {
name = device.value.name
type = "usb"
properties = {
vendorid = device.value.vendorid
productid = device.value.productid
}
}
}
lifecycle {
ignore_changes = [
config["cloud-init.user-data"],
config["user.coder-agent-token"],
config["raw.qemu.conf"],
image,
device,
]
}
config = merge(
{
"limits.cpu" = tostring(local.cpu)
"limits.memory" = "${local.memory}GiB"
"raw.qemu.conf" = <<-QEMUCONF
[device "qemu_balloon"]
driver = "virtio-balloon-pci"
bus = "qemu_pcie0"
addr = "00.0"
multifunction = "on"
free-page-reporting = "on"
QEMUCONF
"security.secureboot" = false
"boot.autostart" = data.coder_workspace.me.start_count == 1
"user.coder-agent-token" = local.agent_token
},
local.is_nixos ? {} : {
"cloud-init.user-data" = <<-EOF
#cloud-config
hostname: ${lower(data.coder_workspace.me.name)}
users:
- name: ${local.workspace_user}
uid: 1000
groups: sudo
shell: /bin/bash
sudo: ALL=(ALL) NOPASSWD:ALL
write_files:
- path: /opt/coder/init
permissions: "0755"
encoding: b64
content: ${base64encode(local.agent_init_script)}
- path: /opt/coder/init.env
permissions: "0600"
content: |
CODER_AGENT_TOKEN=${local.agent_token}
CODER_AGENT_URL=${data.coder_workspace.me.access_url}
- path: /etc/systemd/system/coder-agent.service
permissions: "0644"
content: |
[Unit]
Description=Coder Agent
After=network-online.target
Wants=network-online.target
[Service]
User=${local.workspace_user}
EnvironmentFile=/opt/coder/init.env
ExecStart=/opt/coder/init
Restart=always
RestartSec=10
TimeoutStopSec=90
KillMode=process
OOMScoreAdjust=-900
SyslogIdentifier=coder-agent
runcmd:
- apt-get update -qq && apt-get install -y curl adb
- chown -R ${local.workspace_user}:${local.workspace_user} /home/${local.workspace_user}
- |
if [ ! -s /opt/coder/init ]; then
curl -fsSL '${data.coder_workspace.me.access_url}/bin/coder-linux-amd64' -o /opt/coder/coder-agent-bin
chmod +x /opt/coder/coder-agent-bin
printf '#!/bin/bash\nsource /opt/coder/init.env\nexec /opt/coder/coder-agent-bin agent\n' > /opt/coder/init
chmod +x /opt/coder/init
fi
- systemctl enable --now coder-agent.service
EOF
}
)
}
# Set disk size. Pool is set by the profile at creation time.
resource "null_resource" "root_disk" {
triggers = {
instance = incus_instance.dev.name
size = local.disk
}
depends_on = [incus_instance.dev]
provisioner "local-exec" {
command = <<-EOT
REMOTE="${local.incus_remote}"
INSTANCE="${incus_instance.dev.name}"
SIZE="${local.disk}GiB"
# Override size only pool is already set by the profile at creation time
incus config device override "$REMOTE:$INSTANCE" root size=$SIZE 2>/dev/null || \
incus config device set "$REMOTE:$INSTANCE" root size=$SIZE
EOT
}
}
# Token refresh for Ubuntu/cloud-init VMs
resource "null_resource" "token_refresh" {
count = data.coder_workspace.me.start_count == 1 && !local.is_nixos ? 1 : 0
triggers = {
agent_token = local.agent_token
instance = incus_instance.dev.name
}
depends_on = [incus_instance.dev]
provisioner "local-exec" {
command = <<-EOT
echo "Waiting for VM agent to be ready..."
for i in $(seq 1 30); do
if incus exec ${local.incus_remote}:${incus_instance.dev.name} -- true 2>/dev/null; then
break
fi
echo "Attempt $i: VM agent not ready yet, waiting..."
sleep 5
done
echo "Waiting for cloud-init to complete..."
incus exec ${local.incus_remote}:${incus_instance.dev.name} -- bash -c '
for i in $(seq 1 60); do
if [ -f /var/lib/cloud/instance/boot-finished ]; then
break
fi
sleep 5
done
'
echo "Updating Coder agent token..."
incus config set ${local.incus_remote}:${incus_instance.dev.name} user.coder-agent-token ${local.agent_token}
incus exec ${local.incus_remote}:${incus_instance.dev.name} -- bash -c '
printf "CODER_AGENT_TOKEN=${local.agent_token}\nCODER_AGENT_URL=${data.coder_workspace.me.access_url}\n" > /opt/coder/init.env
chown root:root /opt/coder/init.env
chmod 600 /opt/coder/init.env
systemctl restart coder-agent
'
EOT
}
}
resource "incus_instance_snapshot" "on_stop" {
count = data.coder_parameter.snapshot_on_stop.value == "true" ? 1 : 0
remote = local.incus_remote
instance = incus_instance.dev.name
name = try(data.coder_parameter.snapshot_name[0].value, "snap")
stateful = false
lifecycle {
ignore_changes = all
}
}
locals {
incus_remote = data.coder_parameter.host.value == "ThinkStation" ? "thinkstation" : "local"
workspace_user = lower(data.coder_workspace_owner.me.name)
cpu = data.coder_parameter.cpu.value
memory = data.coder_parameter.memory.value
disk = data.coder_parameter.disk.value
agent_id = data.coder_workspace.me.start_count == 1 ? coder_agent.main[0].id : ""
agent_token = data.coder_workspace.me.start_count == 1 ? coder_agent.main[0].token : ""
agent_init_script = data.coder_workspace.me.start_count == 1 ? coder_agent.main[0].init_script : ""
# USB device map add more entries here as needed
usb_devices = {
kindle = { name = "kindle", vendorid = "1949", productid = "0004" }
nook = { name = "nook", vendorid = "2080", productid = "0003" }
kindle_fire = { name = "kindle_fire", vendorid = "1949", productid = "0006" }
}
usb_passthrough_value = data.coder_parameter.usb_passthrough.value
usb_device = (
local.usb_passthrough_value != "none" &&
data.coder_parameter.host.value == "ThinkStation" &&
contains(keys(local.usb_devices), local.usb_passthrough_value)
) ? local.usb_devices[local.usb_passthrough_value] : null
}
resource "coder_metadata" "info" {
count = data.coder_workspace.me.start_count
resource_id = incus_instance.dev.name
item {
key = "instance"
value = incus_instance.dev.name
}
item {
key = "image"
value = "images:${data.coder_parameter.image.value}"
}
item {
key = "cpus"
value = tostring(local.cpu)
}
item {
key = "memory"
value = tostring(local.memory)
}
}

View File

@ -0,0 +1,185 @@
# NixOS-specific provisioning for incus-vm workspaces.
#
# NixOS doesn't support cloud-init, so instead we:
# 1. Push the coder agent init script and env file via incus file push
# 2. Generate /etc/nixos/coder.nix declaring the user + coder-agent service
# 3. Patch configuration.nix to import coder.nix
# 4. Run nixos-rebuild switch
# 5. Restart coder-agent.service with the fresh token
#
# This provisioner runs on every workspace start (null_resource is recreated
# each cycle), which also handles token rotation.
#
# Binary cache: an Attic server runs on the ThinkStation at 10.78.3.1:8080.
# VMs use it as a substituter so builds are shared across all NixOS VMs.
# A post-build hook auto-pushes new store paths to the cache after each build.
locals {
# NixOS images on images.linuxcontainers.org use just "nixos/25.11" with no
# arch suffix in the alias unlike Ubuntu which appends e.g. "/amd64".
is_nixos = startswith(data.coder_parameter.image.value, "nixos/")
# Attic binary cache on ThinkStation (incusbr0 gateway, always reachable from VMs).
attic_url = "http://10.78.3.1:8080"
attic_cache = "main"
attic_pubkey = "main:+O2V0KSKDos1vrth+xucxa7DCW3UX05JVwc+2WKKEUw="
# Push token pull+push to main cache, no admin rights.
attic_push_token = "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjI2NDA5Nzk5NjQsIm5iZiI6MTc3NzA2NjM2NCwic3ViIjoibml4b3Mtdm0iLCJodHRwczovL2p3dC5hdHRpYy5ycy92MSI6eyJjYWNoZXMiOnsibWFpbiI6eyJyIjoxLCJ3IjoxfX19fQ.GhVnty_hfoEjp1WHId9a8UUGahtbDJpTL-gt7tJqkwM"
}
resource "null_resource" "provision_nixos" {
count = data.coder_workspace.me.start_count == 1 && local.is_nixos ? 1 : 0
triggers = {
agent_token = local.agent_token
instance = incus_instance.dev.name
}
depends_on = [incus_instance.dev]
provisioner "local-exec" {
command = <<-EOT
set -e
REMOTE="${local.incus_remote}"
INSTANCE="${incus_instance.dev.name}"
WUSER="${local.workspace_user}"
ARCH="${data.coder_parameter.host.value == "ThinkStation" ? "amd64" : "arm64"}"
ATTIC_URL="${local.attic_url}"
ATTIC_CACHE="${local.attic_cache}"
ATTIC_PUBKEY="${local.attic_pubkey}"
ATTIC_TOKEN="${local.attic_push_token}"
echo "Waiting for NixOS VM incus-agent to be ready..."
for i in $(seq 1 60); do
if incus exec "$REMOTE:$INSTANCE" -- true 2>/dev/null; then
echo "incus-agent ready after $i attempts"
break
fi
echo "Attempt $i: incus-agent not ready yet, waiting..."
sleep 5
done
# Write init script into the VM
incus exec "$REMOTE:$INSTANCE" -- mkdir -p /opt/coder
echo "${base64encode(local.agent_init_script)}" | base64 -d | incus file push - "$REMOTE:$INSTANCE/opt/coder/init"
incus exec "$REMOTE:$INSTANCE" -- chmod 755 /opt/coder/init
# Write env file into the VM
printf 'CODER_AGENT_TOKEN=${local.agent_token}\nCODER_AGENT_URL=${data.coder_workspace.me.access_url}\n' \
| incus file push - "$REMOTE:$INSTANCE/opt/coder/init.env" --mode 0600
# Write the attic post-build hook script.
# Runs after every nix build and pushes new store paths to the cache.
# attic-client uses `attic login <server> <url> <token>` + `attic push <server>:<cache>`.
printf '#!/bin/sh\nset -eu\nexport HOME=/root\nATTIC_URL="%s"\nATTIC_CACHE="%s"\n[ -f /etc/nix/attic-token ] || exit 0\nTOKEN=$(cat /etc/nix/attic-token)\n/run/current-system/sw/bin/attic login thinkstation "$ATTIC_URL" "$TOKEN" 2>/dev/null || true\n/run/current-system/sw/bin/attic push "thinkstation:$ATTIC_CACHE" $OUT_PATHS 2>&1 || true\n' \
"$ATTIC_URL" "$ATTIC_CACHE" \
| incus file push - "$REMOTE:$INSTANCE/etc/nix/post-build-hook.sh" --mode 0755
# Write the attic push token (readable by nix-daemon = root)
printf '%s' "$ATTIC_TOKEN" \
| incus file push - "$REMOTE:$INSTANCE/etc/nix/attic-token" --mode 0600
# Write the NixOS coder module, substituting the username
NIXMOD=$(cat <<NIXMOD_EOF
{ config, pkgs, lib, ... }:
{
users.users."$WUSER" = {
isNormalUser = true;
uid = 1000;
home = "/home/$WUSER";
shell = pkgs.bash;
extraGroups = [ "wheel" ];
};
security.sudo.wheelNeedsPassword = false;
nix.settings.trusted-users = [ "root" "$WUSER" ];
nix.settings.allowed-users = [ "*" ];
# Make <nixpkgs> resolve for all users via NIX_PATH, and allow unfree
# packages by default so nix-build works without extra env vars.
nix.nixPath = [ "nixpkgs=/nix/var/nix/profiles/per-user/root/channels/nixos" ];
nixpkgs.config.allowUnfree = true;
# Attic binary cache on ThinkStation shared across all NixOS VMs.
# Builds are fetched from here on cache hit; new builds are pushed via
# the post-build hook below.
nix.settings.extra-substituters = [ "$ATTIC_URL/$ATTIC_CACHE" ];
nix.settings.extra-trusted-public-keys = [ "$ATTIC_PUBKEY" ];
# Auto-push every build result to the Attic cache.
nix.settings.post-build-hook = "/etc/nix/post-build-hook.sh";
# attic client needed by the post-build hook.
environment.systemPackages = [ pkgs.attic-client ];
systemd.services.coder-agent = {
description = "Coder Agent";
after = [ "network-online.target" ];
wants = [ "network-online.target" ];
wantedBy = [ "multi-user.target" ];
serviceConfig = {
User = "$WUSER";
EnvironmentFile = "/opt/coder/init.env";
ExecStart = "/opt/coder/init";
Environment = "PATH=/run/current-system/sw/bin:/run/wrappers/bin:/usr/local/bin:/usr/bin:/bin";
Restart = "always";
RestartSec = 10;
TimeoutStopSec = 90;
KillMode = "process";
OOMScoreAdjust = -900;
SyslogIdentifier = "coder-agent";
};
};
}
NIXMOD_EOF
)
echo "$NIXMOD" | incus file push - "$REMOTE:$INSTANCE/etc/nixos/coder.nix"
# Patch configuration.nix to import coder.nix if not already imported
incus exec "$REMOTE:$INSTANCE" -- \
env PATH=/run/current-system/sw/bin /run/current-system/sw/bin/bash -c \
"grep -q coder.nix /etc/nixos/configuration.nix || \
sed -i 's|imports = \[|imports = [\n ./coder.nix|' /etc/nixos/configuration.nix"
# Restore the nixos channel for root if missing this is what NIX_PATH
# points at so <nixpkgs> resolves for all users.
incus exec "$REMOTE:$INSTANCE" -- \
env PATH=/run/current-system/sw/bin /run/current-system/sw/bin/bash -c \
"NIX_CHANNEL_URL=https://channels.nixos.org/nixos-25.11; \
CHANNEL_LINK=/nix/var/nix/profiles/per-user/root/channels; \
if [ ! -e \"\$CHANNEL_LINK/nixos\" ]; then \
echo 'Restoring nixos channel...'; \
nix-channel --add \"\$NIX_CHANNEL_URL\" nixos; \
nix-channel --update nixos; \
fi"
# Set up user-level nixpkgs config (allowUnfree) so nix-build works
# without NIXPKGS_ALLOW_UNFREE=1 for the workspace user.
incus exec "$REMOTE:$INSTANCE" -- \
env PATH=/run/current-system/sw/bin /run/current-system/sw/bin/bash -c \
"mkdir -p /home/$WUSER/.config/nixpkgs && \
if [ ! -f /home/$WUSER/.config/nixpkgs/config.nix ]; then \
printf '{ allowUnfree = true; }\n' > /home/$WUSER/.config/nixpkgs/config.nix; \
chown -R 1000:1000 /home/$WUSER/.config; \
fi"
echo "Running nixos-rebuild switch (this may take a few minutes)..."
incus exec "$REMOTE:$INSTANCE" -- \
env PATH=/run/current-system/sw/bin /run/current-system/sw/bin/bash -l -c \
"nixos-rebuild switch; EC=\$?; [ \$EC -eq 0 ] || [ \$EC -eq 4 ] || exit \$EC"
echo "Restarting coder-agent service..."
incus exec "$REMOTE:$INSTANCE" -- \
env PATH=/run/current-system/sw/bin /run/current-system/sw/bin/bash -c \
"systemctl daemon-reload; systemctl restart coder-agent.service; sleep 3; systemctl status coder-agent.service || true"
# Ensure home dir ownership
incus exec "$REMOTE:$INSTANCE" -- \
env PATH=/run/current-system/sw/bin /run/current-system/sw/bin/bash -c \
"mkdir -p /home/$WUSER && chown 1000:1000 /home/$WUSER && chmod 755 /home/$WUSER"
echo "NixOS provisioning complete."
EOT
}
}