Backend SSE handler already accepts ?tenantId= as an additive filter
on top of canSeeOp (Phase G stays load-bearing); frontend now passes
the global tenant filter chip's value through both NotificationBell
and ActivityTab. Watcher clears + restarts the stream when the
super-admin switches tenant context. Dismissed Set is user-level
and survives the switch.
#2 Dismiss / Clear failed:
- Per-row [×] (text 'Dismiss', shown on hover) on terminal ops
- 'Clear' button in dropdown header when any dismissable rows exist
- Dismissed IDs persisted to localStorage (tower_dismissed_ops)
- Pruned during the 30s sweep when the underlying op falls out
of the recent window — keeps storage from growing forever
- badgeCount + failedCount filter dismissed entries so the red
pill clears the moment the operator acks the failure
#4 Toast popups:
- useToast composable + Toaster.vue mounted at App.vue
- Triggered from NotificationBell.upsert on terminal transition
when the op was running ≥30s AND the bell isn't open
- Success: 5s auto-dismiss; Failure: sticky until clicked away
- Click-through links to the per-instance Activity timeline
(#op_<id>) for any op carrying instanceCode
- Stack of 3, oldest drops on overflow
- No external dep — hand-rolled to match v3's component style
NotificationBell + ActivityTab opened EventSource without auth
(native EventSource API can't set Authorization headers). Phase G's
canSeeOp guard correctly dropped every event for the resulting
anonymous viewer, leaving the bell silent except for the one-shot
backfill on mount.
Backend: claimsFromRequest now falls back to ?token= query param
when the Authorization header is absent. HTTPS-only ingress means
the token stays inside the TLS tunnel; the 15-min access-token TTL
bounds any leakage if it ever surfaces in browser history or proxy
logs.
Frontend: streamOperation + streamAllOperations append the access
token via streamURL(). Plus token-expiry-aware reconnect: on
EventSource error, debounce 5s, close, run authFetch('/me') to let
the 0.61.18 refresh path renew the access token, then re-open with
a fresh streamURL. Without this, the native auto-reconnect would
loop forever with the now-stale token after 15 min.
The grep -oE for instanceCode matches BOTH provenance.instanceCode
AND the v2 root mirror, returning two lines. sed processes each line
but the resulting SOURCE_CODE shell variable was multi-line, which
made the directory check fail (-d "/var/lib/odoo/filestore/odoo16
[newline]odoo16") → rename branch silently skipped → Odoo with
db_name=odoo16v2 looks at /var/lib/odoo/filestore/odoo16v2/, finds
nothing, returns 500 on every asset.
Added head -1 to the pipe so SOURCE_CODE is single-line, plus an
echo so the rename branch's path is visible in Job logs even when
it short-circuits.
Phase C made instance-create tenant-aware for Cloudflare DNS, but
migrate.go and templates_deploy.go kept using the legacy global
*cloudflareClient (zone=odoosky.org). Result: a tenant migrate to
4th.online silently created the A record under odoosky.org as a
literal subdomain ('odoo16v2.tenants.4th.online.odoosky.org' →
right IP) — Tower logged 'DNS A record set' successfully because
the API accepted the call, but the actual hostname the user
browses to was never published to the right zone.
Both flows now use cfResolver.clientFor(tenantID, fqdn) to find
the tenant's CF token + correct zone. If no token covers the
domain, the op fails with a clear 'configure tenant CF token'
message instead of silently writing to the wrong zone.
MigrateDrawer hardcoded '<option value="">Platform server (default)</option>' as
the first picker entry, regardless of tenant scope. A tenant operator
saw it as a selectable default — and selecting it (or just leaving
the default empty) sent the migrate to the platform cluster, which
the operator has no business deploying to.
Now: removed the hardcoded option. Auto-pick the first deployable
non-platform server on load (matches DeployInstanceDrawer pattern).
Picker shows 'Pick a server…' as a disabled placeholder when nothing
is selected.
The export Job was using a stale platform Secret (s3-backup-creds)
and hardcoded bucket/endpoint, so bundles landed in odoosky-v3-backups
while Tower's verify (tenant-scoped after Phase F) read from the
tenant's bucket. Result: 'bundle missing from S3 after job
succeeded' even though the upload itself worked.
Same bug existed in import. Both fixed: keys+region+endpoint+bucket
now come from Tower's resolver view of the tenant, passed directly
into the Job env.
Plus: BackupsView crashed on r.backups.runs.length when runs is
null. Added the missing null guard.
Frontend authFetch was bouncing every 401 straight to /login,
ignoring the 30-day refresh-token cookie the backend already issues.
Result: access-token TTL is 15 min, so the operator was kicked to
login every 15 min of idle.
Now: on 401, authFetch silently calls /api/auth/refresh, retries
the original request once with the new access token, and only
bounces to /login if refresh ALSO fails (refresh cookie expired or
revoked). Concurrent 401s coalesce onto a single in-flight refresh
to avoid rotating the refresh-token jti N times in a burst.
handleUpdateAddons was fully synchronous including BuildKit Jobs per
addon image. cetmix_tower auto-pulls 8 deps × ~30-90s each on a fresh
cluster = 5-15 min. Reverse proxy timeout (60-120s) cuts the request
mid-build, browser shows 'Saving' forever, drawer eventually closes
on the timeout error, AND the cancelled context kills the goroutine
mid-flight so values.yaml never gets committed.
Now: handler validates inline (immediate feedback on bad input),
spawns an addon-stage op, returns 202 + opId in <1s. The goroutine
runs phases (resolve → build → commit → refresh) with a fresh
context that survives client disconnect. Operator watches it in the
bell + Activity tab, can keep working in another tab. Same pattern
Deploy/Migrate/Apply already use.
Two bugs the calm re-review surfaced:
1. substrateGateRejection used ListApplications (instance-only
selector) — same bug we fixed in handleListServers but missed in
the gate function. Result: the BACKEND gate was a pass-through
for every preparing server. Frontend disable held the line in
normal use, but the non-bypassable layer wasn't actually
non-bypassable. Now uses ListClusterPlatformApps.
2. PlatformAppBadge had its own Argo→state mapping that classified
Health=Degraded as 'Failed' immediately. Backend's
deriveSubstrateStatus is forgiving within the budget (Degraded
inside 10-min window reads as preparing). The two parallel
logics meant the badge could show red 'Failed' while the gate
considered it 'preparing'. Refactored badge to consume the
backend's substrateStatus enum directly — single source of
truth, no more disagreement.
DNS-01 against Cloudflare reliably takes 6-8 min for first-issue
(Cloudflare TXT write + global propagation + LE polling). The 5-min
budget red-flagged normal installs. Bumped to 10 min.
Also: Argo flickers to Degraded during retries while waiting on the
Certificate hook. Within the budget window we now classify Degraded
as 'preparing' instead of 'degraded' — only declare degraded after
the budget fully expires. Stops the badge from showing 'Failed'
during a healthy install.
ListApplications uses selector odoosky.io/component=instance which
hides cluster-platform apps. Separate ListClusterPlatformApps method
selects on app.kubernetes.io/managed-by=tower then filters by
-platform suffix — no backfill needed for existing servers.
Without this, substrateStatus + the Phase B4 PlatformAppBadge both
silently fell to 'unknown' / invisible because the lookup map was
empty.
Servers connected via the URL-token flow take ~5 min for the
cluster-platform-v3 chart to install. Without a gate, an over-eager
operator could click Deploy on a fresh server and land an instance
on a half-built cluster — Argo sync errors on missing CRDs, Pod
ImagePullBackOff on the in-cluster registry that doesn't exist yet.
Two layers, both shipped here:
Backend (defensive, non-bypassable):
- argoCluster gains substrateStatus enum (ready | preparing |
degraded | unknown) and substrateETA (RFC3339, when preparing)
derived from the per-cluster <name>-platform Argo App's
health+sync + the App's createdAt + 5min budget.
- handleCreateInstance + handleApplyMigration refuse with 409 +
"server still preparing — please wait" when the gate fails.
Same posture as the in-flight-delete check we shipped in 0.61.11.
Frontend (visible UX, prevents the user reaching the gate):
- DeployInstanceDrawer + MigrateDrawer server pickers disable
rows where substrateStatus !== ready/unknown. Disabled rows
show "Preparing · ~Xm" with a live countdown.
- Default-server selection prefers a deployable row.
- PlatformAppBadge shows the same countdown on the server card +
detail vitals panel — operator can watch it tick down without
leaving the Servers list.
- Drawers auto-poll listServers() while a preparing server is
visible — rows enable themselves the moment the chart lands.
Customer running the connect URL was getting the entire k3s install
transcript scrolled to their terminal — including the base64-encoded
kubeconfig (cluster-admin certs visible in scrollback). Two problems:
1. UX: violates "Tower silent in the background" platform principle.
2. Security: cluster-admin material visible to anyone shoulder-surfing
or screen-sharing.
wrapQuiet() in connect_token.go now wraps bootstrap + trailer:
- all output → /var/log/odoosky-connect.log (operator-readable)
- ONE friendly line to terminal at start ("Connecting…")
- ONE outcome line at end (✓ success / ⚠ failure)
- on non-zero exit: dump last 30 log lines so customer isn't
staring at a silent terminal
Kubeconfig is already tee'd to /tmp/odoosky-kubeconfig.yaml by the
bootstrap, so the trailer reads it from disk — never needs stdout.
End-to-end smoke surfaced two bugs that combine to nuke a fresh
instance:
1. ForceDeleteApplication was unreachable because Argo's HTTP PATCH
is gRPC-gateway-backed and requires a structured body
`{name, patch, patchType}` where patch is the merge-patch encoded
as a string. We were sending a raw merge-patch with `?type=merge`,
which Argo rejects with HTTP 500: `proto: required field
"patchType" not set`. Result: every instance-delete in history is
marked "failed" because force-delete fallback never worked.
2. handleCreateInstance only checked `applicationExists` for the
conflict guard. ArgoCD's cascade can take 5+ minutes to actually
tear down (PVC protection, dead customer cluster) — long enough
that applicationExists returns false but the cascade is STILL
running. If a fresh create lands in that window, the new App
takes the same name and the stale cascade clobbers it when it
finally finishes.
Fix#2: also reject create when an instance-delete op for the same
code is in pending/running state. Operator gets a clear "delete
still in progress — please wait" message instead of the silent
ten-minutes-later "instance vanished" failure.
Confirmed in production by op log:
10:28:46 instance-delete odoo16 (running 6 min, finally errored
out at 10:34:48 with the patchType marshal error)
10:29:59 instance-create odoo16 (succeeded at 10:30:02 — landed
inside the still-cascading window)
10:36:08 addon-apply odoo16 (failed: HTTP 403 — Argo App gone)
Pre-test review of 0.61.9 surfaced two issues in the manifest reader:
1. v3 stores instanceCode under provenance.* but readManifestString
only looked at recipe → root. Today the v2 root mirror covers it,
but a future v4 dropping that mirror would silently lose the
filestore-rename hint.
2. Adding a blanket provenance lookup re-opened the leak: a poison
bundle could embed provenance.tenantId and have it reachable to
any future caller.
Fix: provenance lookup is now allowlisted to {instanceCode}. Any
new provenance field requires an explicit constant addition,
which is a code-review gate against re-introducing the leak.
Round-trip simulation (tools/phase_i_simulate.go) passes for v3,
v3-pure (no v2 mirrors), v3-poison, and v2.
Bundles now emit odoosky.export.v3 schema with a recipe sub-object
and explicit provenance section. The `source` field is dropped from
addon entries (the Phase I tenant leak we identified). Readers handle
both v2 and v3 schemas; v2 bundles in flight stay restorable but their
addon `source` field is silently ignored at restore time.
Phase I closes the export-bundle side of the cross-tenant data leak.
bootstrap.sh now writes /etc/rancher/k3s/registries.yaml BEFORE k3s
starts, mapping the cluster-platform-v3 registry's in-cluster DNS
hostname to the localhost NodePort the host's containerd can reach.
Without this, every Odoo Pod ImagePullBackOffs on its addon
initContainers — caught 2026-04-30 mid-migrate.
ApplyConnectSecrets now also applies docker-mirror-pull (a docker-
config-json Secret in odoosky-system) when the platform-side env
provides DOCKER_MIRROR_{REGISTRY,USERNAME,PASSWORD}. Until today
the customer cluster's BuildKit Jobs sat in Init:0/1 for ~14 minutes
waiting on a non-existent docker-mirror-pull, blocking every
addon-build the migrate flow needs.
Both gaps were silent — neither produced a visible error in Tower's
op log; the cluster sat there waiting on a kubelet that couldn't
resolve and a Job that couldn't mount. Connect now fully provisions
both at substrate setup time, no manual post-step.
Threads:
- new EnvProvider methods: DockerMirror{Registry,Username,Password}
- new ConnectSecrets fields + applier method
- chart values pull from existingSecret keys DOCKER_MIRROR_*
- bootstrap.sh idempotent registries.yaml + systemctl restart on
re-Connect to pick up updated routing rules
Phase G: every Operation now carries (TenantID, ActorUserID, ActorEmail)
stamped at opStore.Create from the request scope. The bell SSE stream
filters per event against the caller's scope before emitting (closes
the cross-tenant leak — non-super-admin users no longer see other
tenants' ops). Get / Cancel / Stream-one return 404 (not 403) when
the caller can't see the op so existence isn't probable across
tenants. List endpoint uses op.TenantID directly (covers in-flight
ops with no Argo App yet); legacy ops with empty tenant fall back
to the Argo lookup so the upgrade is seamless.
Delete leak: cascade-delete failure used to fail the whole flow,
stranding the Gitea overlay repo + DNS A record. Now: cascade
fails → escalate to ForceDeleteApplication (strip finalizers) →
continue to repo + DNS cleanup. Both fail only when ArgoCD itself
is unreachable. Caught when odoo16v2 left tenant-havari/instance-
odoo16v2 orphaned across the smoke test.
Tests + build green.
The MigrateDrawer's auto-suggest watcher compared form.value.domain
against staged.sourceCode (the *initial* suggestion from the bundle)
to decide whether the domain field had been hand-edited. Result: the
moment the operator typed past the initial suggestion (e.g. typed
'odoo16v' then 'odoo16v2'), every subsequent code edit was treated
as a hand-edit and the domain stuck at the previous suggestion.
Concrete failure caught live: typing 'odoo16v2' as the new code left
domain='odoo16v.tenants.4th.online' (missing the '2'). Argo App
annotation, instance row, and Cloudflare A record all carried the
truncated domain.
Match DeployInstanceDrawer's pattern: track the last value we
auto-suggested in a previousDerivedDomain ref, compare against that
on each watcher fire.
handleListServers's existing tenant filter would still admit a
platform cluster (type='platform') if it had been mis-labeled with
a tenant or if a tenant happened to host an instance there. Belt-
and-braces: explicit reject of any cluster with type='platform'
when the request is tenant-scoped (non-super-admin). The platform
control-plane runs Tower itself + platform-tenant template builds
— it is not a deployment target for customer tenants and surfacing
it in their Server picker breaks the bring-your-own-cluster model.
Caught while smoke-testing MigrateDrawer: a fresh tenant's Server
dropdown defaulted to 'Platform server', risking a customer
deploying their tenant data onto the operator's shared infra by
accident.
The instance row's '<code>.tenants.odoosky.org' was being computed
client-side from the code alone, so a tenant whose domain is
'4th.online' still saw 'odoo16.tenants.odoosky.org' in the list +
the Open button — wrong zone, no cert, scary Firefox warning.
Backend: Argo App now carries an 'odoosky.io/domain' annotation
written at create time from req.Domain (the values.yaml domain),
read back in argoApplicationSummary.Domain. Delete handler reads
the same annotation so DNS cleanup hits the right Cloudflare zone
instead of the platform default.
Frontend: Instance.domain field, used by InstancesView, Vitals,
ActionBar, with a fallback to the legacy pattern for any pre-Phase-F
Argo App that hasn't been backfilled yet.
Backfill for live odoo16: kubectl annotate done out-of-band.
DeployInstanceDrawer + MigrateDrawer were hardcoding
'<code>.tenants.odoosky.org' as the auto-suggested instance domain,
even when the operator's tenant has its own domain set in Settings.
A tenant whose wildcardHost is '*.tenants.4th.online' would still see
the wizard pre-fill 'odoo16.tenants.odoosky.org' in the Domain field —
the suggestion landed in the wrong zone, instance create would fail
DNS write.
Both drawers now fetch /api/tenants/<id>/settings on mount and use
'tenants.<domain>' (with leading '*.' stripped from wildcardHost) as
the suffix, falling back to odoosky.org only if the call errors or
the field is empty.
UpsertCluster was not setting the odoosky.io/tenant-id label on the
ArgoCD cluster Secret. Result: handleListServers tenant filter
attributed every freshly-connected cluster to no-tenant, making the
just-connected box invisible to the operator who registered it.
Threads ownerTenantID from the connect token through to the body of
the POST /api/v1/clusters call. Backfill of any pre-fix cluster
Secret is a one-line kubectl label.
cloudflare_resolver.go reads data['token'] but the writer (and the
test endpoint) stores under data['api_token']. Result: every fresh
tenant's CF resolver returned no token even when one was saved,
killing DNS records view AND any instance lifecycle DNS write.
Caught while smoke-testing the multi-tenant signup flow.
3 bugs caught while smoke-testing the multi-tenant signup → first
deploy flow:
1. /api/me returned capabilities: null when scope.role was nil
(a tenant member viewing the platform tenant context). Frontend's
useAuth.can() does for-of on the array → TypeError. Fix: backend
returns []string{} not nil; frontend defends against null.
2. WelcomeView slug input pattern '[a-z][a-z0-9-]{2,39}' is rejected
by Firefox's HTML5 form validation under the v-flag (treats trailing
'-' as a class-range start). Move hyphen to start of the class.
3. Domain & DNS Save button is genuinely separate from the Cloudflare
token Save (by design — token rotates independently). Documenting
here so the next person doesn't think they saved when they didn't.
Refactor s3Resolver from a single-global-creds reader into a
tenant-scoped factory. Each tenant brings their own S3 endpoint,
region, three named buckets (backups + templates + audit), and
access keys (in Vault at v3/tenants/<id>/s3-credentials).
Touches:
s3.go — s3Resolver becomes factory; tenantS3 wraps
one minio.Client + bucket per tenant
audit.go — events grouped by tenantID per flush, written
to the tenant's audit bucket
backups.go — fleet view fans out one S3 LIST per tenant;
per-instance handlers resolve via Argo App
export/import/migrate — tenant resolved from Argo App label
or scope.TenantID
templates_* — per-template tenant lookup via templateTenantID
(platform tenant for OwnerPlatform manifests)
vitals.go — last-backup probe pulls tenantID before list
Adds AllTenants() to PlatformStore so the templates orphan sweep
can iterate every tenant configured with a templates bucket.
Build: tower:0.61.1 — pushed to registry.odoosky.cloud
