The export Job was using a stale platform Secret (s3-backup-creds)
and hardcoded bucket/endpoint, so bundles landed in odoosky-v3-backups
while Tower's verify (tenant-scoped after Phase F) read from the
tenant's bucket. Result: 'bundle missing from S3 after job
succeeded' even though the upload itself worked.
Same bug existed in import. Both fixed: keys+region+endpoint+bucket
now come from Tower's resolver view of the tenant, passed directly
into the Job env.
Plus: BackupsView crashed on r.backups.runs.length when runs is
null. Added the missing null guard.
Frontend authFetch was bouncing every 401 straight to /login,
ignoring the 30-day refresh-token cookie the backend already issues.
Result: access-token TTL is 15 min, so the operator was kicked to
login every 15 min of idle.
Now: on 401, authFetch silently calls /api/auth/refresh, retries
the original request once with the new access token, and only
bounces to /login if refresh ALSO fails (refresh cookie expired or
revoked). Concurrent 401s coalesce onto a single in-flight refresh
to avoid rotating the refresh-token jti N times in a burst.
handleUpdateAddons was fully synchronous including BuildKit Jobs per
addon image. cetmix_tower auto-pulls 8 deps × ~30-90s each on a fresh
cluster = 5-15 min. Reverse proxy timeout (60-120s) cuts the request
mid-build, browser shows 'Saving' forever, drawer eventually closes
on the timeout error, AND the cancelled context kills the goroutine
mid-flight so values.yaml never gets committed.
Now: handler validates inline (immediate feedback on bad input),
spawns an addon-stage op, returns 202 + opId in <1s. The goroutine
runs phases (resolve → build → commit → refresh) with a fresh
context that survives client disconnect. Operator watches it in the
bell + Activity tab, can keep working in another tab. Same pattern
Deploy/Migrate/Apply already use.
Two bugs the calm re-review surfaced:
1. substrateGateRejection used ListApplications (instance-only
selector) — same bug we fixed in handleListServers but missed in
the gate function. Result: the BACKEND gate was a pass-through
for every preparing server. Frontend disable held the line in
normal use, but the non-bypassable layer wasn't actually
non-bypassable. Now uses ListClusterPlatformApps.
2. PlatformAppBadge had its own Argo→state mapping that classified
Health=Degraded as 'Failed' immediately. Backend's
deriveSubstrateStatus is forgiving within the budget (Degraded
inside 10-min window reads as preparing). The two parallel
logics meant the badge could show red 'Failed' while the gate
considered it 'preparing'. Refactored badge to consume the
backend's substrateStatus enum directly — single source of
truth, no more disagreement.
DNS-01 against Cloudflare reliably takes 6-8 min for first-issue
(Cloudflare TXT write + global propagation + LE polling). The 5-min
budget red-flagged normal installs. Bumped to 10 min.
Also: Argo flickers to Degraded during retries while waiting on the
Certificate hook. Within the budget window we now classify Degraded
as 'preparing' instead of 'degraded' — only declare degraded after
the budget fully expires. Stops the badge from showing 'Failed'
during a healthy install.
ListApplications uses selector odoosky.io/component=instance which
hides cluster-platform apps. Separate ListClusterPlatformApps method
selects on app.kubernetes.io/managed-by=tower then filters by
-platform suffix — no backfill needed for existing servers.
Without this, substrateStatus + the Phase B4 PlatformAppBadge both
silently fell to 'unknown' / invisible because the lookup map was
empty.
Servers connected via the URL-token flow take ~5 min for the
cluster-platform-v3 chart to install. Without a gate, an over-eager
operator could click Deploy on a fresh server and land an instance
on a half-built cluster — Argo sync errors on missing CRDs, Pod
ImagePullBackOff on the in-cluster registry that doesn't exist yet.
Two layers, both shipped here:
Backend (defensive, non-bypassable):
- argoCluster gains substrateStatus enum (ready | preparing |
degraded | unknown) and substrateETA (RFC3339, when preparing)
derived from the per-cluster <name>-platform Argo App's
health+sync + the App's createdAt + 5min budget.
- handleCreateInstance + handleApplyMigration refuse with 409 +
"server still preparing — please wait" when the gate fails.
Same posture as the in-flight-delete check we shipped in 0.61.11.
Frontend (visible UX, prevents the user reaching the gate):
- DeployInstanceDrawer + MigrateDrawer server pickers disable
rows where substrateStatus !== ready/unknown. Disabled rows
show "Preparing · ~Xm" with a live countdown.
- Default-server selection prefers a deployable row.
- PlatformAppBadge shows the same countdown on the server card +
detail vitals panel — operator can watch it tick down without
leaving the Servers list.
- Drawers auto-poll listServers() while a preparing server is
visible — rows enable themselves the moment the chart lands.
Customer running the connect URL was getting the entire k3s install
transcript scrolled to their terminal — including the base64-encoded
kubeconfig (cluster-admin certs visible in scrollback). Two problems:
1. UX: violates "Tower silent in the background" platform principle.
2. Security: cluster-admin material visible to anyone shoulder-surfing
or screen-sharing.
wrapQuiet() in connect_token.go now wraps bootstrap + trailer:
- all output → /var/log/odoosky-connect.log (operator-readable)
- ONE friendly line to terminal at start ("Connecting…")
- ONE outcome line at end (✓ success / ⚠ failure)
- on non-zero exit: dump last 30 log lines so customer isn't
staring at a silent terminal
Kubeconfig is already tee'd to /tmp/odoosky-kubeconfig.yaml by the
bootstrap, so the trailer reads it from disk — never needs stdout.
End-to-end smoke surfaced two bugs that combine to nuke a fresh
instance:
1. ForceDeleteApplication was unreachable because Argo's HTTP PATCH
is gRPC-gateway-backed and requires a structured body
`{name, patch, patchType}` where patch is the merge-patch encoded
as a string. We were sending a raw merge-patch with `?type=merge`,
which Argo rejects with HTTP 500: `proto: required field
"patchType" not set`. Result: every instance-delete in history is
marked "failed" because force-delete fallback never worked.
2. handleCreateInstance only checked `applicationExists` for the
conflict guard. ArgoCD's cascade can take 5+ minutes to actually
tear down (PVC protection, dead customer cluster) — long enough
that applicationExists returns false but the cascade is STILL
running. If a fresh create lands in that window, the new App
takes the same name and the stale cascade clobbers it when it
finally finishes.
Fix#2: also reject create when an instance-delete op for the same
code is in pending/running state. Operator gets a clear "delete
still in progress — please wait" message instead of the silent
ten-minutes-later "instance vanished" failure.
Confirmed in production by op log:
10:28:46 instance-delete odoo16 (running 6 min, finally errored
out at 10:34:48 with the patchType marshal error)
10:29:59 instance-create odoo16 (succeeded at 10:30:02 — landed
inside the still-cascading window)
10:36:08 addon-apply odoo16 (failed: HTTP 403 — Argo App gone)
Pre-test review of 0.61.9 surfaced two issues in the manifest reader:
1. v3 stores instanceCode under provenance.* but readManifestString
only looked at recipe → root. Today the v2 root mirror covers it,
but a future v4 dropping that mirror would silently lose the
filestore-rename hint.
2. Adding a blanket provenance lookup re-opened the leak: a poison
bundle could embed provenance.tenantId and have it reachable to
any future caller.
Fix: provenance lookup is now allowlisted to {instanceCode}. Any
new provenance field requires an explicit constant addition,
which is a code-review gate against re-introducing the leak.
Round-trip simulation (tools/phase_i_simulate.go) passes for v3,
v3-pure (no v2 mirrors), v3-poison, and v2.
Bundles now emit odoosky.export.v3 schema with a recipe sub-object
and explicit provenance section. The `source` field is dropped from
addon entries (the Phase I tenant leak we identified). Readers handle
both v2 and v3 schemas; v2 bundles in flight stay restorable but their
addon `source` field is silently ignored at restore time.
Phase I closes the export-bundle side of the cross-tenant data leak.
bootstrap.sh now writes /etc/rancher/k3s/registries.yaml BEFORE k3s
starts, mapping the cluster-platform-v3 registry's in-cluster DNS
hostname to the localhost NodePort the host's containerd can reach.
Without this, every Odoo Pod ImagePullBackOffs on its addon
initContainers — caught 2026-04-30 mid-migrate.
ApplyConnectSecrets now also applies docker-mirror-pull (a docker-
config-json Secret in odoosky-system) when the platform-side env
provides DOCKER_MIRROR_{REGISTRY,USERNAME,PASSWORD}. Until today
the customer cluster's BuildKit Jobs sat in Init:0/1 for ~14 minutes
waiting on a non-existent docker-mirror-pull, blocking every
addon-build the migrate flow needs.
Both gaps were silent — neither produced a visible error in Tower's
op log; the cluster sat there waiting on a kubelet that couldn't
resolve and a Job that couldn't mount. Connect now fully provisions
both at substrate setup time, no manual post-step.
Threads:
- new EnvProvider methods: DockerMirror{Registry,Username,Password}
- new ConnectSecrets fields + applier method
- chart values pull from existingSecret keys DOCKER_MIRROR_*
- bootstrap.sh idempotent registries.yaml + systemctl restart on
re-Connect to pick up updated routing rules
Phase G: every Operation now carries (TenantID, ActorUserID, ActorEmail)
stamped at opStore.Create from the request scope. The bell SSE stream
filters per event against the caller's scope before emitting (closes
the cross-tenant leak — non-super-admin users no longer see other
tenants' ops). Get / Cancel / Stream-one return 404 (not 403) when
the caller can't see the op so existence isn't probable across
tenants. List endpoint uses op.TenantID directly (covers in-flight
ops with no Argo App yet); legacy ops with empty tenant fall back
to the Argo lookup so the upgrade is seamless.
Delete leak: cascade-delete failure used to fail the whole flow,
stranding the Gitea overlay repo + DNS A record. Now: cascade
fails → escalate to ForceDeleteApplication (strip finalizers) →
continue to repo + DNS cleanup. Both fail only when ArgoCD itself
is unreachable. Caught when odoo16v2 left tenant-havari/instance-
odoo16v2 orphaned across the smoke test.
Tests + build green.
The MigrateDrawer's auto-suggest watcher compared form.value.domain
against staged.sourceCode (the *initial* suggestion from the bundle)
to decide whether the domain field had been hand-edited. Result: the
moment the operator typed past the initial suggestion (e.g. typed
'odoo16v' then 'odoo16v2'), every subsequent code edit was treated
as a hand-edit and the domain stuck at the previous suggestion.
Concrete failure caught live: typing 'odoo16v2' as the new code left
domain='odoo16v.tenants.4th.online' (missing the '2'). Argo App
annotation, instance row, and Cloudflare A record all carried the
truncated domain.
Match DeployInstanceDrawer's pattern: track the last value we
auto-suggested in a previousDerivedDomain ref, compare against that
on each watcher fire.
handleListServers's existing tenant filter would still admit a
platform cluster (type='platform') if it had been mis-labeled with
a tenant or if a tenant happened to host an instance there. Belt-
and-braces: explicit reject of any cluster with type='platform'
when the request is tenant-scoped (non-super-admin). The platform
control-plane runs Tower itself + platform-tenant template builds
— it is not a deployment target for customer tenants and surfacing
it in their Server picker breaks the bring-your-own-cluster model.
Caught while smoke-testing MigrateDrawer: a fresh tenant's Server
dropdown defaulted to 'Platform server', risking a customer
deploying their tenant data onto the operator's shared infra by
accident.
The instance row's '<code>.tenants.odoosky.org' was being computed
client-side from the code alone, so a tenant whose domain is
'4th.online' still saw 'odoo16.tenants.odoosky.org' in the list +
the Open button — wrong zone, no cert, scary Firefox warning.
Backend: Argo App now carries an 'odoosky.io/domain' annotation
written at create time from req.Domain (the values.yaml domain),
read back in argoApplicationSummary.Domain. Delete handler reads
the same annotation so DNS cleanup hits the right Cloudflare zone
instead of the platform default.
Frontend: Instance.domain field, used by InstancesView, Vitals,
ActionBar, with a fallback to the legacy pattern for any pre-Phase-F
Argo App that hasn't been backfilled yet.
Backfill for live odoo16: kubectl annotate done out-of-band.
DeployInstanceDrawer + MigrateDrawer were hardcoding
'<code>.tenants.odoosky.org' as the auto-suggested instance domain,
even when the operator's tenant has its own domain set in Settings.
A tenant whose wildcardHost is '*.tenants.4th.online' would still see
the wizard pre-fill 'odoo16.tenants.odoosky.org' in the Domain field —
the suggestion landed in the wrong zone, instance create would fail
DNS write.
Both drawers now fetch /api/tenants/<id>/settings on mount and use
'tenants.<domain>' (with leading '*.' stripped from wildcardHost) as
the suffix, falling back to odoosky.org only if the call errors or
the field is empty.
UpsertCluster was not setting the odoosky.io/tenant-id label on the
ArgoCD cluster Secret. Result: handleListServers tenant filter
attributed every freshly-connected cluster to no-tenant, making the
just-connected box invisible to the operator who registered it.
Threads ownerTenantID from the connect token through to the body of
the POST /api/v1/clusters call. Backfill of any pre-fix cluster
Secret is a one-line kubectl label.
cloudflare_resolver.go reads data['token'] but the writer (and the
test endpoint) stores under data['api_token']. Result: every fresh
tenant's CF resolver returned no token even when one was saved,
killing DNS records view AND any instance lifecycle DNS write.
Caught while smoke-testing the multi-tenant signup flow.
3 bugs caught while smoke-testing the multi-tenant signup → first
deploy flow:
1. /api/me returned capabilities: null when scope.role was nil
(a tenant member viewing the platform tenant context). Frontend's
useAuth.can() does for-of on the array → TypeError. Fix: backend
returns []string{} not nil; frontend defends against null.
2. WelcomeView slug input pattern '[a-z][a-z0-9-]{2,39}' is rejected
by Firefox's HTML5 form validation under the v-flag (treats trailing
'-' as a class-range start). Move hyphen to start of the class.
3. Domain & DNS Save button is genuinely separate from the Cloudflare
token Save (by design — token rotates independently). Documenting
here so the next person doesn't think they saved when they didn't.
Refactor s3Resolver from a single-global-creds reader into a
tenant-scoped factory. Each tenant brings their own S3 endpoint,
region, three named buckets (backups + templates + audit), and
access keys (in Vault at v3/tenants/<id>/s3-credentials).
Touches:
s3.go — s3Resolver becomes factory; tenantS3 wraps
one minio.Client + bucket per tenant
audit.go — events grouped by tenantID per flush, written
to the tenant's audit bucket
backups.go — fleet view fans out one S3 LIST per tenant;
per-instance handlers resolve via Argo App
export/import/migrate — tenant resolved from Argo App label
or scope.TenantID
templates_* — per-template tenant lookup via templateTenantID
(platform tenant for OwnerPlatform manifests)
vitals.go — last-backup probe pulls tenantID before list
Adds AllTenants() to PlatformStore so the templates orphan sweep
can iterate every tenant configured with a templates bucket.
Build: tower:0.61.1 — pushed to registry.odoosky.cloud
Backend rebuild for the route I added in 0.60.2 + ui:0.61.2 cycle —
forgot to rebuild Go binary alongside the chart bump. 0.60.2 binary
didn't carry the handler, hence the 404 on Provision now click.
POST /api/tenants/{id}/settings/s3-buckets/provision uses stored
creds to HeadBucket+MakeBucket the three derived buckets idempotently,
then persists their names to YAML. Surfaces in Settings → Backups
panel as 'Provision now' button next to the bucket list. Lets the
operator create buckets without rotating keys.
handleTestS3Credentials no longer requires bucket names in YAML —
testTenantS3Buckets derives from slug, same as Save's auto-create.
Test stored creds now works whether the buckets have been created
yet or not. Per-bucket result still surfaces 404 vs 403 vs other.
Tenant pastes endpoint + region + access keys. Tower auto-derives
3 bucket names from slug (<slug>-backups, -templates, -audit) and
HeadBucket+MakeBucket each at credential-save time. UI removes the
3 bucket-name inputs entirely; shows what will be created instead.
Removes the 'go to MEGA dashboard, click new bucket × 3' toil.
One credential save = three buckets ready.
Data model: PlatformTenant.S3 = { Endpoint, Region, Buckets: { Backups, Templates, Audit } }
Vault: legacy v3/data/s3{,-templates,-audit} paths wiped (decision in
docs/decisions/0001 path is bring-your-own only; per-tenant only).
UI: 3 bucket fields (Backups / Templates / Audit), single endpoint +
region + credential pair. Test does HeadBucket on each configured
bucket and reports per-bucket pass/fail.
Note: writers (audit/templates/backups handlers) still read from old
paths. Phase F.2 (next) sweeps the ~30 call sites onto a tenant-scoped
s3Factory. Tower compiles + serves API; backups+audit+templates writes
are non-functional until F.2 lands. v3 has no customers, so the
breakage window is tolerable per memory feedback_v3_disposable_no_customers.
Per docs/decisions/0001-platform-fallback-deferred.md, instance DNS
automation no longer silently falls back to v3/platform/cloudflare-token.
Tenants without a configured CF token get a clear error at instance
create instead of pretending to work via shared infrastructure.
The platform Vault entry stays seeded for future revival.
Backend: UpdateTenantSettingsReq → pointer fields. Each card saves
only its own keys without clobbering the other card.
Frontend:
- 'Save Domain & DNS' button inside the Domain & DNS card
- 'Save Backups target' button inside the S3 card
- Wildcard host hidden behind 'Advanced — customize wildcard host'
disclosure. Default is shown as read-only display under Domain so
the operator sees what URLs their instances will land at.
- Removed the global Save settings button (each card now self-saves)
Replace event-based 'touched' flag with computed isCustomWildcard.
Empty wildcard or wildcard==derived → 'auto-derived', auto-fills
on domain change. Different from derived → 'custom', sticks.
Fixes the empty-after-delete trap that kept touched=true forever.
- new GET /api/tenants/{id}/dns/records endpoint lists A+CNAME records
in the tenant's CF zone matching the wildcard pattern (read-only)
- TenantSettingsTab.vue: 'Live DNS records' panel with refresh button
- wildcard host auto-derives from domain (visible value, not placeholder)
- placeholder text now generic *.tenants.example.com
Placeholder text (*.tenants.acme-erp.com) was prescriptive and
indistinguishable from the saved value. Now wildcard auto-derives
from the domain field as a real value (visible, savable, editable).
'reset to default' button surfaces when user customises.
Tenant owners had no nav link to their own tenant settings page.
Adds a 'Tenant' workspace link visible to authenticated members
of any tenant. Super-admins still use /admin/tenants list (which
shows all tenants and lets them switch).
Fixes signup race: verify→/me 401 because activate ran async-reload
and lost the race with the very next /me call. Sync reload eliminates
the window. ~50ms slower per write, much cleaner.
