Victor Agnew
Staff AI Platform Engineer · Vancouver, BC
Boring deploys. work/ Safe AI. ai-lab/ Happy teams. virtual-office.md
An assistant inside the ArborXR portal that answers questions about a customer's device fleet, built on the company's first production MCP server.
ArborXR · Staff Platform Engineer
Customers manage their device fleets in the ArborXR portal. AI agents can answer questions about that data through an MCP server. The starting point was a 1:1 mapping of the OpenAPI spec: more than 160 endpoints, each its own tool.
MCP only helps people who can wire it up. The assistant was aimed at non-technical users, and at regulated, education and enterprise IT customers who can't sign up for third-party LLMs or wire up MCP themselves. They needed answers inside the portal, with no setup.
I built the company's first production MCP server, on Laravel MCP. I replaced the 1:1 mapping with 23 hand-curated tools, so agents reason better. Every tool is read-only. Every call runs as one user in one organization, resolved from that user's access token.
Every vendor reference architecture I checked runs an assistant's tools in-process, under the caller's own identity, with no separate identity service. So the assistant lives inside the portal and calls the MCP server's tool classes directly, with no HTTP hop. Its tools run with the signed-in user's own permissions. Each request builds its own tool list from those permissions and never shares it with another request.
Answers stream in as they're generated. The portal runs on FrankenPHP, where Octane sets a 30-second execution limit and FrankenPHP re-arms it for every request as wall-clock time. An idle stream and a busy loop both died at 30 seconds, and a turn with tool calls can run longer. A bare set_time_limit(0) leaks: every later request on that worker thread runs unlimited until the worker recycles. So each turn lifts the limit for itself and restores the previous value in a finally block. Lifted streams stayed open for three minutes through the network path, so the cut was PHP, not the ingress.
The text still arrived in bursts, because Guzzle read Claude's stream in blocks. In a probe, 67 text deltas reached PHP at two distinct times. After I swapped in Symfony's HTTP client, 72 deltas arrived at 29 distinct times, and the first came at 1.5 seconds instead of 4.0.
The portal's frontend is Blade, Livewire and Alpine, with no React or Vue. The chat panel is a Preact island inside a Livewire component, with TanStack AI managing the chat. The first version used React. After review I moved it to Preact and TanStack AI, which cut the bundle from 154 kB to 50 kB gzipped. Mingle hosts the island, but its JavaScript only supports React and Vue, so I wrote a typed Preact bridge.
Each turn is one POST, and its response is the stream: AG-UI events over server-sent events. A run opens with RUN_STARTED, streams text and tool calls, and ends with RUN_FINISHED or RUN_ERROR. Laravel's eventStream() ends with a line that isn't JSON, and the client throws on it, so the endpoint writes raw frames. The client also never reads the body of an error response, so a turn's errors arrive inside the stream as RUN_ERROR events with a code.
Pluggable guardrails screen input for jailbreaks and abuse. The guard is a seam with interchangeable drivers. The screening driver sends each message to TypeSafe's Jev model before the turn runs, and asks three questions: is this a jailbreak attempt, a harmful request, or abuse? Switching drivers is configuration; the rest of the flow doesn't change.
Claude can also refuse. A refusal comes back as a normal response, and Claude's documentation says the conversation keeps refusing unless the refused turn is removed. At first the refused turn was stored and replayed, which poisoned the conversation. Now the whole turn is deleted, and the user is told the assistant can't help with that request.
Victor Agnew
Staff AI Platform Engineer · Vancouver, BC
Boring deploys. work/ Safe AI. ai-lab/ Happy teams. virtual-office.md
I've spent 10+ years building software and 7+ building the platforms other engineers ship on: from a bare-metal Kubernetes cluster in the data lab of one of Canada's biggest insurers, to GitOps, dynamic secrets and AI agents that hold zero credentials. At ArborXR I build the platform and champion our AI work.
virtual-office.md Virtual office three.js · Cloudflare Workers · LiveKit Built in a day; 26 days later, 1,600+ commits and 12 coworkers committing. Mapzen
Canada terrain data contains information licensed under the Open Government Licence – Canada;
United States 3DEP (formerly NED) and global GMTED2010 and SRTM terrain data courtesy of the U.S. Geological Survey.
/ Impact
/ Work
| Name | Company | Role | Dates |
|---|---|---|---|
| arborxr/ | ArborXR | Staff Platform Engineer | Sept 2022 – present |
| intact/ | Intact Insurance | DevOps Developer | Jan 2019 – Sept 2022 |
| summit-tech/ | Summit Tech | Backend Developer | May 2015 – Jan 2019 |
Staff Platform Engineer · Sept 2022 – present
DevOps Developer · Jan 2019 – Sept 2022
Backend Developer · May 2015 – Jan 2019
/ Case studies
virtual-office.md
A 3D voxel office in the browser, built in a day, that the team made its own.
ai-assistant.md
An assistant inside the ArborXR portal that answers questions about a customer's device fleet, built on the company's first production MCP server.
zero-credential-agents.md
Agents on the Claude Agent SDK that hold no credentials, where every call leaves through one gateway that checks a short-lived run token and injects the real credentials at the edge.
dynamic-credentials.md
Every service on short-lived credentials from Vault, and every production session requested, approved and audited, behind one command.
review-environments.md
Self-serve, full-stack review environments for any feature branch at about $9 a day, which grew into SSO-gated hosting for AI-generated reports.
multi-arch-builds.md
Container images for amd64 and arm64, built in parallel on matching GKE node pools and merged into one, with no privileged containers.
A 3D voxel office in the browser, built in a day, that the team made its own.
Side project for my team
Remote teams lose the hallway. I wanted a place to hang out between meetings and on Friday afternoons, where you can walk up to someone and just talk.
A 3D voxel office in the browser. You walk around as a character you design (human, animal or a special skin), talk to whoever is near you with proximity voice, share your screen to a TV in a private pod, and play: pool, ping pong, darts, mini-golf and racing, with a Hall of Fame for the best scores. There's a koi pond to fish in, too.
I planned the work, then fanned it out to parallel AI coding agents, each in its own git worktree. Kickoff to production took about an hour and forty minutes. By the end of day one, 643 tests passed, and a second batch had shipped mini-golf, a Hall of Fame and flag sweaters.
With two people walking around, remote avatars stuttered. A simulation of the real interpolation code over a modelled network found two causes: the playout buffer shrank by the one-way latency, and the clients' sends beat against the server's snapshots. Past about 67 ms of round-trip time, remote avatars froze between snapshots, on 38% of frames at 120 ms and 81% at 250 ms. An arrival-aligned playout clock brought stalls under 1% from 10 to 250 ms.
Rendering was never the bottleneck: measured on an M3 Pro, frames held 60 fps with 0.8–1.6 ms of main-thread work each.
Twenty-six days after the first commit, the repo had more than 1,600 commits, and 12 coworkers had committed to it, adding features and fixes.
Agents on the Claude Agent SDK that hold no credentials, where every call leaves through one gateway that checks a short-lived run token and injects the real credentials at the edge.
ArborXR · Staff Platform Engineer
Our agents read content we don't control. A scheduled one refreshes an internal dashboard from several data sources. A prompt injection in any of them could make the agent leak its credentials, so the agent pod must hold none.
All egress therefore goes through one policy enforcement point, which injects upstream credentials at the edge. That needs least privilege per run, checked offline at the gateway with no callback to a control plane. Runs in flight must also survive a Vault or control-plane outage.
Mesh identity alone is per service account, not per run. Our Istio mesh already gives pods SPIFFE identities, but Envoy Gateway can't authorize on them.
SPIRE JWT-SVIDs can carry custom claims, but SPIRE takes no per-request input from the caller. It can't put a run's connections in the token.
Off-the-shelf OAuth issuers (Keycloak, Hydra, Zitadel, dex) can't mint caller-supplied, per-request claims under token exchange.
A credential sidecar per pod was lighter, but one shared gateway gave a single place for credential injection, rotation and token metering.
A custom token service is what I built: Vault-Transit-signed ES256 run tokens with a connections allow-list, validated by Envoy's jwt_authn filter against a local JWKS.
I built zero-credential agents on the Claude Agent SDK.
The runtime is one config-driven image on the Claude Agent SDK, behind a harness interface so another agent SDK can be added without a rewrite. Each run is a Kubernetes Job. Data sources are MCP servers, which we call connections. The pod gets a service account with no RBAC.
Every model and tool call leaves through Envoy AI Gateway. It validates the run token, strips it and injects the real upstream credential. Model calls go to Claude, with a second route as failover. Each MCP connection has its own route, which denies any token whose connections claim doesn't name it. BigQuery needs no key: its MCP server runs in the cluster under Workload Identity. The gateway also labels token usage per run.
Each run gets a short-lived ES256 JWT. Its subject is the run ID, its audience is the gateway, and its connections claim lists what the run may reach. Vault Transit signs it, so the key never leaves Vault. The gateway validates it against a local JWKS and never calls Vault or the registry per request. When the registry launches a run, the token expires five minutes after the job's deadline.
If Vault is down, no new run starts, but runs in flight finish. There's no fallback signer: an identity issuer has to fail closed.
The registry is the control plane, a Go service. It defines agents and their versions, each pinned to an image digest, and grants each agent connections with an exact list of tools. Its token service writes every mint to a ledger before the token goes out. It launches runs as Kubernetes Jobs and never proxies agent traffic.
Later I added RFC 8693 token exchange, for runs that act on someone's behalf. A client signs a short-lived subject token, and the token service verifies it offline against that client's keys only, with no shared secret. Only clients registered to assert users can name a person. The run token then carries the person, their organisation and an actor chain for the audit trail. The gateway forwards the person and organisation upstream as headers, and strips inbound copies.
Envoy AI Gateway writes a 60-second request timeout into every route it generates unless you set one, and that caps each retry too. Under it, a refresh run took 55 minutes and cost $3.49. With 600 seconds on every rule, runs took 6 to 9 minutes and cost $1.40 to $2.30.
Every service on short-lived credentials from Vault, and every production session requested, approved and audited, behind one command.
ArborXR · Staff Platform Engineer
Credentials were rotated every six months. Fixing that meant answering four questions at once: where do secrets live, how do they rotate on their own, how do we stop credential files from lying around, and how does all of it work in Kubernetes without a person in the loop?
Sealed Secrets keeps encrypted secrets next to the infrastructure code and is easy to set up. But encrypted values are hard to read, the encryption key is created and stored by hand (lose it and everything is lost), and every change means building a new SealedSecret by hand.
Other common answers: Mozilla SOPS, AWS and GCP Secret Manager, Infisical, Bitwarden, 1Password.
HashiCorp Vault is secure by default, supports many auth methods (OIDC, JWT, LDAP, GitHub, tokens), is production-ready, and has a simple UI and a large ecosystem. Its cost is that it's complicated to set up and run, so most of the work below goes into automating that away.
Vault runs in-cluster in high-availability mode with Raft storage, auto-unseals through the cloud KMS, and uses workload identity to reach cloud services. Nobody runs the setup by hand: a Kubernetes Job, part of the infrastructure rollout, waits for Vault to come up, initialises it, stores the recovery keys in Secret Manager, enables the secret engines, policies and roles, and finally revokes the root token.
The Vault Secrets Operator (VSO) syncs secrets from Vault into native Kubernetes Secrets, detects and reconciles drift, and restarts the deployments that use a secret when its value changes. It authenticates with Kubernetes service-account tokens, rotates its own tokens, and each role is scoped to specific namespaces and policies.
Static secrets were only the first step. Vault's database engine creates a database user per lease from a creation statement and removes it with a revocation statement. Services receive credentials that rotate every 24 hours with zero downtime: VSO renews the lease, writes the new credentials and restarts the deployment.
Developers reach databases and clusters through HashiCorp Boundary: a controller with Google sign-in, and a worker inside each environment's network. Vault brokers short-lived credentials into each session, whether that's a database user or a Kubernetes service account for exec-ing into pods.
Production is protected. You give a reason, the request lands in Slack, and an approver clicks approve. The approval creates a temporary Boundary role that lets that one person authorize a session on that one target, then deletes the role seconds later. The session keeps running; the permission doesn't.
A small CLI wraps all of it: Google sign-in, pick the environment and the target, wait for approval when it's production, open the tunnel, and launch the database client with the brokered credentials already filled in.
Self-serve, full-stack review environments for any feature branch at about $9 a day, which grew into SSO-gated hosting for AI-generated reports.
ArborXR · Staff Platform Engineer
The product stack includes a web app and API, an MQTT broker for devices, a docs site, Vault, Boundary, object storage, Postgres and Redis. Testing a feature branch against all of it takes a full environment. An environment built like production has its own cluster, network and managed database. In June 2026 I measured two of them at $139 and $178 a day while they ran.
The internal developer platform started at KubeCon 2023, as an idea a colleague and I came up with. I was part of all of its initial planning, and much of the design was mine. It gives every feature branch a self-serve, ephemeral review environment that runs the full product stack in a vCluster, via Pulumi and FluxCD. A developer picks a branch in the web UI, or from an AI tool over MCP.
Each environment is a vCluster: a virtual Kubernetes cluster with its own control plane, on one shared host cluster. It runs Postgres and Redis in-cluster and RustFS for object storage. Stateless services, such as the web app, workers and docs site, run on spot nodes. Each vCluster's control plane stays on on-demand nodes, because preempting it would disrupt the whole environment, and so do the databases and the MQTT broker. An environment can also run a review copy of the platform itself.
The Pulumi program that builds production also builds every review environment. In review mode it creates a vCluster on the shared host cluster; otherwise it creates a dedicated GKE cluster. That second flavour, the integration environment, is built like production: its own cluster and VPC, AlloyDB and Cloud Storage.
Creating, updating or destroying an environment queues a Pulumi run. The platform's worker starts a Kubernetes Job for it and is free again in seconds. The Job runs Pulumi, records the result and retries up to three times on failure. It runs outside the platform's own release, so a platform deploy doesn't kill it mid-run. A reconciler checks every five minutes for runs whose pod died before it could report.
Pulumi installs FluxCD inside each environment. FluxCD watches the branch's own image repository, deploys the newest version and reports the deploy back to the platform through a webhook.
Deploy notifications go to Slack and Linear. Slack gets a message when provisioning or destruction starts, finishes or fails. When a new version deploys, the platform comments on the Linear issue named in the branch or its commits.
Environments tear down automatically after the merge request closes. By default, a daily job destroys every environment whose branch has no open merge request left, after a dry run posts the list to Slack. If GitLab can't be reached, the environment stays. The same cleanup covers integration environments, so their dedicated clusters go too.
I conceived and led the platform's extension into an internal knowledge-sharing product: SSO-gated artifact hosting with versioning, inline comments and review workflows, exposed through an MCP server. I wrote its agent skill, so internal docs stop being shared as Markdown files, screenshots or text pasted into Slack.
An AI tool uploads a static HTML bundle over MCP and gets back a link that only the company's Google Workspace members can open. It's visible company-wide by default, or private behind a revocable share link, or a draft. Every publish keeps the previous version, which can be restored. Viewers leave Google-Docs-style comments that update live, and assigned reviewers approve or reject it.
I built its scheduled refresh. On a schedule, an agent run with no credentials of its own regenerates an artifact's data.json, so the page stays current. It reaches data through the same gateway as my zero-credential agents.
Container images for amd64 and arm64, built in parallel on matching GKE node pools and merged into one, with no privileged containers.
ArborXR · Staff Platform Engineer
We needed container images for both amd64 and arm64, and security compliance meant no privileged containers. That ruled out docker buildx and buildah, which both depend on them. Kaniko builds without privileges, but only for the architecture of the machine it runs on. Starting from a multi-arch base image doesn't help: Kaniko's output is still single-architecture.
So I worked with that limit instead of against it: build each architecture on matching hardware, then merge the results in the registry.
I added an arm64 node pool to the GKE cluster with Pulumi: t2a-standard-8 machines, autoscaling from 1 to 20 nodes. GKE taints arm64 nodes automatically, so amd64 workloads stay off them.
One GitLab Runner serves both architectures. Its node_selector_overwrite_allowed option, undocumented at the time, lets each job override the node selector through a variable, so every build lands on nodes of its own architecture.
The helper image had to be multi-arch too. The arm64 helper images on Docker Hub were single-architecture and carried the wrong OS/ARCH label, and an alternative helper image broke the pipelines immediately. So I built a multi-arch helper image of my own and pointed the runner at it. The runner runs with privileged mode off.
The amd64 and arm64 Kaniko builds run in parallel in a stageless pipeline. Once both finish, a final job runs manifest-tool, which reads their manifests remotely from the registry and combines them into one multi-arch image.
/ AI lab
/ Writing
· ArborXR blog
amd64 and arm64 images without privileged containers. Kaniko builds each architecture on a matching GKE node pool, one GitLab Runner schedules jobs onto both through a node-selector override, and manifest-tool merges the results into a single multi-arch image.
Staff Platform Engineer · Vancouver, BC
10+ years building software, 7+ in platform engineering
Staff Platform Engineer · Sept 2022 – present
DevOps Developer · Jan 2019 – Sept 2022
Backend Developer · May 2015 – Jan 2019
École de technologie supérieure (ÉTS), Montréal
Bachelor of Engineering, Software Engineering, Dec 2018
Developer’s Journal: Building Unprivileged Multi-Arch Images with Kaniko and Gitlab CI (opens in a new tab) — ArborXR blog, April 6, 2023
Bouldering, pickleball, cycling, hiking, kayaking, Formula One, reading, board games, video games.
/ Contact
vicos:~$ help
help list commands
whoami print the current user
ls list apps
open <app> open an app, e.g. open work
cat <file> print a file, e.g. cat README.md
history show command history
kubectl get experience list roles
sudo hire vic request access
clear clear the screen
exit close the terminal
3 files changed, 7 insertions(+), 3 deletions(-)
