Company OS Desktop local computer control

Shell note (ADR 0004, 2026-09-16): the GPUiX port is abandoned. The status panel described below was built in that shell and does not exist in the canonical Zeron company-os-desktop yet; the execution path, the engine projection (crates/engine/src/company_surfaces/desktop_control.rs), the computer_control_cli.rs entry point, the helper, and the installer steps are shell-independent and remain current.

The GPUiX desktop had a Computer control panel. It reads the authenticated Company OS host status, including the allowed apps, Accessibility permission, Screen Recording permission, and local stop state. The panel does not execute native input or manufacture a run grant.

The execution path is:

Company OS admitted provider run
  → separate company-os-desktop MCP process
  → Company OS desktop observe/computer HTTP routes
  → company-bound local adapter
  → pinned macOS helper (Accessibility + ScreenCaptureKit)
  → observed result + Company OS settlement receipt

The native helper requires macOS 14 or newer. It can inspect running allowed apps, capture an app window, focus an app or accessibility node, click, type, scroll, set a value, and send shortcuts such as Meta+Q / Command+Q. App launching is not implemented: open the target app first.

Local setup

Build the helper and MCP service from the repository root:

bash services/company-os-desktop-control/build.sh
npm --prefix services/company-os-desktop-mcp-server run build

The API host must use a rebuilt hsm_console containing the adapter registration. Configure its launch environment with the following values; replace the company UUID with the company bound to the desktop. These settings contain no credentials.

export HSM_COMPANY_DESKTOP_HELPER_PATH="$PWD/services/company-os-desktop-control/build/company-os-desktop-control"
export HSM_COMPANY_DESKTOP_COMPANY_ID="<company UUID>"
export HSM_COMPANY_DESKTOP_PRINCIPAL="<exact authenticated Company OS actor>"
export HSM_COMPANY_DESKTOP_ALLOWED_BUNDLE_IDS="com.apple.TextEdit"
export HSM_COMPANY_DESKTOP_STOP_FILE="$PWD/.local/company-os-computer-control.stop"

The adapter is absent when these settings are absent; partial configuration is an error. A desktop run requests this capability only when the desktop host's launch environment includes COMPANY_OS_DESKTOP_COMPUTER_CONTROL=1. The backend still requires a configured adapter for that same company and the exact locally pinned principal. Use the non-secret actor identity returned by Company OS authentication, not a token or a display name. New signed grants then contain exactly the two desktop MCP tools and the native desktop_control ceiling. Existing runs are not silently upgraded. Child runs never inherit desktop authority. Every desktop call must authenticate as the same pinned operator named in its signed grant; company membership alone grants no control.

The provider broker uses the managed local API origin http://127.0.0.1:3847. It refuses a provider-supplied alternate origin before reconstructing host credentials. Custom-port and remote desktop hosts are not supported by this bootstrap. Standalone explicitly configured desktop MCP clients retain their existing configuration contract.

The default Company OS/OpenRouter bridge, standard ACP providers, and the Pi MCP bridge route desktop tools to a second MCP process. The ordinary Company OS MCP process excludes these tools. Provider discovery alone is not proof of a working native adapter.

Image-capable ACP clients receive screenshots as MCP image blocks. The current OpenRouter bridge is text-only: it receives the accessibility tree and an explicit warning that the image is unavailable. It must use semantic nodes and keyboard actions rather than guessing coordinates. Multimodal OpenRouter tool-result transport is not part of this implementation.

Permissions and stop

Installed app update

The local installer installs the canonical Zeron company-os-desktop shell. To bundle the helper and enable new desktop runs, build both Rust binaries and install with the operator identity resolved by the authenticated operator/usage API (never infer it from a display name):

# Root runtime; desktop is a separate Cargo workspace.
bash scripts/cargo-reliable.sh build --bin hsm_console
(cd external/company-os-desktop && cargo build -p company-os-desktop)
COMPANY_OS_DESKTOP_COMPUTER_CONTROL=1 \
HSM_COMPANY_DESKTOP_COMPANY_ID='<company UUID>' \
HSM_COMPANY_DESKTOP_PRINCIPAL='<authenticated principal>' \
bash scripts/install-company-os-desktop-app.sh --update

--update stages and signs the app before replacing it, preserves timestamped app/runtime rollback copies, and restores those copies if publication fails. Without --update, an existing app is not overwritten. The helper is installed at Company OS.app/Contents/Helpers/company-os-desktop-control. Only six non-secret installation settings enter LSEnvironment; credentials remain in their protected host files. The installed allowlist is initially TextEdit only.

Quit the old app and its managed runtime, then launch the new app through LaunchServices/Finder. A running backend retains its old environment until it restarts. Request permissions through the installed app identity:

open -n "$HOME/Applications/Company OS.app" --args computer-control-permissions

This explicit setup command only requests macOS permissions; it does not start the engine or perform native input. Its health result is written to ~/.company-os-desktop/logs/computer-control-permissions.json. The operator must grant the requested permissions in System Settings. Normal app launches and status checks do not prompt.

Use the Computer control panel to inspect status. macOS must grant Accessibility and Screen Recording to the helper/host identity shown in System Settings. The helper does not prompt during health checks or observation. To explicitly request the system permission dialogs after configuring a stable helper path:

printf '%s\n' '{"schema":"hsm.company_os.desktop_native.v1","operation":"request_permissions","allowed_bundle_ids":["com.apple.TextEdit"]}' \
  | "$HSM_COMPANY_DESKTOP_HELPER_PATH"

Only the operator can grant those macOS permissions. Development binary replacement may require granting them again; production packaging needs a stable path and code signature.

To stop further native control, create the configured stop file:

mkdir -p "$(dirname "$HSM_COMPANY_DESKTOP_STOP_FILE")"
touch "$HSM_COMPANY_DESKTOP_STOP_FILE"

The API adapter and native helper check this file before effects. An event already submitted to macOS cannot be recalled. Remove the file only when deliberately resuming control. Existing Company OS safe mode remains an additional authority fence. The former GPUiX panel displayed status and setup guidance; it did not offer a permission-grant, enable, or stop mutation button.

First end-to-end check

node scripts/company-os-desktop-live-smoke.mjs checks installed authenticated status without creating a task or run. After opening a new blank TextEdit document, --execute --provider <selected-provider> --model <selected-model> runs a direct API proof through the signed grant, observation, action, and settlement routes. It does not call a model or establish provider-driven chat readiness. It refuses missing permissions and ambiguous/nonempty documents, verifies the exact proof sentence, and saves a redacted receipt artifact. Its server-created task remains as an evidence trail.

The trusted broker and binding wrappers bootstrap their local build tools with Rust 1.97.0 (already installed), debug information disabled, incremental builds disabled, and the repository target directory. This restores required host paths after the harness clears its environment and avoids a second compiler cache on first use. It preserves typed command admission and grant validation; it never changes the user's default toolchain or downloads a missing compiler.

Open a new empty TextEdit document. Start a new explicitly enabled desktop run and ask it to observe com.apple.TextEdit, focus the observed root node if needed, observe again, type a short sentence, and verify the returned text/image. Do not use an existing document as a test fixture.

Each action needs the server-issued frame or snapshot identity. Multiple allowed apps require an explicit target. Native snapshots expire after 30 seconds; changed apps/windows and mismatched run identities are refused. Unsupported postconditions are rejected before input. If input was submitted but a fresh result cannot be captured, the action remains unknown and requires reconciliation rather than an automatic retry.

Verification and current evidence

• Native Swift build and behavioral tests passed, including a red/green Meta alias regression proof.
• Desktop MCP build/tests passed for adapter discovery, origin binding, selected app forwarding, and image delivery.
• Local adapter behavior tests passed through a focused production-source harness; its actual subprocess health probe reached the Swift helper.
• Signed-grant tests passed, including default denial and non-inheritance by all supported child roles.
• The (since abandoned) GPUiX panel passed typecheck, focused tests, and a native fixture smoke.
• The OpenRouter two-process bridge passed four focused external-process fixture tests, including the text-only image warning.
• Full Company OS library and hsm_console binary cargo check passed after reclaiming generated build caches. Desktop harness and engine test targets also passed cargo check --tests; those test binaries were not linked or executed. Full root test linking requires more free disk space on this host.

At initial implementation time, the native health probe reported both Accessibility and Screen Recording false; installation had not occurred.

Local rollout, September 8, 2026

• Rebuilt hsm_console and the full company-os-desktop executable successfully with Rust 1.97.0. The runtime build reported an existing unused function and a linker compact-unwind size warning; neither prevented binary creation.
• Built and prewarmed the typed command runner, credential resolver, and provider broker CLI. The protected MCP authentication probe passed with bearer, principal, and company bound, without printing credentials.
• Installed and verified the signed app and helper at ~/Applications/Company OS.app; installed the managed runtime at ~/.company-os-desktop/runtime-target/debug/hsm_console.
• Preserved the previous app as ~/Applications/Company OS.rollback-20260908T090020Z-14544.app and runtime as ~/.company-os-desktop/runtime-target/debug/hsm_console.rollback-20260908T090020Z-14544.
• Restarted the managed stack. Authenticated live status confirmed configured=true, authorized_for_principal=true, and stopped=false for the selected company and exact owner principal.
• Invoked the installed app's computer-control-permissions command through LaunchServices. Its recorded health and the restarted backend still reported Accessibility and Screen Recording false. The operator must grant both.
• Native helper tests, installer signing/update/rollback integration tests, cleared-environment command bootstrap tests, syntax checks, and scoped drift checks passed. Shared-tree doctor reported zero fatal findings and five existing warnings.

The governed TextEdit action test and provider-driven chat proof remain not run, blocked by the macOS permissions. No native input or app screenshot was captured during this rollout. Full root unit-test linking was not attempted on this low-disk host. The Computer control panel has no Zeron equivalent yet; this local update preserves the existing full desktop shell.