Self-hosted AIOE
Common problems running self-hosted AIOE, what causes them, and how to fix them.
Problems people meet installing and running self-hosted AIOE, grouped by where they show. Start with the API's log: most of these leave a clear line there at start-up. The lines to expect are listed in Observability.
The API does not start
A required setting is missing. DATABASE_URL is required when STORE=postgres. Once AIOE_BOOTSTRAP_TENANT is set, AIOE_BOOTSTRAP_OIDC_ISSUER, AIOE_BOOTSTRAP_OIDC_AUDIENCE, AIOE_BOOTSTRAP_OIDC_CLIENT_ID and AIOE_BOOTSTRAP_API_SCOPE are required too. With Helm, check that the API's secret exists in the same namespace and holds the key: the chart treats the secret as optional, so a missing secret does not stop the pod from being created. See Environment variables.
The key is set but is not 32 bytes once decoded. Make one with openssl rand -base64 32. On Azure, an empty secretsKey parameter is fine: it means no key. See Platform keys and rotation.
The value is misspelled. Self-hosted AIOE uses STORE=postgres and leaves AIOE_EDITION unset (it defaults to enterprise).
The API checks the database before anything else. Check the host, port, user and password in DATABASE_URL, that the database exists, and that the network allows the API to reach it. Managed PostgreSQL usually needs ?sslmode=require at the end of the connection string. On Azure, the server's firewall admits Azure-internal addresses only.
Discovery
The host name the request arrived at does not match your organisation's domains. A host matches when it equals a domain in AIOE_BOOTSTRAP_DOMAINS or ends with . and that domain. Add your email domains, and the API's host name when it is not under one of them (for a local evaluation, localhost). Restart the API. See Connect your identity provider.
Check the record with dig +short CNAME aioe-discover.<domain> and dig +short TXT aioe-discover.<domain>. A TXT record must read v=aioe1; url=https://<api-host>, with https. Check that the email's domain is in AIOE_BOOTSTRAP_DOMAINS. Until DNS is right, people can choose Enter the platform URL instead in the workbench. See Publish the discovery record.
Signing in to the console
The client registration needs a Single-page application redirect URI equal to the console's address with a trailing slash, for example https://console.aioe.example.com/ (or http://localhost:8080/ for a local evaluation). A Web platform entry does not work for the console.
The API did not accept the token. The usual causes:
- The API registration issues version 1 tokens. Set
requestedAccessTokenVersionto2in its manifest: AIOE expects the issuerhttps://login.microsoftonline.com/<tenant-id>/v2.0. AIOE_BOOTSTRAP_OIDC_AUDIENCEis not the API registration's Application (client) ID.AIOE_BOOTSTRAP_OIDC_ISSUERnames another tenant, or has a typo.
The aioe.auth.failures metric counts each refusal.
The console's image has the API's address built into it. An image built for another deployment calls that deployment instead of yours. Use a console image built for your API's public address. See Install with Helm.
Roles come from the token. Check that the person (or a group they are in) has the admin app role on the API registration's enterprise application, then have them sign out of the console and in again. A request refused for this reason answers 403 naming the role.
The client registration lacks the Microsoft Graph delegated permissions User.ReadBasic.All and GroupMember.Read.All, or they have no admin consent. See Connect your identity provider.
Enrolling workbenches
Codes expire ten minutes after the workbench shows them, and each works once. Start again in the workbench to get a new code. Check also that the console and the workbench are using the same AIOE.
The workbench collects its token on its next poll, every few seconds. If it never does, the workbench cannot reach the API: check that it can open https://<api-host>/healthz, and any proxy between them.
Remote control and the relay
Something between the workbench and the API is buffering or cutting the relay's stream. Turn off response buffering and allow reads of up to an hour on your proxy or ingress (the Helm chart's annotations do this for ingress-nginx).
You are running more than one API replica without Redis, and requests reach a replica that does not hold the workbench's connection. Each replica's log says relay broker: in-process in that case. Set REDIS_URL on every replica. See Redis for more than one replica.
Harmless with one replica. With more, set the same AIOE_SIGNING_KEY on all of them, or requests signed by one replica fail checks against another's key.
The overlay
Look at the node with aioe-mesh status. Without outbound UDP, nodes need a mesh relay: check AIOE_MESH_RELAYS on the API and that the relay answers on its /healthz. The console's Mesh page shows, for each node, whether its traffic is direct or relayed.
Put a Basic user name and password in the proxy address (HTTPS_PROXY=http://user:password@proxy:8080), or allow the platform's host through the proxy without authentication. Windows integrated proxy sign-in is not supported.
Your proxy inspects TLS. Give the agent the proxy's CA certificate with --ca-file or AIOE_CA_FILE.
The platform rejected its token: the workbench was revoked, or its token expired. Sign the workbench in to the organisation again.
Features that say they are not set up
AIOE_SECRETS_KEY is not set. Set it and restart the API. The same key enables linked source-control accounts and workbench backups. See Platform keys and rotation.
AIOE_ENTRA_CLIENT_SECRET is not set, so Connect through Entra is unavailable and people can only link with a personal token. See Let AIOE act for people in Azure DevOps.
Nothing is published yet: a workbench keeps its own policy file until a policy is published in Policies. A policy that applies to chosen projects reaches only those projects; publish one for the whole organisation as the default. See Policy.
Azure
The deployment ran without apiCertificateName and consoleCertificateName, which replaces the custom-domain bindings with none. Bind the host names again, put the certificate names in your parameters file, and carry them on every deployment. See Install on Azure.
Its image is not in the registry with the tag you deployed (imageTag, and meshImageTag for the overlay apps). Check with az acr repository show-tags -n <registry> --repository aioe-api.
Still stuck? Contact Mojo Up support with the API's start-up log lines and the version from /healthz.