# 4minds — Helm-Native Install Guide

Deploy the 4minds platform to **your own Kubernetes cluster** with a single
`helm install`. The umbrella chart carries every service and datastore, and its
own hooks do all in-cluster orchestration (OpenBao init/unseal, DB migrations,
Kafka topics) — **no external installer binary is required.**

You configure the deployment through one values file: copy
[`values-customer-template.yaml`](./values-customer-template.yaml) to
`my-values.yaml` and fill in your hostname, inference endpoints, and options.

> The prerequisite commands below use **Amazon EKS** as the worked example
> (`eksctl`, EBS CSI, IRSA). On another Kubernetes distribution, substitute the
> equivalent steps — a default StorageClass, a CSI driver, an ingress
> controller, and (for KMS auto-unseal) a cloud KMS key + workload identity.

---

## What you'll do

1. **Prerequisites (Steps 1–7)** — prepare the cluster and platform plumbing the
   chart does *not* create (cluster/OIDC, storage, ingress, namespace, TLS,
   image pull, and — only for production KMS unseal — a KMS key + IAM role).
2. **Configure** — copy the template to `my-values.yaml` and fill it in.
3. **Install** — one `helm install` with your values.
4. **Verify** — confirm pods are Running and the UI answers.

Sections after that cover **upgrades**, **uninstall**, and **troubleshooting**.

---

## Before you start

**Required CLI tools:** `aws`, `kubectl`, `helm` (v3.8+, OCI support), `eksctl`
(for the EKS example steps), and `openssl` (only if you want a self-signed test
cert). Make sure your `aws` CLI is authenticated to the account that owns the
cluster.

**Set these environment variables** — every command below references them:

```bash
export CLUSTER=<your-cluster>          # e.g. 4minds-prod
export REGION=<your-region>            # e.g. us-east-1
export HOSTNAME_FQDN=<your-host>       # e.g. 4minds.your-company.com
export NAMESPACE=4minds
export ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
```

Then confirm none are empty:

```bash
echo "$CLUSTER $REGION $HOSTNAME_FQDN $NAMESPACE $ACCOUNT"
```

> **Copy-paste tip.** If a multi-line command (with a trailing `\`) breaks in
> your shell, paste it as a single line instead.

---

## Prerequisites

### Step 1 — EKS cluster with OIDC

**What this does:** creates (or reuses) the Kubernetes cluster. `--with-oidc` is
the only hard requirement — IRSA (used for KMS auto-unseal and the EBS driver)
needs it.

Skip the `create cluster` command if you already have a cluster; use your own
version / instance type / node count.

```bash
export K8S_VERSION=<your-k8s-version>      # e.g. 1.31
export NODE_TYPE=<your-instance-type>      # e.g. m5.2xlarge
export NODE_COUNT=<your-node-count>        # e.g. 3

eksctl create cluster --name "$CLUSTER" --region "$REGION" \
  --version "$K8S_VERSION" --node-type "$NODE_TYPE" --nodes "$NODE_COUNT" --with-oidc

aws eks update-kubeconfig --name "$CLUSTER" --region "$REGION"
kubectl get nodes                          # all should be Ready
```

**If the cluster already exists,** just make sure the OIDC provider is
associated:

```bash
eksctl utils associate-iam-oidc-provider --cluster "$CLUSTER" --region "$REGION" --approve
```

> **Sizing (guidance, not a requirement).** The full platform runs many services
> plus stateful datastores. A reasonable starting point for the **application
> tier** is **3 × `m5.2xlarge`** (8 vCPU / 32 GiB each); scale to your workload.
>
> **Inference is separate.** This chart does NOT run models — it points at
> OpenAI-compatible endpoints you set in `my-values.yaml` (`llm.*`,
> `mlai.embedding`, `symi-gateway.config`, `wren-ai`). Those can be a managed
> service or GPU nodes in this same cluster (add a `g5`/`p4` node group and point
> the endpoints at the in-cluster services).

### Step 2 — Default StorageClass + EBS CSI driver

**What this does:** gives stateful services (PostgreSQL, Redis, Kafka, MinIO,
Qdrant, OpenBao, mlai) a default StorageClass backed by a working CSI driver.
OpenBao persists its vault data on a 2Gi PVC, so this is required **even for
`seal.mode: lab`** — without persistence, a pod restart re-initializes the vault
and drops the seeded secrets.

```bash
# 2a. Mark gp2 (or gp3) as the default StorageClass.
kubectl annotate storageclass gp2 \
  storageclass.kubernetes.io/is-default-class=true --overwrite
kubectl get storageclass                   # exactly one shows "(default)"

# 2b. Install the EBS CSI driver (IRSA role + managed addon).
eksctl create iamserviceaccount --name ebs-csi-controller-sa --namespace kube-system \
  --cluster "$CLUSTER" --region "$REGION" --role-name "${CLUSTER}-ebs-csi" \
  --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
  --approve --role-only
eksctl create addon --name aws-ebs-csi-driver --cluster "$CLUSTER" --region "$REGION" \
  --service-account-role-arn "arn:aws:iam::${ACCOUNT}:role/${CLUSTER}-ebs-csi" --force
kubectl -n kube-system rollout status deploy/ebs-csi-controller --timeout=180s
```

### Step 3 — ingress-nginx controller

**What this does:** installs the NGINX ingress controller. The chart creates an
`Ingress` of class `nginx`; the controller itself is a prerequisite.

```bash
# 3a. Install the controller (as a LoadBalancer).
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx --create-namespace \
  --set controller.service.type=LoadBalancer
kubectl -n ingress-nginx rollout status deploy/ingress-nginx-controller --timeout=180s

# 3b. Read the load-balancer hostname, then point your DNS record for
#     $HOSTNAME_FQDN at it (CNAME). For a quick local test you can instead map
#     the LB's IP to $HOSTNAME_FQDN in /etc/hosts.
kubectl -n ingress-nginx get svc ingress-nginx-controller \
  -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'; echo
```

### Step 4 — Namespace

**What this does:** creates the namespace everything installs into.

```bash
kubectl create namespace "$NAMESPACE"
```

### Step 5 — TLS secret for your hostname

**What this does:** provides the certificate the Ingress uses to terminate TLS.
The Ingress reads the secret named by `frontend-backend.ingress.tlsSecretName`
(default `frontend-tls`). The host and TLS entry derive from `global.hostname`
automatically, so you only create the secret here.

**Option A — you already have a cert** (ACM-issued, Let's Encrypt, etc.):

```bash
kubectl -n "$NAMESPACE" create secret tls frontend-tls \
  --cert=/path/to/tls.crt --key=/path/to/tls.key
```

**Option B — self-signed** (testing/PoC only; browsers will warn):

```bash
openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
  -keyout tls.key -out tls.crt \
  -subj "/CN=$HOSTNAME_FQDN" -addext "subjectAltName=DNS:$HOSTNAME_FQDN"
kubectl -n "$NAMESPACE" create secret tls frontend-tls --cert=tls.crt --key=tls.key
```

> Using a different secret name? Set `frontend-backend.ingress.tlsSecretName`
> in `my-values.yaml` to match.

### Step 6 — Image pull (usually nothing to do)

**What this does:** lets the cluster pull the images. On AWS Marketplace the
images live in the Marketplace ECR and your EKS **node IAM role** pulls them
automatically (attach `AmazonEC2ContainerRegistryReadOnly` to the node role if
it isn't already). In that case leave `global.imagePullSecrets: []`.

Only if you mirror the images into your **own private registry** do you create a
pull secret and list its name under `global.imagePullSecrets`.

### Step 7 — OpenBao KMS auto-unseal (production only)

**What this does:** sets up AWS KMS + an IRSA role so OpenBao auto-unseals
without in-cluster keys. **Skip this entire step if you use `seal.mode: lab`**
(Shamir keys stored in-cluster — fine for test/PoC).

```bash
# 7a. Create the KMS key + alias.
KEY_ARN=$(aws kms create-key --description "4minds-openbao-${CLUSTER}" \
  --region "$REGION" --query KeyMetadata.Arn --output text)
aws kms create-alias --alias-name "alias/4minds-openbao-${CLUSTER}" \
  --target-key-id "$KEY_ARN" --region "$REGION"

# 7b. Create a least-privilege IAM policy scoped to just that key.
aws iam create-policy --policy-name "4minds-openbao-kms-${CLUSTER}" \
  --policy-document "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Action\":[\"kms:Encrypt\",\"kms:Decrypt\",\"kms:DescribeKey\",\"kms:GenerateDataKey\"],\"Resource\":\"${KEY_ARN}\"}]}"

# 7c. Create the IRSA ROLE ONLY (--role-only). The chart creates and owns the
#     `platform-openbao` ServiceAccount, so do NOT let eksctl create one too.
eksctl create iamserviceaccount \
  --name platform-openbao --namespace "$NAMESPACE" \
  --cluster "$CLUSTER" --region "$REGION" \
  --role-name "4minds-openbao-irsa-${CLUSTER}" \
  --attach-policy-arn "arn:aws:iam::${ACCOUNT}:policy/4minds-openbao-kms-${CLUSTER}" \
  --role-only --approve
```

Then set these in `my-values.yaml`. The chart binds `roleArn` onto the OpenBao
ServiceAccount as `eks.amazonaws.com/role-arn` automatically (no manual
annotation, survives `helm upgrade`):

```yaml
openbao:
  seal:
    mode: kms
    kms:
      provider: aws
      region: <your-region>
      keyId: alias/4minds-openbao-<your-cluster>
      roleArn: arn:aws:iam::<account>:role/4minds-openbao-irsa-<your-cluster>
```

> **Azure / GCP:** set `provider: azure` (`tenantId`/`vaultName`/`keyName`) or
> `provider: gcp` (`projectId`/`locationId`/`keyRing`/`cryptoKey`) and bind the
> pod identity via `openbao.serviceAccount.annotations` (Workload Identity).

### Prerequisites summary

| # | Prerequisite | Why |
|---|---|---|
| 1 | EKS cluster + OIDC | run the platform; OIDC needed for IRSA |
| 2 | Default StorageClass + EBS CSI | stateful services (PVCs) |
| 3 | ingress-nginx controller | serves the Ingress the chart creates |
| 4 | Namespace `4minds` | where everything installs |
| 5 | `frontend-tls` secret | TLS termination for your hostname |
| 6 | Node IAM ECR pull | pull the images (usually automatic on Marketplace) |
| 7 | KMS key + IRSA role | OpenBao auto-unseal (only for `seal.mode: kms`) |

---

## Configure

**What this does:** creates your deployment's values file. Copy the template and
fill in the required fields — hostname, email, LLM/embedding endpoints, symi,
wren-ai, seal mode, and any SSO/integrations you use. Everything else has a
sensible default.

```bash
cp values-customer-template.yaml my-values.yaml
$EDITOR my-values.yaml
```

### (Optional) External S3 instead of the bundled MinIO

By default object storage is the in-cluster MinIO and you set nothing. To use
AWS S3 (or any S3-compatible endpoint) instead, add to `my-values.yaml`:

```yaml
storage:
  endpoint: "https://s3.us-east-1.amazonaws.com"   # empty = bundled MinIO
  region: us-east-1
  bucketName: 4minds-uploads
secrets:                    # keep ONE secrets: block — merge with OAuth secrets
  s3AccessKey: "<access-key>"
  s3SecretKey: "<secret-key>"
```

Leave `storage.endpoint` / `secrets.s3*` empty to keep the bundled MinIO. If you
also set OAuth client secrets, put the S3 keys under the **same** `secrets:`
block — a second `secrets:` key silently overrides the first (YAML has no merge).
For Azure Blob, set `storage.useAzure: "true"` and `secrets.azureStorageAccountKey`
in that same block.

---

## Install

**What this does:** deploys the platform with your values. The chart's hooks then
run automatically (OpenBao init+unseal, backend-secret seeding, Kafka topics, the
mlai schema migration) — no manual steps.

```bash
helm install 4minds . -n "$NAMESPACE" -f my-values.yaml
kubectl -n "$NAMESPACE" get pods -w        # watch until all are Running/Completed
```

> **AWS Marketplace subscribers.** After subscribing, the AWS console gives you a
> `helm pull` (from the Marketplace ECR) + `helm install` snippet. That snippet
> installs with **defaults only** — you MUST append your config, or the platform
> comes up unconfigured (no hostname, no inference endpoints). Keep the AWS
> `--set` flag AND add `-f my-values.yaml`:
>
> ```bash
> helm install 4minds ./<pulled-chart-dir> -n 4minds --create-namespace \
>   --set global.awsmpServiceAccountName=backend-service \
>   -f my-values.yaml
> ```

---

## Verify

**What this does:** confirms the platform is up and serving.

```bash
kubectl -n "$NAMESPACE" get pods                       # all 1/1 Running
kubectl -n "$NAMESPACE" get ingress 4minds-ingress     # ADDRESS = your LB
curl -k https://$HOSTNAME_FQDN/                        # returns the 4MINDS frontend
```

Then open `https://$HOSTNAME_FQDN` and sign up (email + password works out of the
box; SSO only if you configured a provider).

---

## Upgrades

```bash
helm upgrade 4minds . -n "$NAMESPACE" -f my-values.yaml
```

Hooks are idempotent (OpenBao "already initialized", mlai migration re-verify).
Auto-generated secrets (PG/Redis/MinIO/Fernet/Tier/SYMI) are preserved across
upgrades via `helm.sh/resource-policy: keep`.

---

## Uninstall

```bash
helm uninstall 4minds -n "$NAMESPACE"

# PVCs and the generated-secrets Secret are retained by design. To wipe fully:
kubectl -n "$NAMESPACE" delete pvc --all
# platform-openbao-keys only exists in seal.mode: lab (Shamir); in kms mode there
# are no unseal keys to delete — --ignore-not-found keeps this harmless.
kubectl -n "$NAMESPACE" delete secret 4minds-generated-secrets platform-openbao-keys --ignore-not-found
```

> ⚠️ Deleting PVCs destroys all data (Postgres, Qdrant, MinIO, etc.).

---

## Troubleshooting

| Symptom | Likely cause / fix |
|---|---|
| Pods `Pending` on PVC | No default StorageClass / EBS CSI not ready (Step 2) |
| `ImagePullBackOff` | Node IAM lacks ECR pull, or wrong `imagePullSecrets` (Step 6) |
| Ingress has no ADDRESS | ingress-nginx controller not installed (Step 3) |
| App pods `CrashLoopBackOff` early on | OpenBao not unsealed yet — `kubectl logs job/4minds-openbao-bootstrap` |
| Symi: "No API key for provider …" | `symi-gateway.config.llm.baseUrl` + `llmModel` not set → set your Symi LLM endpoint |
| Symi UI: "Backend service is unreachable" | leave `symi-gateway.config.apiKey` empty so it shares the backend's generated key |
| Dataset stuck "processing" until refresh | check `kubectl logs deploy/postgres-kafka-bridge` (the `wait-for-tables` initContainer gates trigger install) |