# AINS Microservices — Migration Plan

Convert the legacy Yii2 monolith at `/var/www/html/ains` into a Go/Gin
microservices system with an HTMX + Vue (no build tools) frontend.

Status: **planning**. Phase 0 not started.

---

## 0. Locked decisions

| # | Decision |
|---|----------|
| 1 | **MySQL kept.** Same `ains` database initially; each service owns a table subset. Existing tables/columns reused as-is during migration. |
| 2 | **JWT, stateless.** No Redis session store. Access + refresh JWTs. SSR/HTMX receives the JWT via HttpOnly cookie and verifies it statelessly each request; Vue/API use `Authorization: Bearer`. Short TTL + Redis denylist for logout/revocation. |
| 3 | **Admin rebuilt in Go/HTMX, but the Dynaweb frontend builder is dropped.** No `frontend_site`, `frontend_page`, `frontend_content`, `frontend_content_assign`, `frontend_content_category/tags/template`, no page/block editor. **Pages are code-defined routes + Go templates**; only publishing data (articles/news, announcements, sliders, downloads, galleries, menus) remains DB-driven. |
| 4 | **No git operations.** Files are written; the repository owner commits manually. |
| 5 | **Single site only: AINS NILAM** (`frontend_site.site_id = 20`, default/active). The old `site_id = 1` is ignored. |
| 6 | **Google OAuth** is the auth provider (credentials in `frontend/config/params-local.php`). IAM Delima SSO optional/later. |
| 7 | **Reporting is in scope and designed for millions of rows** (dev `record` is already 17.8M; production is larger). |
| 8 | **No `eval()` anywhere.** All dynamic-PHP endpoints are redesigned. |

---

## 1. What the legacy app is

A **Yii2 (Dynaweb CMS) monolith** — a national school reading-programme portal
(KPM/NILAM-style: reading records, student-created ebooks, chatbot,
reports/scoreboards) plus a DB-driven page-builder CMS.

Key evidence:

- Config: `frontend/config/main.php`, `common/config/db.php` (`dbname=ains`).
- Live DB volumes (dev subset):
  - `record` = **17,825,776** rows (per-student cumulative reading aggregates).
  - `reading_record_v1` ≈ 4,179,981 (timestamped per-student reading totals).
  - `frontend_user` = 10,389 (`teacher` 10,221, `ppd` 145, `jpn` 18, `kpm` 1, `murid` 3).
  - `school` = 10,308, `reading_record` = 55, `total_student` = 164.
  - `frontend_content` = 481 (314 PHP / 167 HTML), `frontend_page` = 148.
- The legacy frontend is **not views** — only `frontend/views/site` exists; every
  page is a `frontend_content` row with `content_is_php=1` executed live
  (e.g. block `270` student dashboard, `340` record-approved list ~61 KB).
  Business logic lives in SQL-in-PHP inside the DB.
- Legacy backend is the Dynaweb admin: modules `readingrecord`, `fruserdynav2`,
  `school`, `chat`, `book`, `refdynawebv2`, `content*`, `theme`, `uiuxBuilder`,
  `moduleRepository`, `mimin`, plus audit/i18n/support.
- Auth: `frontend_user` (roles `murid/teacher/ppd/jpn/kpm`) + `dmg_user`, RBAC
  (`auth_item`), bcrypt hashes, Google OAuth
  (`frontend/modules/loginprovider/GoogleAuthController.php`), IAM Delima SSO,
  `common/components/JWT.php`.
- Dangerous `eval()` ports:
  - `frontend/controllers/DownloadReportController.php:13` (`report_code`).
  - `frontend/controllers/SiteController.php:145` (`api` table).
  - `frontend/controllers/ChatController.php:494` (`answer_bot.php_status = 1`).
- Files: GCS via `common/components/GcsHelper.php` + signed URLs; ebook proxy
  `frontend/controllers/EbookController.php`; FlowPaper apps in `www/ebook`,
  `www/flowpaper`.
- Build tooling to drop: `package.json` uses webpack/vue2/vuesax.

---

## 2. Domain model (bounded contexts)

- **Identity & Directory** — `frontend_user` (roles `murid/teacher/ppd/jpn/kpm`),
  `school` (state -> ppd -> school scope), `ref` lookups, `frontend_role_mapping`,
  `auth_*` RBAC.
- **Reading Records** — `reading_record` (current schema), `reading_record_v1`
  (timestamped per-student totals), `record` (per-student cumulative aggregates,
  no date column), `total_student` (denominators), `report_student_record*`.
- **Library / MyBook** — `book`, `mybook`, `mybook_chapter`; ebook/PDF reading via
  FlowPaper.
- **Chatbot** — `chat_*` (question tags, `answer_bot` with `php_status`, live chat,
  suggestions).
- **Analytics / Reporting** — `report_student_active_by_state/ppd`,
  `report_average_record_by_state/ppd`, `data_dashboard`, `report_code`.
- **Content publishing** — `content_article*`, `content_application`,
  `content_slider*`, `content_downloads`, `content_photo_*`, `content_video*`,
  `frontend_menu`. (No page/block builder.)
- **Media** — uploads, GCS signed URLs, PDF proxy.

---

## 3. Target architecture

```
                         ┌──────────────┐
  Browser (HTMX + Vue) ─▶│  edge / web  │  Gin: JWT verify, RBAC+scope, HTML shells,
                         │  (BFF)       │  HTMX fragments, i18n, reverse-proxy legacy
                         └──────┬───────┘
         internal REST/JSON + events (NATS)
   ┌───────────┬───────────┬───────────┬───────────┬───────────────┐
  auth-svc   record-svc  library-svc chatbot-svc content-svc
 (identity/  (reading    (book/      (Q&A/live)  (publishing
  directory)  records +   mybook)                 + reporting)
              reporting)
   └───────────┴───────────┴───────────┴───────────┴───────────────┘
     worker (nightly aggregates, exports, notifications)
     MySQL (shared first -> per-schema)  Redis  GCS  NATS
```

### Boundary rule

Tightly-coupled domain/process lives in **one** service.

- **Within a service:** direct SQL against its own tables (shared `ains` DB).
- **Between services:** REST/JSON or NATS events only — **no cross-service SQL**.
- Cross-domain data needed for reads is either fetched from the owner's
  API/events or **denormalized locally at write time** (e.g. `report_student_record`
  copies school/state/ppd so record reporting never calls auth).

### Stack

- **Backend:** Go + Gin.
- **Frontend:** Go `html/template` + HTMX, Vue 3 global build (no bundler).
- **Data:** MySQL (existing), Redis (cache, rate limits, denylist, scoreboards),
  NATS (events/jobs), GCS (files).
- **Migrations:** `golang-migrate`, one folder per service.

---

## 4. Service specification

| Service | Responsibility | Owns (tables) |
|---|---|---|
| **edge/web** | SSR shell, routing, JWT verification, RBAC + scope guard, i18n, reverse-proxy to legacy for un-migrated routes | none |
| **auth-svc** | Google OAuth, users, schools, ref, roles, profiles | `frontend_user`, `school`, `ref`, `auth_*`, `frontend_role_mapping` |
| **record-svc** | reading-record CRUD, teacher batch entry, approval workflow, points engine, rate limits, totals, **plus reading-record reporting** (per-student snapshot, scoreboards, state/PPD aggregates, async exports) | `reading_record`, `reading_record_v1`, `total_student`, `report_student_record(+_log)`, `report_average_record_by_*`, `report_student_active_by_*`, record scoring config. (`record` deferred.) |
| **library-svc** | books, mybook + chapters, approval -> book, ebook/FlowPaper | `book`, `mybook`, `mybook_chapter` |
| **chatbot-svc** | greeting, Q&A matching, tags, suggestions, live chat (WebSocket) | `chat_*` |
| **content-svc** | publishing (articles/news, announcements, sliders, downloads, galleries, menus) **plus content-side reporting** | `content_article*`, `content_application`, `content_slider*`, `content_downloads`, `content_photo_*`, `content_video*`, `frontend_menu`, `data_dashboard` |
| **worker** | nightly aggregation, export jobs, notifications | — |
| **pkg/media** | GCS upload + signed URL, `/ebook/proxy` (shared library, not a service) | — |

Dropped: page/block engine, `frontend_site/page/content*/theme/settings/header/footer`
admin, `uiuxBuilder`, `moduleRepository`, `grape_component`, webpack.

### Admin rebuild scope

Rebuild the Dynaweb admin in Go/HTMX for: auth-svc (users, schools, ref, roles),
record-svc (records + approval + report configs), library-svc (books, mybook
approval), chatbot-svc (Q&A, tags), content-svc (articles, sliders, menus,
downloads). **Excluded:** Dynaweb frontend site/page/content-block
builder.

---

## 5. Pages are code

Each AINS NILAM page (`frontend_site.site_id = 20`) maps to a Go route +
template under `web/templates/pages/`:

- `dashboard` -> `student/dashboard.html`
- `teacher-record-list` -> `teacher/record_list.html`
- `kpm-report-average-record-state` -> `officer/report_average_state.html`

Shared context layouts: `public`, `student`, `teacher`, `officer`.
Existing URL prefixes (`/web/<page>`, `/portal-main/<page>`, `/paper/<page>`)
are preserved by edge routes.

### Page inventory -> service (site 20, ~120 pages)

- **Public / Content:** `home`, `home2`, `article`, `reading-center`,
  `forgot-password`, `auth-failed`, `under-maintenance`, `feed`, `articlelist`,
  support/faq.
- **Identity:** `view-profile-*`, `update-profile-*`, login/profile flows.
- **Records:** `dashboard`, `student-record-list/-details`,
  `new-record-book/digital/article`, `teacher-record-list/-details/-approved-list/-pending-list`,
  `teacher-new-record-*`, `teacher-student-list`.
- **Library:** `mybook*`, `book-generator`, `reading-material-*`,
  `student-mybook-*`, `teacher-mybook-*`.
- **Reporting:** all `scoreboard-*`, `*-statistics`, `*-report-*`,
  `ppd/jpn/kpm-dashboard`.

---

## 6. Reporting design at scale

- Reporting lives **with the data owner**: reading-record reporting in
  `record-svc`, content reporting in `content-svc`. No standalone reporting
  service (keeps reports on same-service SQL, no cross-service calls).
- `record` (legacy millions + external sync) is **deferred**; not part of the
  rewrite yet.
- **Nightly worker** (event/scheduled) rebuilds `report_student_active_by_*`,
  `report_average_record_by_*`, `total_student`, plus school-level summaries.
  record-svc reads its summaries directly.
- **Scoreboards** = top-N via summary tables + Redis sorted sets.
- **Exports/downloads** (formerly `report_code` `eval()`) -> async job (NATS) ->
  CSV/XLSX in GCS + signed URL.
- Add covering indexes; put heavy reports on a **read replica**. ClickHouse only
  if profiling demands it (deferred).
- Aggregation jobs are idempotent upserts run from `cmd/worker`.

---

## 7. Auth design (Google OAuth, JWT)

- Server-side Google OAuth (Authorization Code) in auth-svc using existing
  client id/secret; callback -> find/create `frontend_user`
  (`username = email`, default role `murid`, `provider = google.com`).
- **JWT access + refresh**, stateless. No session store.
  - SSR/HTMX: HttpOnly cookie carrying the access token, verified per request.
  - Vue/API: `Authorization: Bearer`.
  - Short TTL + Redis denylist for logout/revocation.
- **RBAC + scope**: role (`murid/teacher/ppd/jpn/kpm`) x scope
  (state/ppd/school), resolved from `frontend_user` + `school`.
- Retire the hardcoded `JWT::SECRET_KEY`. Credentials/env via config.
- IAM Delima SSO as an optional second provider behind the same interface.

---

## 8. Frontend (HTMX + Vue, no build tools)

- Go `html/template`: context layouts + page templates; HTMX swaps partials.
- **Vue 3 global build** vendored (`web/static/vendor/vue.global.prod.js`) for
  islands: live chat, scoreboards/charts, multi-step record form, upload preview.
- Vendored assets: `htmx.min.js`, Bootstrap, Bootstrap Icons, chart lib.
- i18n MY/EN with Malay default (matches current fallback behaviour).
- Current blocks' `fetch` calls become HTMX attributes + a few JSON endpoints
  consumed by Vue.

---

## 9. Data migration strategy

- **Strangler fig:** keep Yii2 running; edge reverse-proxies un-migrated routes
  to legacy until each page moves.
- **Shared MySQL first**, each service owns its tables; no cross-service SQL
  joins (use service APIs/events). Split to per-service schemas once stable.
- `golang-migrate` per service under `db/migrations/<svc>/`; ORM never
  auto-migrates.
- Migrate page-by-page: legacy route -> Go handler + template, verify parity,
  flip edge route.

### No `eval()` (redesign map)

| Legacy | Replacement |
|---|---|
| `report_code` (`DownloadReportController.php:13`) | record-svc async export templates |
| `api` table (`SiteController.php:145`) | explicit Go endpoints |
| chatbot `answer_bot.php_status=1` (`ChatController.php:494`) | safe placeholder/template expansion |

---

## 10. Repo layout (`/var/www/html/ains-microservices`)

```
go.mod
docker-compose.yml                 # mysql(snapshot)+redis+fake-gcs+nats
cmd/{edge,auth,record,library,chatbot,content,worker}/main.go
internal/{auth,record,library,chatbot,content}/{handler,service,repo,model}.go
pkg/{config,db,jwt,gcs,events,middleware,i18n,scope,media}/
web/templates/{layouts,pages,partials}/...
web/static/vendor/                 # htmx, vue, bootstrap (no build)
db/migrations/<svc>/
docs/PLAN.md
```

---

## 11. Execution phases

Each phase ends with explicit exit criteria.

- **Phase 0 — Foundations (1–2 wk):** repo skeleton, `docker-compose`
  (MySQL snapshot + Redis + fake-gcs + NATS), shared packages, edge healthcheck,
  lint/test CI, `docs/PLAN.md`.
  *Exit:* every service `/healthz` = 200 under `docker compose up`.
- **Phase 1 — Identity + shell (2–3 wk):** Google OAuth, users/schools/ref,
  JWT + RBAC + scope, layouts/nav, i18n, page routing with legacy fallback.
  *Exit:* login + profile + scoped nav served by Go.
- **Phase 2 — Reading records (3–4 wk):** record CRUD, teacher batch, approval,
  points, rate limits, lists/details; uploads via media.
  *Exit:* student + teacher record pages live.
- **Phase 3 — Library / MyBook (2–3 wk):** books, mybook + chapters,
  approval -> book, ebook proxy / FlowPaper.
- **Phase 4 — Reporting (3–5 wk):** pre-aggregation jobs, scoreboards,
  statistics, async exports, read replica. Work lands inside record-svc
  (reading-record reporting) and content-svc (content reporting), not a
  standalone service.
  *Exit:* parity on report/scoreboard pages at scale.
- **Phase 5 — Chatbot (2 wk):** Q&A matching, live chat WS, safe answer templating.
- **Phase 6 — Content publishing + admin (4–6 wk):** articles/news, sliders,
  menus, downloads; admin CRUD in Go/HTMX (excluding the Dynaweb page builder).
- **Phase 7 — Hardening / cutover:** observability, load test at millions,
  security (remove plaintext creds), decommission Yii2, drop webpack.

### Phase 0 deliverables

```
go.mod
docker-compose.yml
cmd/{edge,auth,record,library,chatbot,content,worker}/main.go
internal/{auth,edge}/
pkg/{config,db,jwt,gcs,events,i18n,scope,middleware}/
pkg/{oauth2,delima}/
web/ (embed.go, view.go, static.go, i18n.go, templates/, static/vendor/)
db/migrations/<svc>/
.golangci.yml, .gitlab-ci.yml, Makefile
docs/PLAN.md
```

Exit criteria — **met (2026-10-16)**: `docker compose up`; `/healthz` 200 on every
service; Google OAuth login flow implemented (mock + real wiring); base
layouts/vendor assets; lint/test CI green.

---

## 12. Risks & open items

Risks:

1. **PHP-in-DB blocks** are the legacy app's real logic — the new system drops
   them; each page's behaviour must be reimplemented in Go templates/handlers.
2. **`record` at millions of rows** — reporting needs pre-aggregation and a read
   replica; heavy live queries must be avoided.
3. **`eval()` endpoints** need redesign, not porting.
4. **Data volume discrepancy** — dev DB is a small subset; plan for production
   scale in indexing, pagination, and exports.
5. **Two sites in the legacy DB** — only site 20 is used; ignore site 1.

Open items to pin before Phase 1:

- **Student data source**: `record` (per-student, no date, older-system import)
  is **deferred**; revisit only if historical carry-over is required.
- **Google OAuth callback host**: `GOOGLE_REDIRECT_URL` is wired (default
  `http://localhost:8080/auth/google/callback`); set the real deploy host + add it
  to the Google Console redirect URIs.
- **Notification service** (email) — Phase 1 or later.
- **Denylist**: in-memory per instance in the Phase 0 stub; move to Redis before
  scaling auth beyond one replica.
```