ï»¿
Your IP : 216.73.216.79

Current File : /var/www/html/ains/docs/gcs-upload-migration.md
# Migrate Record Cover Uploads to Google Cloud Storage

Goal: replace local-disk uploads (`frontend/web/uploads/record/`) with
Google Cloud Storage (GCS) for reading-record cover images, using a Yii2
extension (`diecoding/yii2-flysystem`) instead of hand-rolled SDK calls.

Status: **code complete**. Remaining work is infra/config only (see
"Setup checklist" below) — nothing else needs code changes.

## Why this extension

- `diecoding/yii2-flysystem` (v1.8): actively maintained Flysystem 3
  integration for Yii2, ships a ready-made
  `diecoding\flysystem\GoogleCloudStorageComponent`.
- Underlying adapter: `league/flysystem-google-cloud-storage` 3.x on the
  official `google/cloud-storage` SDK (REST transport, no grpc ext needed).
- The classic alternative `creocoder/yii2-flysystem` is abandoned (2019,
  Flysystem 1.x) and riskier on PHP 8.4.

## Installed packages

```
composer require diecoding/yii2-flysystem "league/flysystem-google-cloud-storage:^3.0"
```

Pulled in: `google/cloud-storage`, `google/auth`, `google/gax`, etc.
(see `composer.lock`).

## Files changed

| File | Change |
| --- | --- |
| `composer.json` / `composer.lock` | new deps above |
| `common/config/params.php` | added `gcs` params block (bucket, keyFilePath, publicBaseUrl) |
| `frontend/config/main.php` | registered `gcs` application component |
| `common/components/GcsHelper.php` | **new** reusable class; `GcsHelper::save($tempFile, $objectPath)` streams a temp file to GCS via `Yii::$app->gcs->writeStream()`, returns bool |
| `frontend/controllers/ReadingRecordController.php` | in both `actionSaverecord` and `actionTeachersaverecord` the local `$coverImg->saveAs()` step was replaced by `\common\components\GcsHelper::save(...)`; validation logic unchanged and still inline in each action |
| `frontend/models/ReadingRecord.php` | `getcover()` resolves stored path to URL (see "URL resolution") |

## Component config (already wired)

`frontend/config/main.php`:

```php
'components' => [
    'gcs' => [
        'class' => diecoding\flysystem\GoogleCloudStorageComponent::class,
        'bucket' => $params['gcs']['bucket'],
        'keyFilePath' => Yii::getAlias($params['gcs']['keyFilePath']),
    ],
    ...
],
```

`common/config/params.php` (placeholders — must be replaced):

```php
'gcs' => [
    'bucket' => 'your-bucket-name',
    'keyFilePath' => '@common/config/gcs-credentials.json',
    'publicBaseUrl' => 'https://storage.googleapis.com/your-bucket-name',
],
```

## Setup checklist (manual)

1. **Create bucket** in Google Cloud Console (region of choice).
   Suggest name pattern `ains-uploads-<env>`.
2. **Service account**:
   - IAM & Admin -> Service Accounts -> Create (e.g. `ains-gcs-uploader`).
   - Grant `roles/storage.objectCreator` (or `objectAdmin`) **on the bucket**
     (bucket-level IAM binding preferred over project-level).
   - Keys -> Add key -> JSON. Downloaded file = the credentials JSON:
     ```json
     { "type": "service_account", "project_id": "...",
       "private_key": "-----BEGIN PRIVATE KEY-----...", ... }
     ```
3. **Place key file** at `common/config/gcs-credentials.json`.
   - Must be readable by PHP user (`www-data`).
   - Add to `.gitignore`: it is as sensitive as a password.
4. **Fill real values** into `gcs` params (bucket + publicBaseUrl).
5. **Public reads**: not needed — images are served via **signed URLs**
   (see "Signed URLs" section below). The bucket stays fully private.
   Remove any existing `allUsers` -> `roles/storage.objectViewer` grant
   if present.
6. **Uniform bucket-level access (UBLA)**: keep it enabled. Note: the
   league GCS adapter defaults to `visibility: private`, which sends a
   legacy `predefinedAcl=projectPrivate` on every write and fails with
   400 "Cannot insert legacy ACL..." on UBLA buckets. `GcsHelper::save()`
   passes `PortableVisibilityHandler::NO_PREDEFINED_VISIBILITY` to omit
   the ACL entirely — keep that if you bypass the helper.
   Reads are served via signed URLs (not bucket IAM), so no public
   grants are required.

## How it works after migration

Upload flow (unchanged validation, new storage step):

1. Same validation as before, inline in each action: extension whitelist
   (jpg/jpeg/png/gif/webp), MIME sniff via finfo (anti-spoofing), max 5 MB.
2. Safe random filename: `uploads/record/<32-char-random>.<ext>`
   (same relative-path convention as before).
3. Stored via helper: `\common\components\GcsHelper::save($tempName, $relativePath)`
   -> `Yii::$app->gcs->writeStream()` (errors logged, returns false on failure).
   Reusable from any controller: backend, console, other uploads.
4. DB column `reading_record.record_cover_img` still stores the relative
   object path (DB stays portable across storage backends/domains).

URL resolution (`ReadingRecord::getcover()`):

- value starts with `http(s)://` -> used as-is (external book thumbnails;
  also fixes pre-existing bug where these got a bogus leading `/`);
- not a URL and NOT present under `@webroot/` -> served from GCS via
  a **signed URL** generated by `GcsHelper::getSignedUrl()` (expires in
  1 hour; fresh URL generated per request);
- otherwise -> legacy local `/uploads/...` (old records keep rendering).

## Signed URLs

Objects are served via short-lived signed URLs instead of public access.
This keeps the bucket fully private — no `allUsers` grant needed.

`GcsHelper::getSignedUrl($objectPath, $ttl = 3600)`:

```php
// Default 1-hour expiry
$url = \common\components\GcsHelper::getSignedUrl('uploads/record/abc123.jpg');

// Custom TTL (e.g. 30 minutes)
$url = \common\components\GcsHelper::getSignedUrl('uploads/record/abc123.jpg', 1800);
```

- Returns the signed URL string, or `null` on failure.
- DB stores the relative path only — signed URLs are never persisted.
- Usable from any controller, model, or view across the application.

## If app later runs ON Google Cloud (Compute Engine/GKE/Cloud Run/App Engine)

You can drop the JSON key entirely: the VM/pod's attached service account is
auto-detected via Application Default Credentials.

```php
// frontend/config/main.php — remove 'keyFilePath'
'gcs' => [
    'class' => diecoding\flysystem\GoogleCloudStorageComponent::class,
    'bucket' => $params['gcs']['bucket'],
],
```

Just attach the SA with `roles/storage.objectCreator` on the instance.
On self-hosted servers (current setup: `/var/www/html` + local MySQL) there
is no ambient identity, so the explicit key file is required.

## Testing against a local fake GCS

Run [fake-gcs-server](https://github.com/fsouza/fake-gcs-server) (default:
self-signed **HTTPS** on 4443):

```bash
docker run -d --name fake-gcs-server -p 4443:4443 fsouza/fake-gcs-server -data /data
```

All `curl` calls against it need `-k/--insecure`.

1. **Create a throwaway service-account key** — fake server ignores auth, but
   `google/auth` signs JWTs locally, so the private key must be real RSA:
   generate any dummy SA JSON into `common/config/gcs-credentials.json`
   (already gitignored).
2. **Params** (`common/config/params.php`) for local testing:
   ```php
   'gcs' => [
       'bucket' => 'ains-uploads-test',
       'keyFilePath' => '@common/config/gcs-credentials.json',
       'apiEndpoint' => 'https://localhost:4443',
       'insecure' => true,
       // fake-gcs-server serves raw bytes at /download/storage/v1/b/<b>/o/<object>
       'publicBaseUrl' => 'https://localhost:4443/download/storage/v1/b/ains-uploads-test/o',
   ],
   ```
3. **Component** (`frontend/config/main.php`) — pass the endpoint through and
   skip self-signed cert verification when `insecure` is set:
   ```php
   'gcs' => [
       ...
       'apiEndpoint' => $params['gcs']['apiEndpoint'] ?? null,
       'httpHandler' => empty($params['gcs']['insecure']) ? null : static function ($request, array $options = []) {
           return (new \GuzzleHttp\Client())->send($request, ['verify' => false, 'http_errors' => false] + $options);
       },
   ],
   ```
4. **Create the bucket** and test an upload from CLI:
   ```bash
   curl -k -X POST 'https://localhost:4443/storage/v1/b?project=test-project' \
        -H 'Content-Type: application/json' -d '{"name":"ains-uploads-test"}'

   php -r "
   define('YII_ENV','prod'); define('YII_DEBUG',false);
   require 'vendor/autoload.php'; require 'vendor/yiisoft/yii2/Yii.php';
   require 'common/config/bootstrap.php';   // registers @common alias
   new yii\web\Application(require 'frontend/config/main.php');
   \$tmp = tempnam(sys_get_temp_dir(), 'gcs'); file_put_contents(\$tmp, 'hello');
   var_dump(\common\components\GcsHelper::save(\$tmp, 'uploads/record/test.txt'));
   var_dump(Yii::\$app->gcs->fileExists('uploads/record/test.txt'));
   "
   curl -k "https://localhost:4443/download/storage/v1/b/ains-uploads-test/o/uploads/record/test.txt"
   ```

Notes:
- Flysystem 3's `writeStream()` returns void, so `GcsHelper::save()` must
  return `true` explicitly after the call (otherwise every successful upload
  would report `failed:upload_error` in the controller) — fixed.
- Direct paths (`https://localhost:4443/<bucket>/<object>`) are NOT served by
  this image; use the `/download/storage/v1/...` URL above.
- Browsers will still warn on the self-signed cert when rendering cover
  images; accept the exception or run the container with `-scheme http`.
- Before going to production: restore real values in `gcs` params and remove
  `apiEndpoint`/`insecure` handling (they are no-ops when unset).

## Verification once configured

```bash
php -r "
define('YII_ENV','prod'); define('YII_DEBUG',false);
require 'vendor/autoload.php'; require 'vendor/yiisoft/yii2/Yii.php';
new yii\web\Application(require 'frontend/config/main.php');
var_dump(Yii::$app->gcs->fileExists('uploads/record/test.txt'));
"
```

Then upload a record cover through the UI and confirm:
- object appears in bucket under `uploads/record/`,
- `record_cover_img` holds `uploads/record/<random>.ext`,
- cover renders on the portal (signed GCS URL for new records, `/uploads/...`
  untouched for old ones).

Rollback = revert the controller/model commits; old records were never moved,
and no data format changed in the DB.

## FlowPaper PDF viewing from GCS

### Problem

FlowPaper (standalone PHP app at `www/ebook/`) renders PDFs via browser-side
XHR. Passing a GCS signed URL directly as `PDFFile` fails with a CORS error
because GCS does not include `Access-Control-Allow-Origin` headers on signed
URLs by default.

### Solution: same-origin proxy

Instead of pointing FlowPaper at GCS directly, route the PDF through a Yii2
controller that fetches from GCS and streams it back to the browser. The
browser sees a same-origin request — no CORS issue.

### Files changed

| File | Change |
| --- | --- |
| `frontend/controllers/EbookController.php` | **new** proxy action; accepts `?doc=<object-path>`, fetches from GCS via signed URL, streams PDF bytes back |
| `frontend/config/main.php` | added `'ebook/proxy' => 'ebook/proxy'` URL rule |
| `frontend/models/Mybook.php` | added `getPdfUrl()` — returns proxy URL for GCS-stored PDFs, local path for legacy uploads |
| `www/ebook/php/view_doc.php` | accepts `?pdf_url=<url>` query param; uses it as FlowPaper's `PDFFile` when present |

### How it works

1. Button link generates: `/ebook/php/view_doc.php?pdf_url=/ebook/proxy?doc=uploads/ebook/xxx.pdf`
2. `view_doc.php` sets FlowPaper's `PDFFile` to the `pdf_url` value
3. FlowPaper fetches from `/ebook/proxy?doc=uploads/ebook/xxx.pdf` (same origin)
4. `EbookController::actionProxy()` generates a short-lived GCS signed URL (5 min),
   fetches the PDF bytes, and streams them back with `Content-Type: application/pdf`
5. No cross-origin request — CORS is never a factor

### Mybook model helper

```php
$model->getPdfUrl()
// Returns:
//   '/ebook/proxy?doc=uploads/ebook/xxx.pdf'  (GCS-stored PDFs)
//   '/uploads/ebook/xxx.pdf'                   (legacy local uploads)
//   'https://...'                              (external URLs)
//   null                                       (no file)
```

### Button template

```php
$url = '#';
if(!empty($model->mybook_file_url)){
    $pdfUrl = $model->getPdfUrl();
    if ($pdfUrl) {
        $url = '/ebook/php/view_doc.php?pdf_url=' . urlencode($pdfUrl);
    } else {
        $ebook_link_arr = explode('/',$model->mybook_file_url);
        $url = '/ebook/php/view_doc.php?subfolder=&doc='.end($ebook_link_arr);
    }
}
```

### Proxy endpoint

`GET /ebook/proxy?doc=<object-path>`

- Fetches from GCS via `GcsHelper::getSignedUrl()` (5-minute TTL)
- Returns PDF bytes with `Content-Type: application/pdf`
- Supports legacy local files (not yet migrated to GCS)

### Rollback

Revert the four files above; FlowPaper falls back to reading PDFs from the
local `path.pdf` directory.