Content Pipelines (SOP)

Each row below links to a dedicated SOP file for the pipeline's full setup, credentials, error handling, and recovery steps. When any pipeline's schedule, source, destination, or status changes, the corresponding src/content/sop/<slug>.md file must be updated in the same commit as the code change, and SOP-IT-011 in Claude_Lives_Here/SOPs/IT/ gets the detailed narrative.

Live

Pipeline	Source	Destination	Schedule	Updated
Discord Action Items Discord	`#work-action-items + #personal-action-items (Discord)`	`/work-actions + /personal-actions`	On-demand (Phil-triggered pull); bot listener is always-on	`2026-04-18`
Google Drive Blood Panels Drive	`Drive folder 'blood-panels' (id 1Wdt3JumRZzy54yovlpRlgnrJwwjElelr)`	`/blood-panels`	On-demand (Phil tells Real Chuck 'pull blood panels')	`2026-04-18`
Local Editor (engelsplace-dev) Manual	`http://127.0.0.1:4321/edit/ (Phil's laptop/desktop only)`	`All collections — actionitems, bloodmarkers, panels, etc.`	Always-on (pm2 service); on-demand writes	`2026-04-18`
FMX Maintenance API API	`fair-rite-products-corp.gofmx.com/api/v1 (filtered to building 110428)`	`/maintenance + /maintenance-metrics`	On-demand (cron disabled per Phil 2026-04-18). Phil triggers in Real Chuck.	`2026-04-19`
FMX Preventive Maintenance API API	`fair-rite-products-corp.gofmx.com/api/v1 (filtered to building 110428)`	`/preventive-maintenance + /preventive-maintenance-metrics`	On-demand. Phil triggers in Real Chuck.	`2026-04-19`
Flat Rock Facility Meeting Minutes Email	`fairriteworksync@gmail.com weekly meeting minutes .docx (auto-pulled via Gmail OAuth as of 2026-04-20)`	`/safety + /5s-scorecard + /ftz + /kits + /qc + /shipping-receiving + /powder + /facility + /evaluations + /holidays`	Daily 5 AM CDT (auto-pull via Gmail). Meeting Mondays; minutes typically arrive Tue/Wed.	`2026-04-20`

Discord Action Items — full procedure

Phil posts a message in either action-items channel from any device. The engel-ops-bot messageCreate listener captures it, writes to IT/discord-gateway-bot/engelsplace-discord-queue.json, and returns a confirmation reaction. Phil opens the local editor at http://127.0.0.1:4321/edit/ and hits 'Pull now from Discord' which invokes /api/pull to drain the queue into src/content/actionitems/ markdown files. Astro dev server hot-reloads and the items appear on the master list pages.

Phase A pipeline — shipped 2026-04-17 as CHG-IT-013. Works end-to-end once Phil creates the two Discord channels and hands the IDs to Real Chuck for bot.js (done 2026-04-18).

Deep SOP: SOP-IT-011 in Claude_Lives_Here/SOPs/IT/

Google Drive Blood Panels — full procedure

Phil drops a new LabCorp PDF in the Drive folder. When he wants it ingested, he tells Real Chuck 'pull blood panels' in any Claude Code session. Chuck runs the blood-panel-puller one-liner: lists the folder via Google Drive API (service account auth), sends each unseen PDF to Claude (claude-sonnet-4-6 native PDF input), merges the returned JSON into src/content/bloodmarkers/*.md (append reading to values array, dedup by date) and writes src/content/panels/<date>.md. Astro hot-reloads. File IDs tracked in engelsplace-bloodpanel-seen.json. Scheduled polling is intentionally DISABLED — blood panels are infrequent, on-demand is the right cadence.

Phase E pipeline — code complete 2026-04-18. Auto-polling was briefly enabled (cron `*/15 * * * *`) then switched to ON-DEMAND per Phil's 2026-04-18 direction. All wiring (service account at IT/credentials/google-service-account.json, Drive folder shared with engelsplace-drive-reader@dev-access-393721.iam.gserviceaccount.com, puller module, handler registry) stays intact. Re-enabling auto-poll is a single `enabled: true` flip in scheduled-tasks.json + `pm2 restart engel-ops-bot` if Phil ever wants it back.

Deep SOP: SOP-IT-011 in Claude_Lives_Here/SOPs/IT/

Local Editor (engelsplace-dev) — full procedure

Astro dev server running as pm2 service engelsplace-dev bound to 127.0.0.1. Phil opens /edit/ in a browser, fills forms, saves; the SSR API endpoints in src/pages/api/ read/write .md files in src/content/ via markdown-io.ts helpers. Astro watches src/content/ and hot-reloads any page that reads those collections. LAN cannot reach — 127.0.0.1 binding only, so it is zero-risk to run 24/7.

Phase A editor — shipped 2026-04-17 as CHG-IT-013. Entry point for all manual content edits including status colors, reference ranges, and new marker metadata after the Phase E puller creates a stub.

Deep SOP: SOP-IT-011 in Claude_Lives_Here/SOPs/IT/

FMX Maintenance API — full procedure

Puller hits the FMX /api/v1/maintenance-requests endpoint with HTTP Basic auth (Dashboard Sync service user, Viewer role, scoped to building 110428). Pulls all tickets — open and Finalized — into src/content/maintenancerequests/ as one .md per ticket. /maintenance shows open tickets only; /maintenance-metrics computes on-time-rate + p50/p90/p99 cycle time + 24-mo volume from the closed history. Idempotent: re-runs are zero-write when no tickets changed; tickets that disappear from FMX get hard-deleted locally.

LIVE 2026-04-19 — Phase D first puller. Baseline pull: 383 tickets ingested cleanly on first run, zero errors. Idempotent re-pulls confirmed zero-diff.

**One-liner to refresh ticket data (Real Chuck only):**

```bash
cd C:\Users\engelp\Claude_Lives_Here\IT\discord-gateway-bot
node -e "require('dotenv').config({override:true}); const { runFmxPull } = require('./fmx-puller'); runFmxPull().then(s => console.log(JSON.stringify(s, null, 2)));"
```

**Known gotchas:**

- **FMX uses email as HTTP Basic username**, not the user's login handle. `FMX_API_USERNAME=fairriteworksync@gmail.com`. Sending `dashboard-sync` returns 401 silently.
- **FMX returns datetimes without `Z` suffix** (e.g. `"2026-04-13T12:52:07.503"`). Field names end in `*Utc` so UTC is implicit; `fmx-puller.js` calls `normalizeIsoUtc()` to add the suffix before storing, so Astro's Zod `datetime()` validator accepts the values.
- **Request type names require Admin role.** Viewer service user gets 403 on `/request-types`; metrics page surfaces type IDs only and notes the cross-reference cost.

Deep SOP: SOP-IT-011 in Claude_Lives_Here/SOPs/IT/

FMX Preventive Maintenance API — full procedure

Puller hits FMX /api/v1/planned-maintenance/tasks for the 130 PM task definitions and writes one .md per task into src/content/pmtasks/. Then pulls /api/v1/planned-maintenance/occurrences over a rolling 24-month window (~6,280 occurrences/quarter), pre-aggregates into src/data/pm-metrics.json. /preventive-maintenance reads the task collection for the open + due-soon list. /preventive-maintenance-metrics reads the JSON aggregates for on-time rate, by-month volume, by-quarter trend, by-user leaderboard, worst-on-time task leaderboard.

LIVE 2026-04-19. Phase D second puller (FMX maintenance-requests shipped earlier same day). Baseline pull: 130 PM tasks + 6,280 occurrences in 24-month rolling window, zero errors.

**One-liner to refresh PM data (Real Chuck only):**

```bash
cd C:\Users\engelp\Claude_Lives_Here\IT\discord-gateway-bot
node -e "require('dotenv').config({override:true}); const { runFmxPmPull } = require('./fmx-pm-puller'); runFmxPmPull().then(s => console.log(JSON.stringify(s, null, 2)));"
```

**Known gotchas (v1):**

- **Exclusion list not applied yet.** `pm-exclusion-list.json` uses `(?i)` inline regex syntax which Node's RegExp constructor rejects — defaults to 0 patterns loaded. All 6,280 occurrences are currently included in metrics. Patches needed: (a) update JSON to remove `(?i)` prefix, (b) update loader to always use `'i'` flag. Then headline rate will tighten to match Phil's manual quarterly report (96.92% Q4 2025).
- **Request type names show as IDs.** `/request-types` endpoint requires Admin role; Dashboard Sync Viewer gets 403. Workaround pending: hard-code id→name map.
- **Same FMX gotchas as maintenance-requests:** email-as-username for HTTP Basic, datetime strings without trailing Z (puller normalizes via `normalizeIsoUtc()`).

Deep SOP: SOP-IT-011 in Claude_Lives_Here/SOPs/IT/

Flat Rock Facility Meeting Minutes — full procedure

Daily 5 AM CDT cron fires gmail-minutes-puller. Searches fairriteworksync inbox for `subject:"Flat Rock Facility Meeting" has:attachment`, downloads each new .docx, invokes minutes-parser.py (python-docx), writes structured markdown to src/content/minutes/<YYYY-MM-DD>.md. Dedup via gmail-minutes-seen.json (Gmail message IDs). Each minutes-driven page reads the LATEST entry. Manual desktop-file drop still works as a fallback (run minutes-parser.py directly on a local .docx).

LIVE 2026-04-19. Phase D third puller (after FMX maintenance + FMX preventive-maintenance). Baseline: 3/30/26 meeting parsed cleanly, all 10 sections + 2 tables (evaluations, holidays) extracted to structured frontmatter.

**One-liner to parse a meeting docx (Real Chuck only):**

```bash
cd C:\Users\engelp\Claude_Lives_Here\IT\discord-gateway-bot
python minutes-parser.py "C:/path/to/Meeting_.docx"
```

Writes `src/content/minutes/.md` with parsed sections + raw lines body. Astro hot-reloads on file change.

**Pages driven by this pipeline:**

| Page | Section | Special render |
|------|---------|----------------|
| `/safety` | safety | nested list (groups by › headers) |
| `/5s-scorecard` | fiveS | KPI cards, color-coded by score (≥95 green, 80-94 amber, <80 red) |
| `/ftz` | ftz | nested list |
| `/kits` | kitAssembly | nested list |
| `/qc` | qc | nested list |
| `/shipping-receiving` | shipping | $ KPI cards (Last/This/Next week) + nested list |
| `/powder` | powder | nested list + cross-link to /maintenance + /preventive-maintenance |
| `/facility` | facility | nested list |
| `/evaluations` | evaluations | table; red row = overdue, amber = within 30d |
| `/holidays` | holidays | table with ★ marker; past dates faded |

**Known gotchas (v1):**

- **Parser fragility on template changes:** if the meeting template changes (new section, renamed emoji, restructured table), the parser will silently skip the new bits. Bot Resend-fallback should email Phil with the offending file when parsing yields fewer than the expected sections (TODO: wire this).
- **Python output uses Z suffix:** `2026-04-20T02:43:26Z`, NOT `+00:00` (Astro Zod `.datetime()` rejects offset format by default).
- **Cell newlines in special tables:** 5S scores arrive as "Grounds\\n72%" not "Grounds / 72%" — parser normalizes `\\n` to ` / ` before regex matching.

**Gmail puller — LIVE 2026-04-20**

`IT/discord-gateway-bot/gmail-puller.js` exports `runGmailMinutesPull()`. Flow on every fire:
1. Refresh access token from `IT/credentials/gmail-oauth-tokens.json` if expired (uses client_secret from `gmail-oauth-client.json`)
2. Search Gmail with query `subject:"Flat Rock Facility Meeting" has:attachment` (max 25 results)
3. For each message NOT in `gmail-minutes-seen.json`:
- Download every `.docx` attachment via `messages//attachments/` (base64url decode)
- Save to OS tmp dir
- Spawn `python minutes-parser.py ` as subprocess
- Capture parser stdout; record the `WROTE:` line in summary
4. Add processed message ID to seen set, persist to disk
5. Cleanup tmp dir
6. Post Discord summary to `#engelsplace`

**On-demand one-liner (Real Chuck only):**

```bash
cd C:\Users\engelp\Claude_Lives_Here\IT\discord-gateway-bot
node -e "require('dotenv').config({override:true}); const { runGmailMinutesPull } = require('./gmail-puller'); runGmailMinutesPull().then(s => console.log(JSON.stringify(s, null, 2)));"
```

**Auth setup is one-time:** completed 2026-04-19 via `gmail-oauth-setup.js` (PKCE + client_secret). Refresh token in `IT/credentials/gmail-oauth-tokens.json` lasts indefinitely (until revoked or 6 months unused).

**Failure mode:** if subject template changes, search returns 0 — manually adjust `GMAIL_QUERY` constant in `gmail-puller.js`. To force re-ingest of all matching emails, delete `gmail-minutes-seen.json`.

Deep SOP: SOP-IT-011 in Claude_Lives_Here/SOPs/IT/

Planned

Pipeline	Source	Destination	Schedule	Updated
Gmail QC News Email	`fairriteworksync@gmail.com (inbox, unprocessed label filter)`	`/qc`	Every 30 min once shipped (cron TBD)	`2026-04-18`
YouTube Playlists API	`Phil's YouTube account playlists (6 mapped)`	`/weightlifting + /health + /finance-videos + /ai-videos + /mma + /political-satire`	Every 6 hours once shipped (TBD — adjustable)	`2026-04-18`

Gmail QC News — specification

Puller polls the QC Gmail inbox, hands each message to Claude for classification (action_item | news | noise) + sensitivity check + redacted summary. Action items drop into src/content/actionitems/ with sourcePage: qc. News items drop into a new qcnews collection with published: false by default (Phil flips to true via /edit/ after review). Sensitive or low-confidence items route to a review queue. Gmail messages labeled 'engelsplace/processed' for dedup.

Phase D work — specified in Projects/engelsplace-rebuild/next-steps.md 2026-04-18, not yet implemented. Requires: Gmail OAuth refresh token (or service account delegation), Claude classification prompt, qcnews content collection, editor view at /edit/qcnews/.

YouTube Playlists — specification

Scheduled puller polls the YouTube Data API v3 for each configured playlist, diffs against a per-playlist seen list (video IDs), and for new videos fetches title + description + thumbnail + duration. Writes each as a markdown entry in the destination page's content collection with sourcePage and publishedAt metadata. Page renders as a card grid with thumbnail + title linking to YouTube. No video transcoding, no embedding — YouTube stays canonical, engelsplace is the curated index.

Phase D work — mapping locked 2026-04-18 per Phil's direction. Six YouTube playlists map to six engelsplace pages:

| YouTube playlist | engelsplace page |
|---|---|
| Weight lifting-Strength | /weightlifting |
| General health videos | /health |
| Financial-videos | /finance-videos |
| General AI videos | /ai-videos |
| mma-videos | /mma |
| political-satire | /political-satire |

All six pages are in the personal bucket and gated by default (Cloudflare Access per Phase 0 Model A). `/political-satire` flagged for extra care at deploy time — Phil accepted misclick risk 2026-04-18 after the default-skip recommendation; double-check Cloudflare Access rule applies before Phase 2 public deploy and consider a stricter policy (signed-in-as-Phil only) on that specific route.

Requires: YouTube Data API v3 credentials (API key for read-only public playlist access, OR OAuth refresh token if playlists are private). Phil's call on which auth path when Phase D ships. Service-account auth does NOT work for YouTube Data API — different platform from Drive.

Updating this document

This page reads from src/content/sop/*.md. Each pipeline is one markdown file. To add, modify, or retire a pipeline:

Edit the matching src/content/sop/<slug>.md (frontmatter: title, status, source, destinationPage, pipelineSummary, schedule, lastUpdated).
Bump the lastUpdated field to today (ISO format, e.g. 2026-04-18).
Update the companion text SOP at Claude_Lives_Here/SOPs/IT/SOP-IT-011-engelsplace-content-pipelines.md with the detailed narrative (setup, credentials, error handling).
Commit both files together with a message starting with sop:.

Rule: if a pipeline changes in code (cron expression, source id, destination slug, credential path) and the SOP is not updated in the same commit, the commit is incomplete.