Content Pipelines
Living index of every pipeline that writes content into engelsplace. Source of truth for what pulls from where, when, and into which page.
Each row below links to a dedicated SOP file for the pipeline's full setup,
credentials, error handling, and recovery steps. When any pipeline's
schedule, source, destination, or status changes, the corresponding
src/content/sop/<slug>.md file must be updated in
the same commit as the code change, and SOP-IT-011 in
Claude_Lives_Here/SOPs/IT/ gets the detailed narrative.
Live
| Pipeline | Source | Destination | Schedule | Updated |
|---|---|---|---|---|
| Discord Action Items Discord | #work-action-items + #personal-action-items (Discord) | /work-actions + /personal-actions | On-demand (Phil-triggered pull); bot listener is always-on | 2026-04-18 |
| Google Drive Blood Panels Drive | Drive folder 'blood-panels' (id 1Wdt3JumRZzy54yovlpRlgnrJwwjElelr) | /blood-panels | On-demand (Phil tells Real Chuck 'pull blood panels') | 2026-08-10 |
| Local Editor (engelsplace-dev) Manual | http://127.0.0.1:4321/edit/ (Phil's laptop/desktop only) | All collections — actionitems, bloodmarkers, panels, etc. | Always-on (pm2 service); on-demand writes | 2026-04-18 |
| FMX Maintenance API API | fair-rite-products-corp.gofmx.com/api/v1 (filtered to building 110428) | /maintenance + /maintenance-metrics | Automated cron, Mon–Fri: 6:02 AM (engelsplace-fmx-ingest-morning) + 11:02 AM (engelsplace-fmx-ingest-midday) + 2:45 PM CT (engelsplace-fmx-ingest-afternoon) | 2026-07-24 |
| FMX Preventive Maintenance API API | fair-rite-products-corp.gofmx.com/api/v1 (filtered to building 110428) | /preventive-maintenance + /preventive-maintenance-metrics | Automated cron, Mon–Fri: 6:02 AM (engelsplace-fmx-pm-ingest-morning) + 11:02 AM (engelsplace-fmx-pm-ingest-midday) + 2:45 PM CT (engelsplace-fmx-pm-ingest-afternoon) | 2026-07-24 |
| YouTube Playlists API | Phil's YouTube channel playlists (7 mapped, channel UCSIzzqXwh2FVyevkvzBOT-Q) | /weightlifting + /health + /peptides + /finance-videos + /ai-videos + /mma + /political-satire | Daily 6:00 AM CT cron (engelsplace-youtube-ingest) | 2026-07-14 |
| Flat Rock Facility Meeting Minutes Email | fairriteworksync@gmail.com weekly meeting minutes .docx (auto-pulled via Gmail OAuth as of 2026-04-20) | /minutes + /work-actions + /safety + /5s-scorecard + /ftz + /kits + /qc + /shipping-receiving + /powder + /facility + /evaluations + /holidays | Daily 5 AM CT cron (engelsplace-gmail-minutes-ingest) + on-demand 'Pull work minutes now' button on /work-actions (local dev only, added 2026-06-08). Meeting Mondays; minutes typically arrive Tue/Wed. | 2026-07-14 |
| Problem Ledger & Operations Dashboard Manual | src/content/problems/ (canonical), written by IT/scripts/problem.js — IT/problems/ is legacy | /ops-dashboard | Event-driven. Agents create/update via node IT/scripts/problem.js; each write self-commits and pushes. | 2026-07-14 |
| Quarterly MTTR Emailer Manual | Projects/fair-rite-maintenance/mttr-log.xlsx (Phil hand-classifies production-downtime Y/N) + FMX PM data | /mttr | Quarterly: 6:00 AM CT, days 1-5 of Jan/Apr/Jul/Oct (emailer) + days 23-25 of Mar/Jun/Sep/Dec (reminder) | 2026-07-14 |
| Infographic Topic Requests Email | /infographics 'Request a Topic' button → tess@engelsplace.com (Cloudflare Email Worker) + #engelsplace Discord watcher | /personal-actions → Problem Ledger work order → /ops-dashboard → published page at /infographics/<slug> | Visitor/Phil-triggered capture; Discord watcher once daily 22:38 CT (infographic-request-watch, bot-side handler); ledger reconcile at every Tess boot | 2026-08-07 |
| NAS watch page API | QNAP HTTP API + SMB free space + local HDP/VSS probes (nas-watch-emit.js) | /personal/nas | Every nas-watch-routine fire (~7 AM CDT), including clean runs | 2026-08-13 |
Discord Action Items — full procedure
Phil posts a message in either action-items channel from any device. The engel-ops-bot messageCreate listener captures it, writes to IT/discord-gateway-bot/engelsplace-discord-queue.json, and returns a confirmation reaction. Phil opens the local editor at http://127.0.0.1:4321/edit/ and hits 'Pull now from Discord' which invokes /api/pull to drain the queue into src/content/actionitems/ markdown files. Astro dev server hot-reloads and the items appear on the master list pages.
Deep SOP: SOP-IT-011 in Claude_Lives_Here/SOPs/IT/
Google Drive Blood Panels — full procedure
Phil drops a new LabCorp PDF in the Drive folder. When he wants it ingested, he tells Real Chuck 'pull blood panels' in any Claude Code session. Chuck runs the blood-panel-puller one-liner: lists the folder via Google Drive API (service account auth), sends each unseen PDF to Claude (claude-opus-4-8 native PDF input), merges the returned JSON into src/content/bloodmarkers/*.md (append reading to values array, dedup by date) and writes src/content/panels/<date>.md. Astro hot-reloads. File IDs tracked in engelsplace-bloodpanel-seen.json. Scheduled polling is intentionally DISABLED — blood panels are infrequent, on-demand is the right cadence. AFTER ingest the puller runs the engelsplace repo's check-panel-interpretation-freshness.mjs and reports the verdict on every run: this pipeline updates the NUMBERS but never the hand-authored INTERPRETATION layer (panel-action-items + panel-context), so a new panel makes that reading stale by definition (P-01422).
## The interpretation layer is NOT auto-updated (P-01422, 2026-08-10)
This pipeline ingests **numbers only** — `bloodmarkers` and `panels`. The clinical reading on `/blood-panels` lives in two hand-authored collections, `panel-action-items` and `panel-context`, and **nothing regenerates them**. So the moment a new panel lands, the previous panel's guidance is rendering next to the new results. On 2026-08-10 that shipped a page telling Phil to "Start Iron — Immediately" and "Restart Statin — This Week" when both were long resolved (ferritin 12 → 63, ApoB 92 → 61).
**What now prevents a silent repeat:**
- Both collections require a **`panelDate`** field naming the panel the text was written against. Required, not optional — an omitted field fails at authoring time instead of rendering as fresh.
- `/blood-panels` renders a **banner naming every stale item**, plus per-item marking, with the loud variant when a stale item carries an urgent order.
- `scripts/check-panel-interpretation-freshness.mjs` — exit `0` current / `1` stale **or ahead** / `2` could-not-check. Run by hand with `npm run check:panel-freshness`. Four verdicts: `current`, `stale` (predates the newest panel), `ahead` (dated *after* every panel — legitimate when guidance is written at a clinic visit before the PDF lands, but reported rather than silently trusted, since one typo'd year would otherwise hide an urgent order for a year), and `unknown`. **`unknown` is never folded into `current`** — it covers a missing or empty directory, an undated or malformed panel date, and a missing `panelDate`.
- The comparison lives in `src/lib/panel-interpretation-freshness.mjs` and is imported by **both** the page and the CLI, so the banner and the check cannot drift apart.
- The puller runs that check after every ingest and posts the verdict to Discord on **every** run, clean ones included, so silence is never ambiguous.
**This deliberately does NOT fail the build.** `assertPanelCoverage` throws because it fires on an authoring mistake fixable in minutes. This condition is created by routine ingestion and cleared only by clinical review, so a throw would freeze the entire site — every collection, every page — on a schedule set by a lab. That is the P-00810 / P-01211 / P-01333 class that left the live site 136 commits stale. Staleness here is data, not an exception.
**Owner of clearing it:** Ashley (health lane) — the reading needs a clinical review against the new panel. Tess owns the machinery.
Deep SOP: SOP-IT-011 in Claude_Lives_Here/SOPs/IT/
Local Editor (engelsplace-dev) — full procedure
Astro dev server running as pm2 service engelsplace-dev bound to 127.0.0.1. Phil opens /edit/ in a browser, fills forms, saves; the SSR API endpoints in src/pages/api/ read/write .md files in src/content/ via markdown-io.ts helpers. Astro watches src/content/ and hot-reloads any page that reads those collections. LAN cannot reach — 127.0.0.1 binding only, so it is zero-risk to run 24/7.
Deep SOP: SOP-IT-011 in Claude_Lives_Here/SOPs/IT/
FMX Maintenance API — full procedure
Puller hits the FMX /api/v1/maintenance-requests endpoint with HTTP Basic auth (Dashboard Sync service user, Viewer role, scoped to building 110428). Pulls all tickets — open and Finalized — into src/content/maintenancerequests/ as one .md per ticket. /maintenance shows open tickets only; /maintenance-metrics computes on-time-rate + p50/p90/p99 cycle time + 24-mo volume from the closed history. Idempotent: re-runs are zero-write when no tickets changed; tickets that disappear from FMX get hard-deleted locally.
**One-liner to refresh ticket data (Real Chuck only):**
```bash
cd C:\Users\engelp\Claude_Lives_Here\IT\discord-gateway-bot
node -e "require('dotenv').config({override:true}); const { runFmxPull } = require('./fmx-puller'); runFmxPull().then(s => console.log(JSON.stringify(s, null, 2)));"
```
**Known gotchas:**
- **FMX uses email as HTTP Basic username**, not the user's login handle. `FMX_API_USERNAME=fairriteworksync@gmail.com`. Sending `dashboard-sync` returns 401 silently.
- **FMX returns datetimes without `Z` suffix** (e.g. `"2026-04-13T12:52:07.503"`). Field names end in `*Utc` so UTC is implicit; `fmx-puller.js` calls `normalizeIsoUtc()` to add the suffix before storing, so Astro's Zod `datetime()` validator accepts the values.
- **Request type names require Admin role.** Viewer service user gets 403 on `/request-types`; metrics page surfaces type IDs only and notes the cross-reference cost.
Deep SOP: SOP-IT-011 in Claude_Lives_Here/SOPs/IT/
FMX Preventive Maintenance API — full procedure
Puller hits FMX /api/v1/planned-maintenance/tasks for the 130 PM task definitions and writes one .md per task into src/content/pmtasks/. Then pulls /api/v1/planned-maintenance/occurrences over a rolling 24-month window (~6,280 occurrences/quarter), pre-aggregates into src/data/pm-metrics.json. /preventive-maintenance reads the task collection for the open + due-soon list. /preventive-maintenance-metrics reads the JSON aggregates for on-time rate, by-month volume, by-quarter trend, by-user leaderboard, worst-on-time task leaderboard.
**One-liner to refresh PM data (Real Chuck only):**
```bash
cd C:\Users\engelp\Claude_Lives_Here\IT\discord-gateway-bot
node -e "require('dotenv').config({override:true}); const { runFmxPmPull } = require('./fmx-pm-puller'); runFmxPmPull().then(s => console.log(JSON.stringify(s, null, 2)));"
```
**Known gotchas (v1):**
- **Exclusion list not applied yet.** `pm-exclusion-list.json` uses `(?i)` inline regex syntax which Node's RegExp constructor rejects — defaults to 0 patterns loaded. All 6,280 occurrences are currently included in metrics. Patches needed: (a) update JSON to remove `(?i)` prefix, (b) update loader to always use `'i'` flag. Then headline rate will tighten to match Phil's manual quarterly report (96.92% Q4 2025).
- **Request type names show as IDs.** `/request-types` endpoint requires Admin role; Dashboard Sync Viewer gets 403. Workaround pending: hard-code id→name map.
- **Same FMX gotchas as maintenance-requests:** email-as-username for HTTP Basic, datetime strings without trailing Z (puller normalizes via `normalizeIsoUtc()`).
Deep SOP: SOP-IT-011 in Claude_Lives_Here/SOPs/IT/
YouTube Playlists — full procedure
Daily cron (engelsplace-youtube-ingest, bot scheduled task) polls the YouTube Data API v3 for each public playlist on Phil's channel, diffs against a per-playlist seen list (video IDs), and for new videos fetches title + description + thumbnail + duration. Writes each as src/content/videos/<playlistSlug>/<videoId>.md with sourcePage and publishedAt metadata. Cross-playlist dedup (added 2026-06-01): a video saved to two playlists files under exactly one category by playlist priority. Pages render as card grids with thumbnail + title linking to YouTube. No video transcoding, no embedding — YouTube stays canonical, engelsplace is the curated index.
| YouTube playlist | engelsplace page |
|---|---|
| Weight lifting-Strength | /weightlifting |
| General health videos | /health |
| Peptides | /peptides |
| Financial-videos | /finance-videos |
| General AI videos | /ai-videos |
| mma-videos | /mma |
| political-satire | /political-satire |
Cross-playlist dedup since 2026-06-01: a video appearing in two playlists produces one .md, filed under the highest-priority playlist. Auth: YouTube Data API v3 key (read-only, public playlists). Quota ~1 unit per playlistItems.list page.
Deep SOP: SOP-IT-011 in Claude_Lives_Here/SOPs/IT/
Flat Rock Facility Meeting Minutes — full procedure
Daily 5 AM CDT cron fires gmail-minutes-puller. Searches fairriteworksync inbox for `subject:"Flat Rock Facility Meeting" has:attachment`, downloads each new .docx, invokes minutes-parser.py (python-docx), writes structured markdown to src/content/minutes/<YYYY-MM-DD>.md. Dedup via gmail-minutes-seen.json (Gmail message IDs). Each minutes-driven page reads the LATEST entry. Manual desktop-file drop still works as a fallback (run minutes-parser.py directly on a local .docx).
**One-liner to parse a meeting docx (Real Chuck only):**
```bash
cd C:\Users\engelp\Claude_Lives_Here\IT\discord-gateway-bot
python minutes-parser.py "C:/path/to/Meeting_
```
Writes `src/content/minutes/
**Pages driven by this pipeline:**
| Page | Section | Special render |
|------|---------|----------------|
| `/safety` | safety | nested list (groups by › headers) |
| `/5s-scorecard` | fiveS | KPI cards, color-coded by score (≥95 green, 80-94 amber, <80 red) |
| `/ftz` | ftz | nested list |
| `/kits` | kitAssembly | nested list |
| `/qc` | qc | nested list |
| `/shipping-receiving` | shipping | $ KPI cards (Last/This/Next week) + nested list |
| `/powder` | powder | nested list + cross-link to /maintenance + /preventive-maintenance |
| `/facility` | facility | nested list |
| `/evaluations` | evaluations | table; red row = overdue, amber = within 30d |
| `/holidays` | holidays | table with ★ marker; past dates faded |
**Known gotchas (v1):**
- **Parser fragility on template changes:** if the meeting template changes (new section, renamed emoji, restructured table), the parser will silently skip the new bits. Bot Resend-fallback should email Phil with the offending file when parsing yields fewer than the expected sections (TODO: wire this).
- **Python output uses Z suffix:** `2026-04-20T02:43:26Z`, NOT `+00:00` (Astro Zod `.datetime()` rejects offset format by default).
- **Cell newlines in special tables:** 5S scores arrive as "Grounds\\n72%" not "Grounds / 72%" — parser normalizes `\\n` to ` / ` before regex matching.
**Gmail puller — LIVE 2026-04-20**
`IT/discord-gateway-bot/gmail-puller.js` exports `runGmailMinutesPull()`. Flow on every fire:
1. Refresh access token from `IT/credentials/gmail-oauth-tokens.json` if expired (uses client_secret from `gmail-oauth-client.json`)
2. Search Gmail with query `subject:"Flat Rock Facility Meeting" has:attachment` (max 25 results)
3. For each message NOT in `gmail-minutes-seen.json`:
- Download every `.docx` attachment via `messages/
- Save to OS tmp dir
- Spawn `python minutes-parser.py
- Capture parser stdout; record the `WROTE:` line in summary
4. Add processed message ID to seen set, persist to disk
5. Cleanup tmp dir
6. Post Discord summary to `#engelsplace`
**On-demand one-liner (Real Chuck only):**
```bash
cd C:\Users\engelp\Claude_Lives_Here\IT\discord-gateway-bot
node -e "require('dotenv').config({override:true}); const { runGmailMinutesPull } = require('./gmail-puller'); runGmailMinutesPull().then(s => console.log(JSON.stringify(s, null, 2)));"
```
**Auth setup is one-time:** completed 2026-04-19 via `gmail-oauth-setup.js` (PKCE + client_secret). Refresh token in `IT/credentials/gmail-oauth-tokens.json` lasts indefinitely (until revoked or 6 months unused).
**Failure mode:** if subject template changes, search returns 0 — manually adjust `GMAIL_QUERY` constant in `gmail-puller.js`. To force re-ingest of all matching emails, delete `gmail-minutes-seen.json`.
Deep SOP: SOP-IT-011 in Claude_Lives_Here/SOPs/IT/
Problem Ledger & Operations Dashboard — full procedure
A unified system tracking preventive maintenance tasks, system problems, and active work orders. problem.js writes one .md per ticket into the problems content collection and auto-commits each ledger write (pathspec-scoped to the single file, since 2026-05-31). The dashboard reads the collection to build live reports, compute metrics, and display on-time resolution statistics.
### Boot Hook Loading
When any agent boots, the Discord gateway bot (`bot.js`) reads all problems in the collection to populate dynamic state for LLM context.
### Command Line Maintenance
Agents and the owner can update status, assignees, and resolution logs via the command-line helper:
```bash
node IT/scripts/problem.js list --active
node IT/scripts/problem.js resolve --id=00090 --notes="P-00090 tunnel resolved after cloudflared restart"
```
Deep SOP: SOP-IT-011 in Claude_Lives_Here/SOPs/IT/
Quarterly MTTR Emailer — full procedure
Quarterly cron (engelsplace-quarterly-emailer) fires the first business days of Jan/Apr/Jul/Oct: quarterly-emailer-handler.js parses mttr-log.xlsx via mttr-log-parser.py, computes MTTR and PM on-time metrics for the closed quarter, emails the report, and archives it as a page under /mttr. A companion reminder cron (engelsplace-quarterly-reminder) nudges Phil on Mar/Jun/Sep/Dec 23-25 to finish hand-classifying downtime rows before quarter close. MTTR source switched from the FMX API to the hand-classified spreadsheet 2026-04-22 per Phil.
**Known gotchas:**
- MTTR truth lives in `Projects/fair-rite-maintenance/mttr-log.xlsx` — Phil hand-classifies each row's production-downtime Y/N. If rows are unclassified at quarter close, the report flags omissions (methodology disclosure section added 2026-04-24).
- `fmx-pm-puller` date-compare bug fixed 2026-04-24 (strict on-time 25% → 59.77%); numbers before that date are not comparable.
Deep SOP: SOP-IT-011 in Claude_Lives_Here/SOPs/IT/
Infographic Topic Requests — full procedure
Two capture paths feed one queue. Path 1 (website): the Request a Topic button opens a mailto to tess@engelsplace.com; the Cloudflare Email Worker (engelsplace/workers/tess-email-worker.js) commits an infographic-request action item into src/content/actionitems/, surfacing at /personal-actions. Path 2 (Discord): IT/scripts/infographic-request-watch.js scans #engelsplace once daily (22:38 CT) for topic requests Phil typed directly, dedups against built slugs and existing items, and files the same action-item shape. Since 2026-08-06 (JOC-0039/P-01291) this runs as a bot-side deterministic handler inside the always-on engel-ops-bot (infographic-request-watch-handler.js) — $0, no LLM boot — replacing the retired Claude Code scheduled task. A PC-side reconciler (IT/scripts/tess-infographic-request-to-ledger.js, runs at every Tess boot) converts open request items into tracked Problem Ledger work orders. Phil gates each build with 'go'; Tess builds via the infographic skill and publishes to /infographics.
**Pieces:**
| Piece | Where |
|---|---|
| Email worker | `engelsplace/workers/tess-email-worker.js` (Cloudflare) |
| Discord watcher | `IT/scripts/infographic-request-watch.js` (bot-side handler `infographic-request-watch-handler.js` in engel-ops-bot, cron `38 22 * * *` = 22:38 CT daily) |
| Boot reconciler | `IT/scripts/tess-infographic-request-to-ledger.js` (idempotent, sidecar map `IT/status/infographic-request-work-orders.json`) |
| Build skill | `agents/tess/skills/infographic/SKILL.md` (11-section template, citation-integrity gate) |
**Known gotchas:**
- The mailto template is loosely structured — topics typed into the questions blob can produce messy auto-parsed titles.
- Requester names NEVER appear on the published page (privacy rule, banked 2026-06-01).
- A request submitted while Tess is offline waits for the next boot reconcile; the Discord watcher closes most of that gap.
Deep SOP: SOP-IT-011 in Claude_Lives_Here/SOPs/IT/
NAS watch page — full procedure
Kara's NAS watch routine polls philsserver every morning. Every fire, including clean ones, runs nas-watch-emit.js --ingest, which writes a structured snapshot and Tess copies it into src/data/nas-latest.json plus nas-history.json. /personal/nas renders plain English on top and charts underneath. Unknown values stay labeled not known. The watch routine does not publish; Tess owns the git push gate.
Deep SOP: SOP-IT-011 in Claude_Lives_Here/SOPs/IT/
Planned
| Pipeline | Source | Destination | Schedule | Updated |
|---|---|---|---|---|
| Gmail QC News Email | fairriteworksync@gmail.com (inbox, unprocessed label filter) | /qc | Every 2 hours, weekdays only, 8 AM – 6 PM CDT (6 runs/day: 8/10/12/2/4/6 PM). Cron once shipped: 0 13,15,17,19,21,23 * * 1-5 (UTC). | 2026-05-12 |
Gmail QC News — specification
Puller polls the QC Gmail inbox, hands each message to Claude for classification (action_item | news | noise) + sensitivity check + redacted summary. Action items drop into src/content/actionitems/ with sourcePage: qc. News items drop into a new qcnews collection with published: false by default (Phil flips to true via /edit/ after review). Sensitive or low-confidence items route to a review queue. Gmail messages labeled 'engelsplace/processed' for dedup.
Updating this document
This page reads from src/content/sop/*.md. Each pipeline is
one markdown file. To add, modify, or retire a pipeline:
- Edit the matching
src/content/sop/<slug>.md(frontmatter: title, status, source, destinationPage, pipelineSummary, schedule, lastUpdated). - Bump the
lastUpdatedfield to today (ISO format, e.g.2026-04-18). - Update the companion text SOP at
Claude_Lives_Here/SOPs/IT/SOP-IT-011-engelsplace-content-pipelines.mdwith the detailed narrative (setup, credentials, error handling). - Commit both files together with a message starting with
sop:.
Rule: if a pipeline changes in code (cron expression, source id, destination slug, credential path) and the SOP is not updated in the same commit, the commit is incomplete.