# Phase 1G-metrics — OpenTelemetry + Custom Instruments Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Ship the OpenTelemetry runtime — server and browser initializers, `getMeter`/`getTracer` accessors, and the 8 custom metric instruments — so that 1F-layout, 1D, 1E, and all downstream features can emit structured metrics with `flightsApiError.add(1, { route })` in both SSR and client contexts.

**Architecture:** `otel.ts` is the **only** file allowed to import from `@opentelemetry/sdk-metrics` and `@opentelemetry/sdk-node` (enforced by 1A-3 ESLint boundaries). It exports `initServerOtel(env)` and `initBrowserOtel(env)` which wire the real `MeterProvider`/`TracerProvider`. `custom.ts` uses `@opentelemetry/api`'s proxy meter to declare instruments at module level — safe because the proxy lazy-resolves after init runs. `otel.ts` imports `Env` from `@/env` and `Logger` from `@/observability/logger/types`.

**Tech Stack:** `@opentelemetry/api`, `@opentelemetry/sdk-node`, `@opentelemetry/sdk-metrics`, `@opentelemetry/exporter-trace-otlp-http`, `@opentelemetry/exporter-metrics-otlp-http`, `web-vitals`.

**Prerequisites:** 1A-1 (skeleton + `Env` type), 1A-3 (ESLint boundaries), 1G-logger (Logger types).

---

## File structure

| File | Responsibility | Task |
|---|---|---|
| `src/observability/metrics/otel.ts` | OTel init + getMeter/getTracer | 2 |
| `src/observability/metrics/otel.test.ts` | Tests | 2 |
| `src/observability/metrics/custom.ts` | 8 custom metric instruments | 3 |

---

## Task 1 — Install OTel dependencies

**Files:**
- Modify: `package.json`

- [ ] **Step 1: Install dependencies**

```bash
pnpm add @opentelemetry/api @opentelemetry/sdk-node @opentelemetry/sdk-metrics @opentelemetry/exporter-trace-otlp-http @opentelemetry/exporter-metrics-otlp-http web-vitals
```

- [ ] **Step 2: Verify installation**

```bash
pnpm typecheck
```

- [ ] **Step 3: Commit**

```bash
git add package.json pnpm-lock.yaml
git commit -m "Add OpenTelemetry and web-vitals dependencies for metrics pipeline"
```

---

## Task 2 — TDD `otel.ts`

**Files:**
- Create: `src/observability/metrics/otel.ts`
- Create: `src/observability/metrics/otel.test.ts`

- [ ] **Step 1: Write failing tests**

Create `src/observability/metrics/otel.test.ts`:

```typescript
import { describe, expect, it, vi, beforeEach, afterEach } from "vitest";
import { metrics, trace } from "@opentelemetry/api";
import {
  InMemoryMetricExporter,
  AggregationTemporality,
  PeriodicExportingMetricReader,
} from "@opentelemetry/sdk-metrics";

describe("otel", () => {
  beforeEach(() => {
    // Reset global providers between tests
    metrics.disable();
    trace.disable();
  });

  afterEach(() => {
    metrics.disable();
    trace.disable();
    vi.restoreAllMocks();
  });

  it("initServerOtel registers a MeterProvider and TracerProvider", async () => {
    const { initServerOtel } = await import("./otel.js");

    initServerOtel({
      OTEL_EXPORTER_OTLP_ENDPOINT: "http://localhost:4318",
      OTEL_SERVICE_NAME: "flights-test",
      NODE_ENV: "test",
    } as any);

    // After init, getMeter should return a working meter
    const { getMeter, getTracer } = await import("./otel.js");
    const meter = getMeter("test");
    const tracer = getTracer("test");

    expect(meter).toBeDefined();
    expect(tracer).toBeDefined();
  });

  it("counter incremented via proxy meter is observable by test reader", async () => {
    const exporter = new InMemoryMetricExporter(AggregationTemporality.CUMULATIVE);
    const reader = new PeriodicExportingMetricReader({
      exporter,
      exportIntervalMillis: 100,
    });

    const { initServerOtelWithReader } = await import("./otel.js");
    initServerOtelWithReader({
      OTEL_EXPORTER_OTLP_ENDPOINT: "http://localhost:4318",
      OTEL_SERVICE_NAME: "flights-test",
      NODE_ENV: "test",
    } as any, reader);

    const counter = metrics.getMeter("flights").createCounter("test.counter");
    counter.add(1, { route: "/smoke" });

    // Force a collection cycle
    await reader.forceFlush();

    const exported = exporter.getMetrics();
    expect(exported.length).toBeGreaterThan(0);

    const testMetric = exported
      .flatMap((rm) => rm.scopeMetrics)
      .flatMap((sm) => sm.metrics)
      .find((m) => m.descriptor.name === "test.counter");

    expect(testMetric).toBeDefined();

    await reader.shutdown();
  });

  it("getMeter returns a meter from @opentelemetry/api", async () => {
    const { getMeter } = await import("./otel.js");
    const meter = getMeter("my-component");
    expect(meter).toBeDefined();
    expect(typeof meter.createCounter).toBe("function");
    expect(typeof meter.createHistogram).toBe("function");
  });

  it("getTracer returns a tracer from @opentelemetry/api", async () => {
    const { getTracer } = await import("./otel.js");
    const tracer = getTracer("my-component");
    expect(tracer).toBeDefined();
    expect(typeof tracer.startSpan).toBe("function");
  });
});
```

- [ ] **Step 2: Run — MUST FAIL**

```bash
pnpm test src/observability/metrics/otel
```

- [ ] **Step 3: Write implementation**

Create `src/observability/metrics/otel.ts`:

```typescript
import { metrics, trace } from "@opentelemetry/api";
import type { Meter, Tracer } from "@opentelemetry/api";
import { NodeSDK } from "@opentelemetry/sdk-node";
import { MeterProvider, PeriodicExportingMetricReader } from "@opentelemetry/sdk-metrics";
import type { MetricReader } from "@opentelemetry/sdk-metrics";
import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-http";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
import type { Env } from "@/env";
import type { Logger } from "@/observability/logger/types";

let initialized = false;

/**
 * Initialize OpenTelemetry for the server (Node) process.
 * Called once per process at startup.
 */
export function initServerOtel(env: Env): void {
  if (initialized) return;

  const endpoint = env.OTEL_EXPORTER_OTLP_ENDPOINT;
  const serviceName = env.OTEL_SERVICE_NAME ?? "flights-web";

  const metricReader = new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter({ url: `${endpoint}/v1/metrics` }),
    exportIntervalMillis: 15_000,
  });

  const sdk = new NodeSDK({
    serviceName,
    traceExporter: new OTLPTraceExporter({ url: `${endpoint}/v1/traces` }),
    metricReader,
  });

  sdk.start();
  initialized = true;
}

/**
 * Test-only variant that accepts a custom MetricReader for in-memory assertions.
 */
export function initServerOtelWithReader(env: Env, reader: MetricReader): void {
  if (initialized) return;

  const serviceName = (env as Record<string, string>).OTEL_SERVICE_NAME ?? "flights-test";

  const meterProvider = new MeterProvider({
    readers: [reader],
  });

  metrics.setGlobalMeterProvider(meterProvider);
  initialized = true;
}

/**
 * Initialize OpenTelemetry for the browser.
 * Called once per tab via useEffect in the root layout.
 * Browser-side uses web-vitals to report CWV as histograms.
 */
export function initBrowserOtel(env: Env): void {
  if (initialized) return;

  const endpoint = env.OTEL_EXPORTER_OTLP_ENDPOINT;
  if (!endpoint) return;

  const meterProvider = new MeterProvider({
    readers: [
      new PeriodicExportingMetricReader({
        exporter: new OTLPMetricExporter({ url: `${endpoint}/v1/metrics` }),
        exportIntervalMillis: 30_000,
      }),
    ],
  });

  metrics.setGlobalMeterProvider(meterProvider);

  // Report web-vitals as OTel histograms
  const cwvMeter = meterProvider.getMeter("web-vitals");
  void import("web-vitals").then(({ onCLS, onFID, onLCP, onFCP, onTTFB }) => {
    const cls = cwvMeter.createHistogram("web_vitals.cls");
    const fid = cwvMeter.createHistogram("web_vitals.fid");
    const lcp = cwvMeter.createHistogram("web_vitals.lcp");
    const fcp = cwvMeter.createHistogram("web_vitals.fcp");
    const ttfb = cwvMeter.createHistogram("web_vitals.ttfb");

    onCLS((m) => cls.record(m.value));
    onFID((m) => fid.record(m.value));
    onLCP((m) => lcp.record(m.value));
    onFCP((m) => fcp.record(m.value));
    onTTFB((m) => ttfb.record(m.value));
  });

  initialized = true;
}

/** Returns a named Meter from the global MeterProvider. */
export function getMeter(name: string): Meter {
  return metrics.getMeter(name);
}

/** Returns a named Tracer from the global TracerProvider. */
export function getTracer(name: string): Tracer {
  return trace.getTracer(name);
}
```

- [ ] **Step 4: Run — ALL MUST PASS**

```bash
pnpm test src/observability/metrics/otel
```

- [ ] **Step 5: Typecheck + lint, commit**

```bash
pnpm typecheck && pnpm lint
git add src/observability/metrics/otel.ts src/observability/metrics/otel.test.ts
git commit -m "Add OTel server/browser initializers with getMeter/getTracer accessors"
```

---

## Task 3 — Create `custom.ts` (declarative, no TDD)

**Files:**
- Create: `src/observability/metrics/custom.ts`

- [ ] **Step 1: Write implementation**

Create `src/observability/metrics/custom.ts`:

```typescript
import { metrics } from "@opentelemetry/api";

/**
 * Module-level metric instruments for the flights remote component.
 * Safe to declare at module scope — @opentelemetry/api's proxy meter
 * lazy-resolves to the real MeterProvider after initServerOtel/initBrowserOtel runs.
 */
const meter = metrics.getMeter("flights");

/** SSR request duration histogram (seconds). */
export const flightsSsrRequestDuration = meter.createHistogram("flights.ssr.request.duration");

/** Upstream API request duration histogram (seconds). */
export const flightsApiRequestDuration = meter.createHistogram("flights.api.request.duration");

/** Upstream API error counter (by route, status). */
export const flightsApiError = meter.createCounter("flights.api.error");

/** SignalR active connections gauge. */
export const flightsSignalRConnected = meter.createUpDownCounter("flights.signalr.connected");

/** SignalR messages received counter. */
export const flightsSignalRMessageReceived = meter.createCounter("flights.signalr.message.received");

/** SignalR disconnection counter (by reason). */
export const flightsSignalRDisconnect = meter.createCounter("flights.signalr.disconnect");

/** Feature component render counter (by feature name). */
export const flightsFeatureRender = meter.createCounter("flights.feature.render");

/** Unhandled React error counter (caught by ErrorBoundary). */
export const flightsReactError = meter.createCounter("flights.react.error");
```

- [ ] **Step 2: Typecheck + lint, commit**

```bash
pnpm typecheck && pnpm lint
git add src/observability/metrics/custom.ts
git commit -m "Add 8 custom metric instruments using OTel proxy meter"
```

---

## Task 4 — Exit-gate verification

- [ ] **Step 1: All gates**

```bash
pnpm typecheck && pnpm lint && pnpm test
```

Expected: all pass. OTel init test proves counter is observable via test reader.

- [ ] **Step 2: Git status clean**

```bash
git status
```

---

## Self-review

**Spec coverage.** Master plan §1G-metrics:
- `initServerOtel(env)` / `initBrowserOtel(env)` — Task 2
- `getMeter(name)` / `getTracer(name)` — Task 2
- 8 custom instruments (`flights.ssr.request.duration`, `flights.api.request.duration`, `flights.api.error`, `flights.signalr.connected`, `flights.signalr.message.received`, `flights.signalr.disconnect`, `flights.feature.render`, `flights.react.error`) — Task 3
- web-vitals histograms created inside `initBrowserOtel` — Task 2
- `otel.ts` is the only file importing `@opentelemetry/sdk-metrics` / `@opentelemetry/sdk-node` — enforced by 1A-3 ESLint rule, verified at exit gate

**Import boundary.** `otel.ts` imports `Env` from `@/env` and `Logger` from `@/observability/logger/types`. `custom.ts` imports only from `@opentelemetry/api` (public API, no SDK).

**Type consistency.** `Meter`, `Tracer` from `@opentelemetry/api`. `Env` from `@/env` (seeded in 1A-1).
