Jerrett DavisThe Overengineer
SiteSite live0 previewsPipelinesLoading

Docs

Telemetry and APIs

Live data surfaces, API contracts, and caching strategy.

Back to docsHome
Always on, never noisy
Telemetry routes keep the site feeling alive without hammering external APIs. Each endpoint is cached, rate-limit aware, and designed for safe manual refreshes.
GitHub Vercel Cache aware Rate limited

API surface map

Endpoint Purpose Refresh
/api/site-build-status Vercel production + preview deployment status. ?refresh=1 (cooldown enforced)
/api/pipeline-status GitHub Actions run summaries for active repos. ?refresh=1 (cooldown enforced)
/api/project-details README snapshots, issues, pulls, latest release, optional NuGet stats. ?repo=OWNER/REPO + optional refresh=1
/api/project-details-refresh Webhook-friendly refresh trigger for a single repo. ?repo=OWNER/REPO (supports bearer token)
/api/github-metrics Historic commit, LOC, star, fork trends. ?refresh=1 (uses metrics store)
/api/github-metrics-update Kick off a background metrics refresh. GET/POST; optional token or bearer auth
/api/nuget-metrics Aggregated NuGet downloads for configured packages. ?refresh=1 (cooldown enforced)
/api/github-metrics-repo Refresh a single repo metrics entry. ?repo=OWNER/REPO (cooldown enforced)
/api/search-index Unified search payload for posts + nav. Cached for 5 minutes

Caching layers

Server cache

  • lib/cacheStore.ts centralizes in-memory cache entries.
  • /api/project-details can persist snapshots in Postgres for cross-instance reuse.
  • Requests dedupe with an in-flight promise to prevent double-fetching.
  • Manual refreshes use cooldown checks to avoid spikes.

Client cache

  • lib/telemetryStore.ts stores payloads in localStorage.
  • Cached payloads render instantly, then refresh in the background.
  • Refresh cadence adapts based on live state (e.g., active builds).

Rate limits + fallbacks

  • GitHub and Vercel responses are inspected for rate-limit headers.
  • When rate limited, the APIs return cached data plus a rateLimitedUntil timestamp.
  • Clients display cooldown messaging and keep existing data visible.
  • Server responses include Retry-After headers when appropriate.

Default TTLs

Endpoint Fresh TTL Stale window
/api/project-details Global TTL Global stale window
/api/pipeline-status Global TTL (active uses PIPELINE_ACTIVE_CACHE_TTL_MS) Global stale window
/api/site-build-status Global TTL (active uses SITE_BUILD_ACTIVE_CACHE_TTL_MS) Global stale window
/api/github-metrics Global TTL Global stale window
/api/nuget-metrics Global TTL (override recommended) Global stale window
/api/search-index 5 minutes 15 minutes

Tip: set NUGET_CACHE_TTL_MS and NUGET_CACHE_STALE_MS to keep NuGet refreshes less frequent (e.g., 6h/24h).

Global cache defaults

All external-API endpoints share a global cache baseline, with per-endpoint overrides when needed.

Variable Purpose Notes
GLOBAL_CACHE_TTL_MS Default fresh TTL Applies to API caches unless an endpoint override is set (default 5 minutes).
GLOBAL_CACHE_STALE_MS Default stale window Controls how long stale responses are served (default 60 minutes).
Endpoint override keys
  • PROJECT_DETAIL_CACHE_TTL_MS / PROJECT_DETAIL_CACHE_STALE_MS
  • PIPELINE_CACHE_TTL_MS / PIPELINE_CACHE_STALE_MS
  • PIPELINE_ACTIVE_CACHE_TTL_MS
  • SITE_BUILD_CACHE_TTL_MS / SITE_BUILD_CACHE_STALE_MS
  • SITE_BUILD_ACTIVE_CACHE_TTL_MS
  • GITHUB_METRICS_CACHE_TTL_MS / GITHUB_METRICS_CACHE_STALE_MS
  • NUGET_CACHE_TTL_MS / NUGET_CACHE_STALE_MS

Metrics store + persistence

GitHub metrics history is stored centrally so charts survive redeploys.

  • 01. File store
    data/githubMetricsHistory.json is the default local store.
  • 02. SQLite (local)
    Default local adapter when no overrides are set.
  • 03. Postgres (prod)
    Automatically selected in production when a Postgres URL is available.
Adapter rules
  • METRICS_STORE=custom forces adapter usage.
  • METRICS_STORE=file forces JSON file storage.
  • Adapters live in scripts/metricsStoreAdapters/ (SQLite + Postgres).
  • METRICS_STORE_ADAPTER can point to any JS module that exports a createMetricsStore factory or metricsStore object.

Project detail cache

Project detail snapshots can be stored in Postgres to reduce GitHub API traffic in production or high-load scenarios. When enabled, the API checks Postgres before hitting GitHub and writes fresh snapshots back into the cache table.

Variable Purpose Notes
PROJECT_DETAIL_CACHE Cache mode auto (default), postgres, or disabled.
PROJECT_DETAIL_CACHE_PG_URL Postgres connection string Falls back to DATABASE_URL and other Postgres env vars.
PROJECT_DETAIL_CACHE_SCHEMA Schema name Defaults to public.
PROJECT_DETAIL_CACHE_TABLE Table name Defaults to project_detail_cache.
PROJECT_DETAIL_CACHE_TTL_MS Cache TTL Override default 10 minutes for project detail snapshots.
PROJECT_DETAIL_CACHE_STALE_MS Stale window Override default 60 minutes for stale serving.
PROJECT_DETAIL_REFRESH_SECRET Webhook auth Bearer token required by /api/project-details-refresh.

Metrics configuration

Use these variables to control where metrics are stored and how refreshes are throttled.

Variable Purpose Notes
METRICS_STORE Storage mode file forces JSON, custom forces adapter.
METRICS_STORE_ADAPTER Adapter module path JS module exporting createMetricsStore or metricsStore.
METRICS_PG_URL Postgres connection string Any Postgres URL also works (e.g. POSTGRES_URL).
DATABASE_URL Neon or Postgres URL Supports pooled or unpooled variants.
METRICS_SQLITE_PATH SQLite file path Defaults to data/githubMetrics.sqlite.
METRICS_UPDATE_SECRET Protect update endpoint Bearer token for /api/github-metrics-update.
METRICS_UPDATE_MIN_INTERVAL_MS Refresh throttle Minimum delay between full metrics updates.

Token requirements for GitHub + Vercel telemetry are covered in Deployment.

Manual refresh rules

  • ?refresh=1 triggers manual refresh for telemetry endpoints.
  • /api/github-metrics-update enforces minimum intervals (configured via METRICS_UPDATE_MIN_INTERVAL_MS).
  • Repo refreshes throttle at 1 minute locally and 5 minutes in production.

Security notes

  • Tokens (GitHub, Vercel) are only read in API routes and never shipped to the client.
  • Public responses contain no secrets; only URLs and aggregate metrics.
Ops Deployment

Environment variables and hosting setup.

 CI/CD Automation

Workflows that keep telemetry updated.

 Live Pipeline metrics

See the telemetry surfaces in action.

 Quality Testing

How telemetry is validated in tests.
← Back to home