fork-sync-all

Sync and mirror infrastructure for the three-org chain:

Interested-Deving-1896  ──►  OpenOS-Project-OSP  ──►  OpenOS-Project-Ecosystem-OOC
        ▲                                                         │
        └─────────── upstream-commits / upstream-prs ────────────┘
                                    │
                                    ▼
                         GitLab openos-project
                    (11 subgroups, ~150 repos mirrored)

Documentation

• Full documentation — architecture, quota management, workflow reference, runbooks
• Workflow Triggers — every workflow, its schedule, and what else triggers it (plain text)

Workflows

Sync & Mirror

Import

Import workflow inputs:

• repo_url — source URL (GitHub, GitLab, Bitbucket, Codeberg, Sourcehut, Gitea, or any git host)
• repo_name — optional rename in Interested-Deving-1896 (defaults to source name)
• mirror_to_osp_ooc — push through the OSP → OOC chain immediately
• ongoing_sync — register in registered-imports.json for re-sync every 6h

Quota and queue management

Security and token management

Maintenance

Secrets

To add a missing secret, run in your terminal (value prompted securely, never logged):

gh secret set <SECRET_NAME> --repo Interested-Deving-1896/fork-sync-all

Registered Imports

registered-imports.json tracks repos imported via import-repo.yml with ongoing_sync enabled. The sync-registered-imports.yml workflow reads this file hourly and re-pulls each source.

Schema:

[
  {
    "source_url":  "https://gitlab.com/some-group/some-repo",
    "target_name": "some-repo",
    "platform":    "gitlab",
    "added":       "2026-05-02T18:00:00Z"
  }
]

To register a repo manually, run import-repo.yml with ongoing_sync: true, or edit the file directly and commit.

Rate limits

All workflows share a single SYNC_TOKEN. Understanding the limits prevents surprise failures and helps diagnose them when they do occur.

GitHub REST API

What a 403/429 means here: GitHub returns HTTP 403 for secondary rate limits and HTTP 429 for primary exhaustion. Both include X-RateLimit-Reset in the response headers. All scripts that call the GitHub API read this header and sleep until the reset window opens before retrying (up to 3 attempts).

Workflows most likely to hit limits: sync-forks.yml (scans all forks), reconcile-org-refs.yml (reads every file in every repo), and resolve-failures.yml (scans all repos across three orgs). These run sequentially within their own concurrency group so they don't compound each other's usage.

If a workflow fails with "API rate limit exceeded": the next scheduled run will succeed once the window resets. resolve-failures.yml will also catch and retry it automatically. No manual intervention is needed unless the token itself has been revoked.

GitHub Models API

Used by resolve-failures.yml and create-readmes.yml / update-readmes.yml for AI-assisted analysis and generation.

HTTP 429 from the Models API includes a Retry-After header. Scripts read this and sleep for the indicated duration before retrying (up to 3 attempts). If the quota is fully exhausted the script logs [models-rate-limit] GitHub Models quota exhausted and skips AI analysis for that run — the workflow still exits 0 so it doesn't generate a false failure notification.

GitLab API

Used by mirror-osp-to-gitlab.yml, sync-from-gitlab.yml, and sync-to-gitlab.yml.

HTTP 429 (and occasionally 403) from GitLab includes a RateLimit-Reset header. Scripts read this and sleep until the window resets before retrying.

git push limits

Mirror scripts that push via HTTPS (mirror-to-osp.yml, mirror-osp-to-ooc.yaml, sync-to-gitlab.yml, sync-registered-imports.yml, etc.) can hit transient push rejections under load — these are not HTTP API limits but git-level errors. All push steps retry up to 3 times with linear backoff (15 s, 30 s, 45 s) before failing.

The mirror-osp-to-ooc.yaml workflow additionally uses a concurrency group (mirror-to-ooc) so concurrent runs queue rather than race, which eliminates the cannot lock ref class of push failures.

Diagnosing a rate-limit failure

1. Open the failed run log and search for [rate-limit] or rate limit exceeded.
2. The log line includes the HTTP status, sleep duration, and attempt number.
3. If all 3 retries were exhausted the next scheduled run will succeed automatically — primary limits reset hourly, secondary limits within ~60 s.
4. If failures persist across multiple scheduled runs, check that SYNC_TOKEN is valid (gh auth status) and has the required scopes (repo, workflow, admin:org).

GitHub Actions runner minutes

Free tier: 2,000 min/month (Linux, resets 1st of each month). All jobs use ubuntu-latest (1× multiplier). At the current schedule density (~7 hourly workflows), this repo exceeds the free tier. A paid plan or self-hosted runner is required.

Symptoms of exhaustion: ubuntu-latest jobs queue indefinitely, 0 in-progress runs, no runners active. Check via Settings → Billing → Actions.

Recovery: Cancel all queued runs (they will never start), then wait for the monthly reset or add a self-hosted runner.

# Bulk cancel all queued runs (requires API quota)
gh api "repos/Interested-Deving-1896/fork-sync-all/actions/runs?per_page=100" \
  --jq '[.workflow_runs[] | select(.status=="queued") | .id] | .[]' | \
  xargs -I{} gh api -X POST \
    "repos/Interested-Deving-1896/fork-sync-all/actions/runs/{}/cancel"

Concurrency groups and stuck runs

All workflows use cancel-in-progress: true. A newer run always supersedes a queued one, preventing permanent queue buildup when runner minutes are exhausted mid-job.

Detecting stuck runs:

gh api "repos/Interested-Deving-1896/fork-sync-all/actions/runs?per_page=100" \
  --jq '[.workflow_runs[] | select(.status=="queued")] | length'

workflow_run trigger cost

workflow_run fires on every completed event regardless of conclusion. All listeners in this repo gate at the job level so they exit immediately (no runner cost) when the upstream conclusion doesn't match:

• Content processors (create-readmes, update-readmes, inject-badges, translate-readmes, lts-readmes, mirror-osp-to-gitlab): gate on conclusion == 'success'
• Watchdogs (mirror-orgs-watchdog): gate on conclusion == 'failure'

See DOCS/OPERATIONS.md for the full operational reference: quota tables, schedule summary, self-hosted runner setup, and quick-reference reset times.

GitLab subgroups

11 subgroups under gitlab.com/openos-project, ~150 repos mirrored:

Subgroup IDs and repo assignments are in config/gitlab-subgroups.yml.

Mirror chain timing

:00  mirror-to-osp.yml          Interested-Deving-1896 → OSP
:05  sync-pieroproietti          pieroproietti forks fast-path
:10  quota-reserve.yml           Cancel low-priority runs if quota < 1000
:15  queue-manager.yml           Deduplicate queued runs
:15  mirror-osp-to-ooc.yaml      OSP → OOC  (per-repo, injected by setup-osp-mirrors)
:23  upstream-prs.yml            OOC/OSP PRs → Interested-Deving-1896
:30  mirror-osp-to-gitlab.yml    OSP → GitLab openos-project
:45  upstream-commits.yml        Direct OSP/OOC commits → PRs in Interested-Deving-1896
:55  sync-registered-imports     External platform imports re-sync

Full chain (validate-config → full-chain-flush) runs on every push to main.

AGENTS.md

Conventions, patterns, and known pitfalls for AI agents working in this repo.

Repository overview

fork-sync-all is the control plane for the Interested-Deving-1896 GitHub org. It mirrors repos into OpenOS-Project-OSP (GitHub) and then to openos-project (GitLab), manages READMEs across ~49 OSP-bound repos, syncs upstream forks, and runs org-wide maintenance workflows.

Key config files:

• config/gitlab-subgroups.yml — single source of truth for GitLab subgroup placement
• registered-imports.json — upstream repos to keep in sync
• scripts/ — all automation scripts
• .github/workflows/ — GitHub Actions workflows

Key directories:

• vendor/ — third-party components hosted/deployed by fork-sync-all (e.g. infra-dashboard). Everything in scripts/ is first-party automation. Do not move scripts into vendor/.

GitHub API quota

Both GH_TOKEN and SYNC_TOKEN belong to the same user (ID 202036334) and share the same 5000 req/hr REST bucket. Treat them as one pool.

• raw.githubusercontent.com fetches do not count against the quota
• GraphQL counts as 1 call regardless of how many repos are queried
• The quota pre-flight in workflows uses MIN_QUOTA (typically 1000–1500) to skip runs when the bucket is too low; quota-monitor.sh retries after reset

When quota is at 0, avoid any gh api, curl .../api.github.com/..., or gh_get calls. Check reset time with:

curl -sf -H "Authorization: token $SYNC_TOKEN" \
  "https://api.github.com/rate_limit" | jq '{remaining, reset: (.resources.core.reset | todate)}'

Script conventions

All logging helpers must write to stderr

Every script defines some combination of info(), warn(), dry(), and log(). All must use >&2:

info() { echo "[script-name] $*" >&2; }
warn() { echo "[warn] $*" >&2; }
dry()  { echo "[dry-run] $*" >&2; }
log()  { echo "[$(date -u '+%H:%M:%S')] $*" >&2; }

Why this matters: Several functions are called inside $(...) subshell captures where their stdout becomes the captured value (e.g. README content, repo lists, API responses). Any logging call without >&2 inside such a function will corrupt the captured data.

This applies to includes/gh-api.sh too — merge_upstream() status messages must go to stderr since callers may capture its output via result=$(merge_upstream ...).

Known functions called inside $(...) captures — never emit to stdout inside these:

• rewrite_readme() in update-readmes.sh
• fill_missing_sections() in update-readmes.sh
• build_readme() in create-readmes.sh
• generate_*() functions in update-readmes.sh
• merge_upstream() in scripts/includes/gh-api.sh

YAML parsing

Always use yaml.safe_load — never hand-rolled regex/indent parsers:

import yaml
with open(config_path) as f:
    config = yaml.safe_load(f)
subgroups = config.get("subgroups", {}) or {}

This applies to gitlab-subgroups.yml parsing in all scripts.

includes/ scripts

scripts/includes/budget.sh, scripts/includes/gh-api.sh, and scripts/includes/quota-instrument.sh are sourced by many scripts and workflows. Changes there have broad impact.

• budget.sh — provides budget_init, budget_check, budget_report, osp_priority_repos, and workflow_min_quota. The latter reads per-workflow min_quota from config/workflow-quota-costs.yml.
• gh-api.sh — provides gh_api, gh_get, gh_api_graphql, merge_upstream, get_default_sha. All status messages use >&2. Guard against double-sourcing is in place (_GH_API_LOADED). gh_get URL is a convenience GET wrapper around gh_api with full retry and reset-aware backoff — the canonical implementation that individual scripts should migrate to (see consolidation note below).

gh_get / gh_api consolidation (complete)

All scripts now source includes/gh-api.sh for gh_get. The three tiers that existed during migration have been fully consolidated:

All new scripts should source includes/gh-api.sh and use gh_get directly. Do not define a local gh_get() in any new script.

• quota-instrument.sh — provides qi_begin / qi_end for measuring REST quota consumption per workflow run. Wire into the main job step of any workflow you want to instrument. Writes a structured HTML comment to GITHUB_STEP_SUMMARY that update-quota-costs.yml parses weekly to compute observed p50/p95 values.

REST → GraphQL conversion

Prefer GraphQL over paginated REST for any loop that fetches the same data for multiple repos. GraphQL counts as 1 REST call regardless of how many repos are queried.

Standard pattern for org repo lists:

result=$(curl -sf \
  -H "Authorization: token ${GH_TOKEN}" \
  -H "Content-Type: application/json" \
  "${GH_API}/graphql" \
  -d "{\"query\":\"{ organization(login: \\\"${ORG}\\\") { repositories(first: 100) { nodes { name } pageInfo { hasNextPage endCursor } } } }\"}" \
  2>/dev/null || echo "{}")
echo "$result" | python3 -c "
import json,sys
d=json.load(sys.stdin)
for n in d.get('data',{}).get('organization',{}).get('repositories',{}).get('nodes',[]):
    print(n['name'])
" 2>/dev/null

Prefetch pattern for per-repo metadata (existence, pushedAt, README): Batch all repos into a single GraphQL call using aliases, populate an associative array, then read from the cache in the loop — zero REST calls per repo:

declare -A _REPO_EXISTS=()
# ... build aliases, fire one GraphQL call, populate _REPO_EXISTS ...
# In the loop:
[[ -z "${_REPO_EXISTS[$repo]:-}" ]] && continue  # skip non-existent repos

See sync-registered-imports.sh (prefetch_repo_metadata), mirror-releases.sh (prefetch_upstream_existence), and inject-badges.sh (list_gh_repos + _README_CACHE) for reference implementations.

What cannot be converted to GraphQL:

• check-runs and statuses endpoints — not exposed in GraphQL
• actions/workflows and actions/secrets — not in GraphQL
• Write operations (create repo, push file, cancel run) — REST only

Tree fetches

Use ?recursive=1 on the git trees endpoint to get all file paths in one call, then check membership with grep -qxF before fetching individual files:

tree_json=$(gh_get "${GH_API}/repos/${owner}/${repo}/git/trees/HEAD?recursive=1")
tree_paths=$(echo "$tree_json" | jq -r '.tree[] | select(.type=="blob") | .path')
echo "$tree_paths" | grep -qxF "package.json" && # file exists, fetch it

Never probe file existence with per-file /contents/ calls in a loop.

YAML-safe shell in run: blocks

GitHub Actions run: blocks are YAML block scalars. The YAML parser processes the file before the shell runner sees it, so certain shell constructs break parsing even though they would be valid bash.

Patterns that break YAML — never use these inside run: blocks:

Safe alternatives:

# Multi-line python: collapse to one line
repos=$(python3 -c "import yaml; d=yaml.safe_load(open('config/x.yml')); print(' '.join(d.get('repos',[])))")

# Multi-line variable: use printf into a temp file
printf 'line1\nline2\n' > /tmp/body.txt

# Multi-line commit message: ANSI-C quoting
git commit -m $'subject\n\nbody line 1\nbody line 2'

# Or chained -m flags (each becomes a paragraph)
git commit -m "subject" -m "body paragraph"

# Heredoc end-marker: use a non-YAML-keyword name
cat > file.yml << 'CONFIG_EOF'
...
CONFIG_EOF

The validator catches these: python3 scripts/validate-workflow-guards.py runs a YAML parse check across all 75 workflow files. Run it after editing any workflow. The full-suite parse check is also embedded in validate-config.yml.

Workflow patterns

Queue and quota management

Three workflows protect the system from quota exhaustion cascades and runner starvation:

Priority tiers — single source of truth in config/workflow-priority-tiers.yml:

• Tier 1 CRITICAL — never cancelled (token rotation, queue/reserve management, config validation)
• Tier 2 HIGH — mirror chain, sync operations
• Tier 3 MEDIUM — READMEs, CI checks (default for unknown workflows)
• Tier 4 LOW — translation, dep graph, maintenance (cancelled first)

When adding a new workflow, add it to both:

1. config/workflow-priority-tiers.yml — by workflow name: field (not filename). Both queue-manager.sh and quota-reserve.sh load tiers from this file at runtime — no script edits needed.
2. config/workflow-sync.yml — under github_only (most workflows) or paired (if it has a GitLab CI counterpart). validate-workflow-guards.py warns on any workflow file not listed in either section.

Run python3 scripts/validate-workflow-guards.py after adding any workflow to confirm zero warnings.

dispatch-and-wait.sh exit codes:

• 0 — workflow completed successfully
• 1 — workflow failed or timed out
• 2 — workflow was cancelled (by queue-manager or manually) — retriable, not a real failure

full-chain-flush.yml and critical-deploy.sh both handle exit 2 with a warning rather than aborting.

Concurrency groups

All workflows triggered by schedule or workflow_run must have a concurrency group to prevent queue pile-ups:

concurrency:
  group: workflow-name
  cancel-in-progress: true

workflow_run triggers

Each workflow should have at most one workflow_run upstream trigger. Multiple triggers cause fan-out: N completions × M downstream workflows = queue explosion.

Every name in workflow_run.workflows: must exactly match the name: field of a workflow file that actually exists in .github/workflows/. A phantom name causes the trigger to fire on every push but the job fails immediately — GitHub cannot resolve the upstream workflow. validate-workflow-guards.py (Check 5) catches this automatically.

Quota pre-flight

All hourly/daily/frequent workflows include a quota pre-flight step before doing any API work. The step sets skip=true when remaining < MIN_QUOTA and subsequent steps check if: steps.quota.outputs.skip == 'false'.

Quota cost registry

config/workflow-quota-costs.yml is the single source of truth for per-workflow REST call cost estimates. It drives:

• quota-reserve.sh — cost-aware cancellation (min_quota per workflow)
• budget.sh workflow_min_quota() — pre-flight helper for self-skipping
• DOCS/quota-costs.md — rendered documentation in mdBook

Phase 1 values are code-audit estimates (basis: code-audit). Phase 2 (update-quota-costs.yml, weekly) replaces them with observed p50/p95 values (basis: observed) once ≥5 run samples exist per workflow.

When adding a new workflow that makes significant REST calls, add it to config/workflow-quota-costs.yml with estimated min_quota, cost_low, cost_mid, cost_high, and basis: code-audit. Wire qi_begin/qi_end from scripts/includes/quota-instrument.sh into its main job step so Phase 2 can measure it automatically.

Instrumented workflows (Phase 2 active):

• Sync All Forks
• Inject Built-with-Ona Badges
• Reconcile Org References
• Cleanup Stale Branches
• Check OSP-Bound CI Status
• Sync Registered Imports
• Mirror Interested-Deving-1896 → OSP
• Pre-Mirror CI Gate
• Verify Mirror Integrity
• Post-Flush Verification
• Pipeline Telemetry
• Translate Docs

Path filters + required status checks (gate job pattern)

When a workflow uses path filters to skip jobs on irrelevant changes, required status checks will block PRs indefinitely if the filtered jobs never run. Fix this with a gate job that always runs and reflects the filtered outcomes:

jobs:
  changes:
    name: Detect changes
    runs-on: ubuntu-latest
    outputs:
      shell: ${{ steps.filter.outputs.shell }}
    steps:
      - uses: actions/checkout@v4
      - uses: dorny/paths-filter@v3
        id: filter
        with:
          filters: |
            shell:
              - '**/*.sh'

  lint:
    name: ShellCheck
    needs: changes
    if: needs.changes.outputs.shell == 'true'
    runs-on: ubuntu-latest
    steps: [...]

  # Set THIS as the required status check — not the individual jobs above.
  ci-required:
    name: CI Required
    runs-on: ubuntu-latest
    needs: [lint]
    if: always()
    steps:
      - name: Check results
        run: |
          if echo "${{ join(needs.*.result, ' ') }}" | grep -qw "failure"; then
            exit 1
          fi

Branch protection must require CI Required (the gate job name), not the individual filtered job names. If the individual names are listed as required checks, PRs that skip those jobs will be permanently blocked.

Applied in: btrfs-dwarfs-framework/.github/workflows/ci.yaml

Template sync profiles

config/template-consumers.yml controls which repos receive automatic file updates from sync-template.yml. Each consumer has a profile that determines what gets injected.

Profile assignments

Critical rule

Never assign mirror profile to consumer repos. The mirror profile injects the full fork-sync-all mirror/sync suite (60+ workflow files, 100+ scripts) into repos that are targets of the mirror chain, not operators of it. This causes template pollution — files that have no purpose in the target repo and clutter its .github/workflows/ and scripts/ directories.

Template pollution cleanup

If a repo has been polluted by the mirror profile:

1. Check which files don't belong:

for f in .github/workflows/*.yml; do
  grep -q "SYNC_TOKEN\|openos-project\|mirror-to-osp\|registered-imports" "$f" \
    && echo "POLLUTION: $(basename $f)" \
    || echo "native:    $(basename $f)"
done

2. Remove them with git rm --cached and commit:

git rm --cached .github/workflows/add-mirror-repo.yml  # etc.
git commit -m "chore: remove fork-sync-all template pollution"

3. Delete the untracked files from disk:

git status --short | grep "^??" | awk '{print $2}' | xargs rm -f

4. Trigger cleanup-pollution.yml (workflow_dispatch) to clean remaining consumer repos automatically.

Repos cleaned of mirror pollution (2026-06-06)

• KPort — 74 files removed
• btrfs-dwarfs-framework — 133 files removed
• All other infra-core consumers — cleaned via cleanup-pollution.yml

Queue pile-up pattern

Workflows that trigger on .github/workflows/** (e.g. validate-config, update-workflow-triggers-doc) must have concurrency: cancel-in-progress: true to prevent stacking. Without it, rapid pushes create a queue of identical runs that consume quota on every reset, causing a deadlock where the queue can't drain because quota is always 0.

concurrency:
  group: workflow-name-${{ github.ref }}
  cancel-in-progress: true

OSP-bound repo list

The canonical list of ~49 repos that are mirrored to GitLab lives in config/gitlab-subgroups.yml. Parse it with yaml.safe_load — do not hardcode repo names anywhere else.

To get the list in bash:

python3 -c "
import yaml
data = yaml.safe_load(open('config/gitlab-subgroups.yml'))
for sg in data.get('subgroups', {}).values():
    for repo in (sg.get('repos') or []):
        print(repo)
"

GitLab subgroup IDs

Parent group: openos-project on GitLab (gitlab.com/openos-project)

All IDs are authoritative — sourced from config/gitlab-subgroups.yml. Do not hardcode them elsewhere.

README management

AI marker format

<!-- AI:start:section-name -->
content
<!-- AI:end:section-name -->

Eight AI-owned sections: what-it-does, architecture, ci, mirror-chain, contributors, origins, resources, license.

Human-owned sections (Install, Usage, Configuration, License) never get AI markers — they get placeholder HTML comments on first creation.

Three modes in update-readmes.sh

• rewrite — no AI markers present → build full template from scratch
• fill — some markers present but missing sections → inject missing ones
• update — all markers present → regenerate AI section content

check-readme-render.sh

Run this against any README before committing. It catches: leaked log lines, unclosed fences, unclosed AI markers, empty sections, missing H1, broken tables, bare [text] links, raw angle brackets.

bash scripts/check-readme-render.sh path/to/README.md

GitLab CI variables

These must be set as masked CI/CD variables in the openos-project/fork-sync-all GitLab project settings (not GitHub secrets):

Headroom proxy

A context compression proxy runs on port 8787 (started automatically via .ona/automations.yaml). To use it with Claude:

ANTHROPIC_BASE_URL=http://localhost:8787 claude
# or
headroom wrap claude

Check savings: headroom stats

Token rotation

Tracked tokens

The "PAT name" column is the display name shown at github.com/settings/tokens (classic).

| ACTIVITYSMITH_API_KEY | n/a (external service) | ActivitySmith API | ActivitySmith | unknown | full-chain-flush.yml (live activity tracking) — optional, skipped if unset | manual | | ACTIVITYSMITH_CHANNELS | n/a (external service) | ActivitySmith channel IDs | ActivitySmith | n/a | full-chain-flush.yml — optional, skipped if unset | manual | | ANTHROPIC_API_KEY | n/a (external service) | Anthropic API | Anthropic | n/a | validate-config.yml (AgentShield scan) — optional, skipped if unset | manual |

How to rotate a repo secret (SYNC_TOKEN, GH_SYNC_TOKEN, etc.)

1. Generate a new PAT at https://github.com/settings/tokens
2. Go to rotate-token.yml → Run workflow
3. Select the secret name from the dropdown
4. Paste the new token value into the token_value field
5. Leave validate checked — it confirms the token works before finishing
6. After the run completes, update the expiry date in this table

How to rotate an OSP org secret (ORG_MIRROR_OSP_TO_OOC, MIRROR_TOKEN)

OSP org secrets live in OpenOS-Project-OSP and require a token with admin:org on that org. SYNC_TOKEN only covers Interested-Deving-1896.

The rotate-token.yml workflow resolves the OSP token automatically in this priority order:

Option 1 — GitHub App (preferred, permanent)

A GitHub App installation token never expires and has fine-grained permissions.

One-time setup:

1. Create a GitHub App at https://github.com/settings/apps/new
   • Name: fork-sync-all-osp-rotator (or similar)
   • Permissions: Organization secrets → Read and write
   • Uncheck everything else
2. Install the App on OpenOS-Project-OSP org
3. Note the App ID (shown on the app settings page)
4. Generate a private key (PEM format) from the app settings page
5. Add two repo secrets to Interested-Deving-1896/fork-sync-all:
   • OSP_APP_ID — the numeric App ID
   • OSP_APP_PRIVATE_KEY — the full PEM contents (including header/footer)
6. Run rotate-token.yml — it will use the App automatically

Option 2 — Dedicated PAT (bridge until App is set up)

1. Generate a new PAT at https://github.com/settings/tokens with:
   • admin:org scope
   • Authorized for OpenOS-Project-OSP org (SSO authorize if required)
2. Add it as repo secret OSP_ADMIN_TOKEN in Interested-Deving-1896/fork-sync-all
3. Run rotate-token.yml — it will use OSP_ADMIN_TOKEN automatically

Option 3 — Manual fallback

If neither OSP_APP_* nor OSP_ADMIN_TOKEN is set, the workflow prints the exact error and the two options above. You can also update manually:

1. Generate a new PAT with admin:org on OpenOS-Project-OSP
2. Go to OSP org secrets and update the secret value directly
3. Update the expiry date in scripts/token-monitor.sh (OSP_ORG_SECRETS array) and in the table above

⚠️ Upcoming rotations (as of 2026-06-08):

• ADD_MIRROR_REPO_SYNC — expires 2026-08-13 (66 days). token-health.yml will open an issue around 2026-06-29.
• MIRROR_TOKEN / ORG_MIRROR_OSP_TO_OOC — expire 2026-09-01 (85 days). Alert ~2026-07-17.
• SYNC_TOKEN — expires 2026-09-02 (86 days). Alert ~2026-07-18.
• GH_SYNC_TOKEN / OSP_ADMIN_TOKEN — expire 2026-09-03 (87 days). Alert ~2026-07-19.

Automated monitoring

token-health.yml runs weekly (Monday 09:00 UTC) and warns at 45 days before expiry. When a token needs attention it opens a GitHub issue labelled token-monitor. Run it manually at any time to get a current status report.

vendor/ conventions

Agnostic-by-default rule

Everything imported into vendor/ must be deployment-agnostic. No distro names, org-specific URLs, org/repo slugs, or arch/repo paths may appear as hardcoded fallback values in shell ${VAR:-...}, YAML || '...', or TypeScript ?? '...' expressions. All deployment-identity values belong in CI variables or repo vars set per deployment.

Enforcement

scripts/check-vendor-agnostic.sh scans a vendor directory and exits 1 on violations:

bash scripts/check-vendor-agnostic.sh vendor/infra-dashboard   # specific component
bash scripts/check-vendor-agnostic.sh vendor                   # all of vendor/

enforce-agnostic-vendor.yml runs this automatically on every push/PR touching vendor/.

To suppress a specific line that is intentionally non-agnostic:

SOME_VAR="${SOME_VAR:-specific-value}"  # check-vendor-agnostic: ignore

What the checker flags vs. allows

Flagged (deployment-identity):

• Public URLs as fallbacks: ${VITE_ENDPOINT_URL:-https://api.myorg.com}
• Org/repo slugs: ${MIRRORLIST_REPO:-MyOrg/my-repo}
• Arch/repo paths: ${MIRROR_REPO_PATHS:-x86_64/core,x86_64/extra}
• Bare distro names: ${DISTRO:-cachyos}, ${DISTRO:-ubuntu}

Allowed (generic defaults):

• Localhost dev URLs: ${API_URL:-http://localhost:5862}
• Generic relative paths: ${MIRRORLIST_PATH:-mirrorlist/mirrorlist}
• Single-word tokens: ${LOG_LEVEL:-info}, ${ENV:-production}
• UI strings: ${APP_NAME:-Infra Dashboard}

Workflow integrations

import-repo → immediate sync

When ongoing_sync=true, import-repo.sh writes to registered-imports.json and then immediately dispatches sync-registered-imports.yml with repo_filter=<name> and force_sync=true. This avoids the up-to-6h wait for the scheduled run to pick up the new entry.

If the dispatch fails (quota, permissions), it falls back gracefully — the entry is still registered and will sync on the next scheduled run.

merge-to-monorepo → OSP mirror chain

merge-to-monorepo.yml has a mirror_monorepo boolean input (default: false). When set, it dispatches add-mirror-repo.yml for the newly created monorepo after a successful merge, entering it into the standard OSP mirror chain automatically.

Known pitfalls

• fill_missing_sections case statement — must handle all 8 AI sections. If you add a new section to ALL_AI_SECTIONS, add it to the case in fill_missing_sections, rewrite_readme, and the update mode loop.

• sync-registered-imports.sh does not create repos — ensure_gh_repo() handles creation now, but the target repo must be reachable via the GitHub API. New entries in registered-imports.json will auto-create the repo on first run.

• GitLab mirror chain — I-D-1896 → OpenOS-Project-OSP (GitHub) → openos-project (GitLab). Adding a repo to gitlab-subgroups.yml is required for GitLab mirroring. Adding to registered-imports.json is required for upstream sync. Both are independent — a repo can be in one without the other.

• _inter_repo_sleep in update-readmes.sh — quota-aware pacing. No delay when quota > 2000; scales to 30s when < 500. The cached _quota_remaining variable is decremented by 10 per repo to trigger re-checks before actually hitting the threshold.

Architecture

The three-org chain

fork-sync-all is the control plane for a three-organisation mirror chain on GitHub, with a fourth leg into GitLab:

Interested-Deving-1896  ──►  OpenOS-Project-OSP  ──►  OpenOS-Project-Ecosystem-OOC
        ▲                                                         │
        └─────────── upstream-commits / upstream-prs ────────────┘
                                    │
                                    ▼
                         GitLab openos-project
                    (12 subgroups, ~157 repos mirrored)

All automation runs in Interested-Deving-1896/fork-sync-all. The other orgs are passive recipients — they do not run their own automation except for the GitLab CI mirror job that pushes back to GitLab.

Data flow

Inbound (upstream → I-D-1896)

Three paths bring upstream changes into Interested-Deving-1896:

1. sync-forks.yml (daily) — syncs all GitHub forks with their upstream parents
2. sync-registered-imports.yml (every 6h) — re-syncs repos registered in registered-imports.json, including non-GitHub sources (GitLab, Bitbucket, Codeberg, etc.)
3. upstream-commits.yml / upstream-prs.yml (every 6h) — detects direct commits and open PRs in OSP/OOC that haven't been reflected upstream, and opens PRs in I-D-1896

Outbound (I-D-1896 → OSP → OOC → GitLab)

The mirror chain runs in sequence, each leg triggered by the previous:

mirror-to-osp.yml  ──►  mirror-osp-to-ooc.yaml  ──►  GitLab CI (sync-to-gitlab.yml)
   (every 6h)              (every 2h at :15)           (on OSP push)

full-chain-flush.yml orchestrates a complete end-to-end run of all three legs plus README updates, sync, and validation in a single coordinated sequence.

GitLab subgroup placement

config/gitlab-subgroups.yml is the single source of truth for which repos go into which GitLab subgroup. The 12 subgroups map to topic areas:

Repos not listed in any subgroup fall into the ops default subgroup.

Quota management

Both GH_TOKEN and SYNC_TOKEN belong to the same GitHub user and share a single 5000 req/hr REST bucket. The system has three layers of protection:

quota-monitor.yml  ──►  quota-reserve.yml  ──►  queue-manager.yml
  (every 10 min)          (every 10 min)          (every 15 min)

Workflow priority tiers are defined in config/workflow-priority-tiers.yml. Tier 1 (CRITICAL) runs are never cancelled. See Operations for the full quota reference.

Config files

vendor/

vendor/ contains third-party components that fork-sync-all hosts or deploys. It is distinct from scripts/ (first-party automation) and config/ (config data).

Current components:

All vendored components must be deployment-agnostic — no distro names, org-specific URLs, or hardcoded deployment values. See Contributing for the enforcement workflow.

Token architecture

Two GitHub PATs are in active use, both owned by the same user (ID 202036334) and sharing the same 5000 req/hr quota:

GitLab operations use GITLAB_SYNC_TOKEN (api, read/write_repository scope).

Token expiry is monitored weekly by token-health.yml. See Token Rotation for rotation procedures.

Workflow Reference

All workflows in .github/workflows/, grouped by priority tier. For trigger details and schedules see Workflow Triggers.

Auto-generated on 2026-06-09 from config/workflow-quota-costs.yml and config/workflow-priority-tiers.yml.

Quota cost columns: Low = fast/cached run · Mid = typical (p50) · High = large/uncached (p95)

Tier 1 — Critical

Tier 2 — High

Tier 3 — Medium

Tier 4 — Low

✦ Mid value is an observed p50 measurement. All other values are code-audit estimates.

Workflow Triggers

All workflows in .github/workflows/. Grouped by function, with every trigger listed.

Plain-text version: docs/workflow-triggers.txt
Auto-generated on 2026-06-09 from .github/workflows/ and config/workflow-quota-costs.yml

Mirror Chain

OSP-Bound Repo Management

Fork & Import Sync

GitLab Sync

README Management

CI & Failure Resolution

Maintenance & Housekeeping

Full Pipeline

Utility / On-Demand

Schedule Summary (UTC)

OTA Update System

The OTA (over-the-air) system delivers workflow and script updates from fork-sync-all to forks that have opted in. It is the mechanism by which downstream forks stay current without manual merges.

Concepts

Upstream — Interested-Deving-1896/fork-sync-all. The source of truth for all OTA payloads.

Fork — any GitHub repo that has forked fork-sync-all and opted in to OTA.

Payload — a diff of files that changed between the fork's pinned_sha and the latest upstream release tag. Only files the fork hasn't locally modified are included.

Registry — config/ota-registry.yml. The list of all opted-in repos.

Blocklist — config/ota-blocklist.yml. Orgs and namespaces excluded from OTA by default (the three mirror-chain orgs and the GitLab namespace).

Lifecycle

Fork owner runs ota-opt-in
        │
        ▼
.ota/config.yml created in fork
Registration PR opened against fork-sync-all
        │
        ▼
PR merged → repo added to config/ota-registry.yml
        │
        ▼
ota-discover.yml (daily) also finds new opt-ins automatically
        │
        ▼
Semver tag pushed to fork-sync-all (v*.*.*)
        │
        ▼
ota-release.yml assembles payload per opted-in repo
Opens PR in each fork with the diff
        │
        ▼
Fork owner merges PR
ota-self-update.yml (runs in fork on schedule) updates pinned_sha

Workflows

ota-opt-in.yml — fork owner runs this once

Propagated to forks via the standalone template profile. The fork owner triggers it via workflow_dispatch. It:

1. Creates .ota/config.yml in the fork with sensible defaults
2. Opens a registration PR against fork-sync-all/config/ota-registry.yml

Inputs:

ota-discover.yml — runs daily in fork-sync-all

Scans GitHub for forks of fork-sync-all that contain .ota/config.yml with enabled: true. For any not already in config/ota-registry.yml, opens a PR to add them.

This is the passive discovery path — fork owners don't need to run ota-opt-in if they create .ota/config.yml manually.

Inputs:

ota-release.yml — triggered on semver tag push

Triggered when a tag matching v*.*.* is pushed to fork-sync-all. It:

1. Iterates all repos in config/ota-registry.yml (skipping disabled: true)
2. For each repo, calls ota-payload-build.sh to assemble the diff
3. Opens a PR in the fork with the payload
4. Updates CHANGELOG.md in fork-sync-all

Repos in the blocklist orgs are skipped unless mirror_chain_opt_in: true is set in their .ota/config.yml.

ota-self-update.yml — runs in the fork on a schedule

Propagated to forks via the standalone template profile. Runs on a schedule in the fork. It:

1. Checks the latest OTA release tag from fork-sync-all
2. Compares against the fork's pinned_sha in .ota/config.yml
3. If behind, applies the payload and updates pinned_sha and pinned_at

This is the self-healing path — if a fork owner doesn't merge the OTA PR, ota-self-update will eventually apply the update automatically.

Payload assembly

scripts/ota-payload-build.sh assembles the payload for a single fork:

1. Detects the fork's upstream parent via GitHub API (or uses upstream_override)
2. Diffs the fork's current state at pinned_sha against the latest upstream tag
3. Filters out:
   • Files listed in the fork's exclude_paths
   • Files the fork has locally modified (detected by comparing against upstream)
   • Files owned by template profiles (from config/template-manifest.yml) unless explicitly claimed via workflow_overrides.claim
4. Applies workflow_overrides.disclaim to remove any files the fork wants to manage independently

The result is a minimal set of files that are safe to overwrite in the fork.

.ota/config.yml reference

Created in the fork by ota-opt-in.yml. All fields except enabled and repo are optional.

enabled: true                  # master switch
repo: "owner/repo-name"        # must match actual GitHub repo
host: "github"                 # "github" only currently
upstream_override: ""          # override upstream detection (fork-of-fork)
pinned_sha: ""                 # managed by ota-self-update — do not edit
pinned_at: ""                  # managed by ota-self-update — do not edit
ota_version: ""                # managed by ota-self-update — do not edit
mirror_chain_opt_in: false     # set true only for mirror-chain repos
workflow_overrides:
  claim: []                    # workflows OTA should manage even if in manifest
  disclaim: []                 # workflows OTA should NOT touch
exclude_paths: []              # glob patterns OTA never writes
include_paths: []              # re-include after exclude_paths

Full field documentation: .ota/schema.yml in this repo.

Blocklist

config/ota-blocklist.yml defines two guards applied before any delivery:

Guard 1 — org/namespace blocklist: The three mirror-chain GitHub orgs (Interested-Deving-1896, OpenOS-Project-OSP, OpenOS-Project-Ecosystem-OOC) and the GitLab namespace (openos-project) are excluded by default. A repo in these orgs can still receive OTA by setting mirror_chain_opt_in: true.

Guard 2 — profile filter: Only repos using the standalone template profile are eligible for OTA. Repos on core, extended, or other profiles are managed by sync-template.yml instead.

Adding a fork to the registry manually

If ota-opt-in is unavailable or the fork owner prefers manual registration:

1. Create .ota/config.yml in the fork (copy from .ota/schema.yml, set enabled: true and repo)
2. Add an entry to config/ota-registry.yml:

opted_in:
  - repo: owner/fork-name
    host: github
    registered_at: "2026-06-07"
    pinned_sha: ""
    discovery: false
    mirror_chain_opt_in: false
    disabled: false

3. Open a PR against fork-sync-all — validate-config.yml will check the entry.

Disabling OTA for a repo

Set disabled: true in the registry entry. The repo stays registered but receives no further deliveries until re-enabled. Alternatively, set enabled: false in the fork's .ota/config.yml — ota-discover will stop treating it as opted-in.

To remove permanently: delete the entry from config/ota-registry.yml.

Registered Imports

All 70 upstream repositories tracked in registered-imports.json. These are synced to Interested-Deving-1896 by sync-registered-imports.yml every 6 hours.

Auto-generated on 2026-06-09 from registered-imports.json.

GitHub (70)

GitLab Subgroup Map

All 158 OSP-bound repositories mapped to their GitLab subgroup under openos-project. This is the single source of truth used by mirror-osp-to-gitlab.sh.

Auto-generated on 2026-06-09 from config/gitlab-subgroups.yml.

git-management_deving

penguins-eggs_deving

immutable-filesystem_deving

linux-kernel_filesystem_deving

incus_deving

taubyte_deving

neon-deving

ops

yaml-tooling_deving

cachyos_deving

ai-agents_deving

rust-systems_deving

Pre-Flush Checklist

Steps to verify before triggering pre-flush-prep.yml or full-chain-flush.yml. Most of these are zero-API-call operations — safe to run while quota is exhausted.

1. Check quota

curl -sf -H "Authorization: token $SYNC_TOKEN" \
  "https://api.github.com/rate_limit" | \
  python3 -c "
import sys, json
from datetime import datetime, timezone
d = json.load(sys.stdin)
core = d['resources']['core']
reset = datetime.fromtimestamp(core['reset'], tz=timezone.utc)
now = datetime.now(tz=timezone.utc)
eta = max(0, int((reset - now).total_seconds()))
print(f'Remaining : {core[\"remaining\"]}/{core[\"limit\"]}')
print(f'Reset at  : {reset.strftime(\"%H:%M:%S UTC\")}')
print(f'ETA       : {eta//60}m {eta%60}s')
"

pre-flush-prep requires at least 500 remaining to proceed past its quota pre-flight. Below that it skips entirely. The flush itself needs ~1000–1500 to complete a full run without hitting the reserve floor mid-way.

2. Run config validators

All must pass with zero errors. Warnings on validate-workflow-guards are acceptable if they are pre-existing (check git log to confirm).

python3 scripts/validate-gitlab-subgroups.py config/gitlab-subgroups.yml
python3 scripts/validate-registered-imports.py registered-imports.json
python3 scripts/validate-cost-profiles.py config/workflow-cost-profiles.yml
python3 scripts/validate-priority-tiers.py config/workflow-priority-tiers.yml
python3 scripts/validate-template-config.py
python3 scripts/validate-workflow-guards.py

Expected output pattern:

config/gitlab-subgroups.yml: 12 subgroups, N repos — ✅ Valid
validate-registered-imports: N entry/entries valid
validate-cost-profiles: N profile(s) valid
validate-priority-tiers: N entries valid (tier1=N, tier2=N, tier3=N, tier4=N)
validate-template-config: N profile(s) valid, N consumer(s) valid
validate-workflow-guards: all checks passed (N workflows, ...)

3. Run the test suites

Python validators:

python3 -m pytest tests/ -v --tb=short

Bash check-readme-render.sh self-test (checks 13–22, mobile/cross-engine):

bash scripts/tests/test-check-readme-render-mobile.sh

All 213 pytest tests and all 24 shell tests must pass. A failure here means a script or config change broke something the validators don't catch.

4. ShellCheck modified scripts

Run against any scripts changed since the last flush:

git diff --name-only HEAD~5 -- 'scripts/*.sh' | xargs shellcheck --severity=warning

Or check the three most commonly modified ones directly:

shellcheck --severity=warning \
  scripts/import-repo.sh \
  scripts/merge-to-monorepo.sh \
  scripts/check-vendor-agnostic.sh

SC1091 (info severity, not following sourced files) is expected and acceptable across all scripts that source includes/budget.sh or includes/gh-api.sh.

5. Check for open PRs

Open PRs on main that are green and mergeable should be merged before the flush so the flush runs against the latest state. pre-flush-prep Step 2 auto-merges eligible PRs, but it's cleaner to do it manually if you're already reviewing.

Dependency-update PRs (e.g. chore(deps): update workflow dependencies) are safe to merge without review — they only bump actions/* versions.

6. Check vendor/ agnostic state

bash scripts/check-vendor-agnostic.sh vendor

Must exit 0. Any violations mean a vendored component has deployment-identity values hardcoded as fallback defaults — fix before flushing so the component is deployable by anyone.

7. Verify working tree is clean

git status --short
git log --oneline -5

Uncommitted changes won't be picked up by the flush. Commit or stash everything before triggering.

8. Trigger

Once all the above are green:

1. Go to pre-flush-prep.yml
2. Click Run workflow
3. Leave all inputs at defaults for a standard flush
4. Optional inputs:
   • skip_merge_prs=true — skip Step 2 if you've already merged PRs manually
   • skip_cleanup=true — skip branch/run cleanup if quota is tight
   • quota_wait_min — lower from 60 if quota is already healthy

pre-flush-prep dispatches full-chain-flush automatically at the end of Step 4. You do not need to trigger full-chain-flush directly.

Quick reference — what each step of pre-flush-prep does

Runbooks

Operational procedures for common and emergency situations.

Quota exhaustion

Symptoms: workflows fail with 403, gh api calls return empty, validate-config skips with "quota too low".

Check current state:

curl -sf -H "Authorization: token $SYNC_TOKEN" \
  "https://api.github.com/rate_limit" | \
  python3 -c "
import sys, json
from datetime import datetime, timezone
d = json.load(sys.stdin)
core = d['resources']['core']
reset = datetime.fromtimestamp(core['reset'], tz=timezone.utc)
now = datetime.now(tz=timezone.utc)
eta = max(0, int((reset - now).total_seconds()))
print(f'Remaining : {core[\"remaining\"]}/{core[\"limit\"]}')
print(f'Reset at  : {reset.strftime(\"%H:%M:%S UTC\")}')
print(f'ETA       : {eta//60}m {eta%60}s')
"

Recovery:

1. Wait for the reset (up to 1 hour). The reset time is shown above.
2. While waiting, use the time productively — all local operations (config validation, ShellCheck, pytest, file edits) work without quota.
3. After reset, trigger pre-flush-prep.yml to clear the queue and restart the mirror chain cleanly.

Prevention: quota-reserve.yml cancels low-priority runs at < 1000 remaining. If exhaustion is recurring, check config/workflow-cost-profiles.yml for unexpectedly expensive workflows and consider raising MIN_QUOTA thresholds.

Queue pile-up

Symptoms: many workflows stuck in "queued" state, runners appear busy but nothing is completing.

Check:

# Via GitHub CLI (requires quota)
gh run list --repo Interested-Deving-1896/fork-sync-all --status queued --limit 50

Recovery:

1. Trigger queue-manager.yml manually — it deduplicates and evicts runs queued > 25 minutes.
2. If the queue is severely backed up, trigger pre-flush-prep.yml with skip_merge_prs=true and skip_cleanup=true — Step 1 aggressively clears stale runs before dispatching the flush.
3. As a last resort, trigger critical-deploy.yml — it performs an aggressive queue clear and dispatches with priority.

Token expiry

Symptoms: token-health.yml opens an issue labelled token-monitor, or workflows fail with 401.

Check expiry:

bash scripts/token-monitor.sh

Rotate a token:

1. Generate a new PAT at https://github.com/settings/tokens
2. Go to rotate-token.yml → Run workflow
3. Select the secret name from the dropdown
4. Paste the new token value
5. Leave validate checked
6. Update the expiry date in AGENTS.md token rotation table

For OSP org secrets (MIRROR_TOKEN, ORG_MIRROR_OSP_TO_OOC), see the Token Rotation section in AGENTS.md — these require a separate PAT with admin:org on OpenOS-Project-OSP.

Mirror chain broken

Symptoms: repos in OSP or OOC are behind I-D-1896 by more than one cycle, or GitLab mirrors show stale commits.

Diagnose:

# Check GitLab sync status (requires quota)
gh workflow run check-gitlab-sync.yml --repo Interested-Deving-1896/fork-sync-all

Recovery by leg:

For a full chain reset, trigger full-chain-flush.yml directly (or via pre-flush-prep.yml for a clean pre-flight first).

Config validation failure

Symptoms: validate-config.yml fails on push, blocking the flush.

Run locally to see the error:

python3 scripts/validate-gitlab-subgroups.py config/gitlab-subgroups.yml
python3 scripts/validate-registered-imports.py registered-imports.json
python3 scripts/validate-cost-profiles.py config/workflow-cost-profiles.yml
python3 scripts/validate-priority-tiers.py config/workflow-priority-tiers.yml
python3 scripts/validate-template-config.py
python3 scripts/validate-workflow-guards.py

Common causes:

• Duplicate repo name in gitlab-subgroups.yml
• Duplicate source_url or target_name in registered-imports.json
• Workflow added to .github/workflows/ but not registered in workflow-priority-tiers.yml or workflow-sync.yml
• Duplicate name in workflow-priority-tiers.yml

Vendor component agnostic check failure

Symptoms: enforce-agnostic-vendor.yml fails on a PR touching vendor/.

Run locally:

bash scripts/check-vendor-agnostic.sh vendor

The output shows the exact file, line, and category of violation. Fix by:

• Removing the hardcoded fallback value (set to empty string)
• Moving the value to a CI variable / repo var
• Adding # check-vendor-agnostic: ignore if the value is genuinely deployment-agnostic (rare — document why)

README render failure

Symptoms: validate-readme-render.yml fails on a PR.

Run locally:

bash scripts/check-readme-render.sh README.md
# Also run the self-test to verify the checker itself is working:
bash scripts/tests/test-check-readme-render-mobile.sh

Common causes: unclosed fences, leaked log lines, bare [text] links without URLs, raw angle brackets, broken tables, missing H1.

OTA delivery failure

Symptoms: ota-release.yml fails for one or more forks, or a fork's ota-self-update.yml fails.

For a single fork:

1. Check the fork's ota-self-update.yml run logs for the specific error
2. Common causes: fork has diverged significantly, pinned_sha is stale, or the fork's .ota/config.yml has an invalid field
3. To reset: update pinned_sha in the fork's .ota/config.yml to the current upstream HEAD SHA, then re-trigger ota-self-update.yml

To skip a fork temporarily: Set disabled: true in its config/ota-registry.yml entry.

To re-deliver to all forks: Push a new semver tag to fork-sync-all — ota-release.yml triggers automatically.

Incident response checklist

For any production incident affecting the mirror chain:

1. Check quota — if exhausted, wait for reset before doing anything else
2. Check queue — trigger queue-manager.yml to clear pile-ups
3. Identify the broken leg — use check-gitlab-sync.yml and manual inspection
4. Fix the specific leg — trigger the relevant mirror workflow directly
5. Validate config — run all validators locally before triggering a flush
6. Run pre-flush-prep — let it clean up and restart the chain
7. Monitor — watch the first few workflow runs after recovery for secondary failures

Quota Cost Registry

GitHub's REST API allows 5,000 requests per hour per user (shared across all tokens belonging to the same user). This page documents how many REST calls each workflow consumes per run, the minimum quota required before a workflow should start, and how the quota management system uses this data.

GraphQL counts as 1 call regardless of how many repos are queried. raw.githubusercontent.com fetches are exempt entirely.

How quota is managed

Three mechanisms work together:

The single source of truth for costs is config/workflow-quota-costs.yml.

Cost table

Costs are estimated from code audit (Phase 1). Phase 2 will replace these with observed p50/p95 values from actual run measurements.

min_quota = minimum REST calls required before this workflow should be allowed to start.

Tier 1 — Critical (never cancelled)

Tier 2 — High

Tier 3 — Medium

Tier 4 — Low (cancelled first)

Daily quota budget

At 5,000 calls/hour reset, the effective daily budget depends on how many resets are consumed cleanly vs. drained by backlog. With the schedule reductions applied (June 2026), the expected daily workflow run count dropped by ~172 runs/day.

REST → GraphQL conversion log

Scripts converted from per-repo REST loops to batched GraphQL calls:

Phase 2: observed cost tracking (planned)

Phase 2 will add lightweight instrumentation to measure actual REST consumption per run:

1. scripts/includes/quota-instrument.sh — records remaining_before and remaining_after as workflow step summary annotations
2. update-quota-costs.yml — weekly workflow that reads the last 30 run summaries via GraphQL, computes p50/p95 per workflow, and commits updated values back to config/workflow-quota-costs.yml with basis: observed

Once Phase 2 is active, the tables above will show observed values alongside the code-audit estimates, and quota-reserve.sh will automatically use the more accurate figures.

Operational Reference: GitHub Actions Limits & Quotas

This document covers the GitHub Actions limits that affect fork-sync-all, what consumes them, how to detect exhaustion, and how to recover.

GitHub API Rate Limit

Quota: 5,000 requests/hour per authenticated user token.

Resets: Top of every hour (rolling window).

What consumes it:

How fork-sync-all burns it:

• Every workflow_run trigger fires a new run, which itself may call the API
• rate-limit-rerun.yml (formerly hourly) scans all recent failed runs
• stuck-run-detector.yml (formerly hourly) lists all queued/in-progress runs
• translate-readmes.yml was triggering after 10 workflows — each trigger consumed dozens of API calls for the org scan
• Bulk-cancelling queued runs during cleanup consumes ~1 req per cancel — if the queue is large and quota is already low, the cancel loop itself can exhaust the remaining quota

Detecting exhaustion:

gh api rate_limit --jq '.resources.core | "remaining: \(.remaining)/\(.limit)  resets: \(.reset | todate)"'

Recovery: Wait until the top of the next hour. GraphQL remains available during REST exhaustion and can be used for read-only queries.

GitHub Actions Runner Minutes

Free tier: 2,000 minutes/month. Resets on your billing cycle date (the day of the month your GitHub account was created — check Settings → Billing → Actions for the exact date).

Paid: Billed per minute beyond the free tier; Linux runners cost 1×, Windows 2×, macOS 10×. All workflows in this repo use ubuntu-latest (Linux, 1×).

What counts against the monthly quota:

• Every job that runs on ubuntu-latest (GitHub-hosted runner)
• Time is measured from job start to job end, rounded up to the nearest minute
• Jobs that are queued but never start do not consume minutes
• Jobs that exit immediately (e.g. if: condition is false at the job level) still consume ~1 minute for runner provisioning

What does NOT count:

• workflow_dispatch triggers that are never clicked
• Runs that are cancelled before a job starts
• Skipped jobs (if: evaluated to false before the runner is assigned)
• Self-hosted runners (zero cost regardless of usage)

How fork-sync-all was burning minutes (before May 2026 fixes):

1. mirror-orgs-watchdog fired after every mirror completion (5 workflows × hourly cadence = ~120 runs/day), each consuming ~1 min even on success
2. update-readmes triggered after 7 workflows including high-frequency syncs
3. inject-badges triggered after mirror workflows that run hourly
4. stuck-run-detector and rate-limit-rerun ran hourly as meta-workflows, each consuming minutes to manage other workflows
5. workflow_run listeners fired on every completed event (success, failure, cancelled) — not just on the outcomes they actually needed

Detecting exhaustion:

Symptoms (in order of appearance):

1. ubuntu-latest jobs queue but never start
2. No in-progress runs despite many queued
3. Runs queued for hours with 0 runners active
4. Billing API returns 404 (needs user OAuth scope — check web UI instead)

Check via GitHub web UI: Settings → Billing → Actions.

Recovery: Wait until the billing cycle reset date. In the meantime:

• Cancel all queued runs (they will never start)
• Do not push commits that trigger new workflow runs
• Use workflow_dispatch manually only for critical operations

Concurrency Groups & Stuck Runs

How they work: A concurrency group allows only one run at a time for a given key. If cancel-in-progress: false, a second run queues behind the first. If the first run never finishes (e.g. runner minutes exhausted mid-job), the queued run is permanently stuck.

The cascade pattern:

1. Runner minutes exhaust mid-job → job hangs in in_progress
2. Next scheduled run queues behind it (cancel-in-progress: false)
3. The in-progress run never finishes → queue grows indefinitely
4. API calls to cancel are themselves rate-limited → nothing can be cleared

Orphaned runs: A run can become permanently orphaned if it was triggered from an older version of a workflow file that contained a job (e.g. Update cost profile) that no longer exists in the current file. The run accepts cancel API calls but GitHub immediately re-queues it because the concurrency group from the old code is still technically active. These runs time out automatically after GitHub's maximum queue wait (~6 hours). New runs from the same workflow are not blocked — they use the current file.

Policy in this repo (May 2026): All workflows use cancel-in-progress: true except those that perform multi-repo writes where mid-run cancellation would leave state partially applied:

Detecting stuck runs:

gh api "repos/Interested-Deving-1896/fork-sync-all/actions/runs?per_page=100" \
  --jq '[.workflow_runs[] | select(.status == "queued")] | length'

Bulk cancel (check quota first — cancel loop consumes ~1 req per run):

gh api rate_limit --jq '.resources.core.remaining'

gh api "repos/Interested-Deving-1896/fork-sync-all/actions/runs?per_page=100" \
  --jq '[.workflow_runs[] | select(.status=="queued") | .id] | .[]' | \
  xargs -I{} gh api -X POST \
    "repos/Interested-Deving-1896/fork-sync-all/actions/runs/{}/cancel"

workflow_run Trigger Cost Model

workflow_run fires on every completed event regardless of conclusion (success, failure, cancelled, skipped). A listener that only needs to act on failures still consumes a runner minute for every successful upstream run unless gated at the job level.

Pattern used in this repo:

# For workflows that act on upstream SUCCESS (content processors):
jobs:
  my-job:
    if: |
      github.event_name != 'workflow_run' ||
      github.event.workflow_run.conclusion == 'success'

# For workflows that act on upstream FAILURE (watchdogs/retriers):
jobs:
  retry:
    if: |
      github.event_name == 'workflow_dispatch' ||
      github.event.workflow_run.conclusion == 'failure'

This exits immediately (no runner cost) when the conclusion doesn't match, while keeping the trigger automatic.

All workflow_run listeners and their gates (May 2026):

Current Workflow Schedule Summary

Workflows that run on a schedule and their cadence after May 2026 fixes:

Hourly workflows are the primary minute consumers. At ~1 min/run, 8 hourly workflows = ~192 min/day = ~5,760 min/month — well over the 2,000 min free tier. A paid plan or self-hosted runner is required for this repo's workload.

Self-Hosted Runner Setup (Recommended)

To eliminate the monthly minute cap entirely, add a self-hosted runner:

1. Go to Settings → Actions → Runners → New self-hosted runner
2. Follow the setup instructions for your host OS
3. Change workflow runs-on from ubuntu-latest to self-hosted (or add a label and use that label)

Self-hosted runners have no minute cost and no concurrent job cap beyond what the host machine can handle.

Quick Reference: Limit Reset Times

Contributing

Conventions for adding workflows, scripts, config entries, and vendor components.

Adding a workflow

1. Create the workflow file in .github/workflows/

2. Register in priority tiers — add an entry to config/workflow-priority-tiers.yml using the workflow's name: field (not the filename):

   - name: "My New Workflow"
     tier: 3   # MEDIUM — adjust based on criticality
   

   Tier guide: 1=CRITICAL (never cancelled), 2=HIGH (mirror chain), 3=MEDIUM (default), 4=LOW (cancelled first under quota pressure)

3. Register in workflow-sync — add to config/workflow-sync.yml:

   • Under github_only if it has no GitLab CI counterpart (most workflows)
   • Under paired if it has a matching GitLab CI job

4. Add a concurrency group if triggered by schedule or workflow_run:

   concurrency:
     group: my-workflow-name
     cancel-in-progress: true
   

5. Add a quota pre-flight if the workflow makes API calls and runs frequently:

   - name: Check quota
     id: quota
     run: |
       remaining=$(curl -sf -H "Authorization: token $GH_TOKEN" \
         "https://api.github.com/rate_limit" | jq '.resources.core.remaining')
       echo "remaining=$remaining" >> "$GITHUB_OUTPUT"
       [[ "$remaining" -lt 500 ]] && echo "skip=true" >> "$GITHUB_OUTPUT" || echo "skip=false" >> "$GITHUB_OUTPUT"
   

6. Validate:

   python3 scripts/validate-workflow-guards.py
   python3 scripts/validate-priority-tiers.py config/workflow-priority-tiers.yml
   

Adding a script

Scripts live in scripts/. All logging must go to stderr — never stdout — because many functions are called inside $(...) captures where stdout becomes the captured value.

info() { echo "[my-script] $*" >&2; }
warn() { echo "[warn] $*" >&2; }

If the script sources includes/budget.sh or includes/gh-api.sh, add the shellcheck directive:

# shellcheck source=includes/budget.sh
source "$(dirname "${BASH_SOURCE[0]}")/includes/budget.sh"

Run ShellCheck before committing:

shellcheck --severity=warning scripts/my-script.sh

Adding a config entry

New repo to GitLab mirror chain

Add to config/gitlab-subgroups.yml under the appropriate subgroup:

  rust-systems_deving:
    repos:
      - my-new-repo

Then validate:

python3 scripts/validate-gitlab-subgroups.py config/gitlab-subgroups.yml

New repo to upstream sync

Add to registered-imports.json:

{
    "source_url": "https://github.com/upstream-org/repo-name",
    "target_name": "repo-name",
    "platform": "github",
    "added": "2026-06-07T00:00:00Z"
}

Then validate:

python3 scripts/validate-registered-imports.py registered-imports.json

New workflow priority tier entry

See "Adding a workflow" above — step 2.

Adding a vendor component

vendor/ is for third-party components that fork-sync-all hosts or deploys. It is not for first-party scripts or config.

Before adding:

• Confirm the component is genuinely third-party (not a script you wrote)
• Confirm it will be deployed or served by fork-sync-all (not just referenced)

When adding:

1. Place under vendor/<component-name>/
2. Strip all distro-specific or org-specific hardcoded defaults — see the agnostic rule below
3. Add a README.md with a "Before the first deploy" section covering all required CI variables
4. Run the agnostic check:

   bash scripts/check-vendor-agnostic.sh vendor/<component-name>
   

Agnostic rule

No deployment-identity values may appear as hardcoded fallback defaults in vendored components. This includes:

• Public URLs: ${VITE_ENDPOINT_URL:-https://api.myorg.com} ❌
• Org/repo slugs: ${MIRRORLIST_REPO:-MyOrg/my-repo} ❌
• Arch/repo paths: ${MIRROR_REPO_PATHS:-x86_64/core} ❌
• Distro names: ${DISTRO:-cachyos} ❌

Allowed:

• Localhost dev URLs: ${API_URL:-http://localhost:5862} ✅
• Generic paths: ${MIRRORLIST_PATH:-mirrorlist/mirrorlist} ✅
• Single-word tokens: ${LOG_LEVEL:-info} ✅
• UI strings: ${APP_NAME:-Infra Dashboard} ✅

To suppress a specific line that is intentionally non-agnostic:

SOME_VAR="${SOME_VAR:-value}"  # check-vendor-agnostic: ignore

enforce-agnostic-vendor.yml runs automatically on every push/PR touching vendor/.

Commit conventions

Follow the existing commit message style:

scope: short description

Longer explanation if needed. Focus on why, not what.

Common scopes: fix, feat, config, docs, vendor, scripts, ci.

Before opening a PR

# Config validators
python3 scripts/validate-gitlab-subgroups.py config/gitlab-subgroups.yml
python3 scripts/validate-registered-imports.py registered-imports.json
python3 scripts/validate-cost-profiles.py config/workflow-cost-profiles.yml
python3 scripts/validate-priority-tiers.py config/workflow-priority-tiers.yml
python3 scripts/validate-template-config.py
python3 scripts/validate-workflow-guards.py

# Test suites
python3 -m pytest tests/ -v --tb=short
bash scripts/tests/test-check-readme-render-mobile.sh

# ShellCheck (for any .sh files changed)
git diff --name-only HEAD -- 'scripts/*.sh' | xargs shellcheck --severity=warning

# Vendor check (if vendor/ was touched)
bash scripts/check-vendor-agnostic.sh vendor

# README render check
bash scripts/check-readme-render.sh README.md

All must pass before the PR is ready for merge.