xonecas.io
8356
Wx

www-xonecas-io / files

www-xonecas-io / tickets / 1003-import-all-public-history-into-a-local-bare.md

2.4 KB
1---
2id: 1003
3title: Import all public history into a local bare repo store
4type: feature
5status: done
6priority: p1
7area: data
8created: 2026-11-07T19:30:00Z
9updated: 2026-11-08T03:58:19Z
10source: agent
11moves: 2026-11-07T19:30:00Z inbox todo agent; 2026-11-08T01:09:36Z todo doing drag; 2026-11-08T01:23:43Z doing review agent; 2026-11-08T03:58:19Z review done drag
12---
13
14Mirror every public repo from github.com/xonecas (44), github.com/sacenox (44) and gitlab.com/xonecas (17) into bare repos, idempotently, with the source list in one file.
15
16Codeberg stays out. Checked: 0 of its commits are missing from GitHub and GitLab.
17
18Facts from discovery: plain https clones work without a token. The GitHub API allows 60 unauthenticated requests per hour, so use it for metadata only. A blobless clone of all 182 remotes was 32 MB, full clones will be larger (not measured). Forks are large, such as pi, solarized, Font-Awesome and BrowserQuest.
19
20Done when: re-running fetches only what changed, and the store holds all 105 GitHub and GitLab remotes.
21
22## Notes
23
24- 2026-11-08 01:23Z (agent): Done: 'make import' mirrors all 105 public remotes into repos/ as bare repos, and a re-run only fetches what changed.
25  
26  WHAT IS THERE:
27  - sources.txt: the one source list, with 105 clone URLs (44 github.com/xonecas, 44 github.com/sacenox, 17 gitlab.com/xonecas). I generated it from the GitHub and GitLab APIs, 3 requests in all. Codeberg is left out.
28  - 'site import' (internal/store): each remote goes to repos/<host>/<owner>/<name>.git, 8 at a time. The first run creates the bare repo and later runs 'git fetch --prune'. Only branches and tags are kept, so GitHub's refs/pull/* (other people's PR commits) are not fetched. HEAD follows the remote's default branch. A clone that fails is logged and makes the command exit non-zero, without stopping the others.
29  - repos/ is gitignored. repo.List now finds *.git at any depth.
30  
31  RESULT: 105 new, 0 failed, in 25 s. The store is 351 MB on disk (full history, forks included). The second run printed '0 new, 0 updated, 105 same, 0 failed' in 16 s.
32  CHECKED: unit tests for new, same and updated against a local remote, the default branch, and parsing sources.txt. Spot checks: ffmpeg-node HEAD is master, mini-coder HEAD is main, and neither has pull refs.
33  NOT CHECKED: what happens when a remote is deleted upstream. That remote would fail and be reported, and its mirror stays.