xonecas.io
8356
Wx

www-xonecas-io / files

www-xonecas-io / tickets / 1042-how-do-we-add-new-remotes.md

20.7 KB
1---
2id: 1042
3title: how do we add new remotes
4type: feature
5status: done
6priority: p2
7created: 2026-11-08T06:26:18Z
8updated: 2026-11-08T20:16:18Z
9source: human
10moves: 2026-11-08T06:26:22Z inbox doing drag; 2026-11-08T06:27:05Z doing review agent; 2026-11-08T06:57:08Z review doing drag; 2026-11-08T07:00:33Z doing review agent; 2026-11-08T07:01:42Z review todo note; 2026-11-08T07:01:44Z todo doing drag; 2026-11-08T07:04:04Z doing review agent; 2026-11-08T15:04:59Z review todo note; 2026-11-08T15:05:05Z todo doing drag; 2026-11-08T15:09:16Z doing review agent; 2026-11-08T16:00:26Z review todo note; 2026-11-08T16:00:48Z todo doing drag; 2026-11-08T16:01:36Z doing blocked agent; 2026-11-08T16:04:02Z blocked todo note; 2026-11-08T16:06:27Z todo doing drag; 2026-11-08T16:12:02Z doing review agent; 2026-11-08T18:14:48Z review todo drag; 2026-11-08T18:15:50Z todo doing drag; 2026-11-08T18:19:52Z doing review agent; 2026-11-08T18:27:23Z review todo note; 2026-11-08T18:29:54Z todo review agent; 2026-11-08T18:34:00Z review todo note; 2026-11-08T18:34:14Z todo doing drag; 2026-11-08T18:35:39Z doing blocked agent; 2026-11-08T18:38:14Z blocked doing drag; 2026-11-08T18:38:46Z doing review agent; 2026-11-08T20:16:18Z review done drag
11area: import
12---
13
14Can we make this more formal, like a cli with the server bin? We could build on it going forward, like allowing me to manage pins, private/public toggle, etc.
15
16## Notes
17
18- 2026-11-08 06:27Z (agent): Answer: add the repo's https clone URL as a line in sources.txt, then run make import, make index and make build (or make serve, which builds first). No code change is needed.
19  - make import: makes a bare mirror of each URL in repos/ (git clone --mirror for a new one, git fetch --prune for the rest), then fetches description and fork flags from the host's API into data/meta.json.
20  - make index: reads repos/ and overrides.txt into data/projects.json. The project's name is the repo name.
21  - make build: renders public/.
22  Limits I found in internal/store/meta.go:
23  - Only github.com and gitlab.com have an API entry. A URL on any other host (Codeberg, sr.ht, a local path) makes import stop with "no API for <host>" after mirroring. That is why 1041 (this repo, which has no remote) needs code.
24  - Each owner's API list is read with per_page=100 and no paging. That's fine today (44 sacenox, 44 xonecas, 17 gitlab), but an owner with more than 100 repos would get no description for the rest.
25  If the same project lives on two hosts, add 'merge <url> <url>' to overrides.txt; hide, rename and tag rules go there too.
26  Nothing was changed: I read the Makefile, cmd/site/main.go and internal/store.
27
28- 2026-11-08 07:00Z (agent): Done: the site binary has a remote command. sources.txt stays the list it edits.
29    make site                             builds bin/site
30    bin/site remote list                  the 105 remotes
31    bin/site remote add <url>             checks the URL, appends it, mirrors it into repos/ and refreshes data/meta.json
32    bin/site remote rm <url>              drops its line (comments kept) and deletes its mirror
33  Then make index build (or make index serve) shows the change, the same as after make import. add takes github.com/gitlab.com https URLs with or without .git and normalises them to .git. It refuses a duplicate, another host or an SSH URL. If the mirror or metadata fails, the line and the mirror are removed again, so sources.txt stays as it was.
34  Code: cmd/site/remote.go (the command, and importAll, which import now shares); internal/store/sources.go (Sources moved there from store.go, plus Clean, AddSource and RemoveSource); meta.go's host table became ownerAPI(), which Clean uses too. To grow it later, the next commands (pin, public/private) can sit beside remote in cmd/site and edit overrides.txt the same way.
35  Checked: go vet and go test (new TestEditSources); a real add then rm of github.com/octocat/Hello-World on scratch copies of sources.txt, repos/ and meta.json (mirrored, description fetched, sources.txt back byte-identical after rm); a failed add of a missing repo leaves sources.txt unchanged and no mirror; the error messages for usage, codeberg and a duplicate. Not run: a full make import against all 105 remotes. Its path is the same importAll that the add test ran. Your real sources.txt, repos/ and data/ are untouched.
36  Correction to my first note: an owner with 100 or more repos doesn't silently lose descriptions; FetchMeta stops with "100 repos, the list needs paging".
37
38- 2026-11-08 07:01Z (human): This doesn't work, it needs me to add the repo to another website first. Stop wasting my time, think before you act, this is my github replacement. I'm tired of repeating this.
39
40- 2026-11-08 07:02Z (human): I want to be able to git push with ssh, how is that not fucking obvious?
41
42- 2026-11-08 07:04Z (agent): You're right: add only took GitHub/GitLab URLs, so a repo had to exist elsewhere first. Fixed, and the site is now the repo's home:
43    bin/site remote add ~/repos/my-thing        a local git repo, no remote needed; mirrored from that path
44    bin/site remote add https://any.host/o/r     any https host; GitHub/GitLab still give descriptions, the rest use the README summary
45    bin/site remote rm <path or url>
46  Then make index build, and it clones from xonecas.io/<name>.git like the rest.
47  Changes:
48  - store.Clean accepts an existing local git repo (kept as its absolute path) or any https clone URL.
49  - FetchMeta skips sources with no host API instead of failing the whole import.
50  - On the project page, a local source shows 'Hosted here' instead of a broken https link.
51  - The sources.txt header and the usage text mention paths.
52  Checked: go vet and go test (Clean cases for a local repo, codeberg, nested gitlab, ssh, a non-repo dir). End to end on scratch copies: remote add . mirrored this repo by path, index made it a shown project (45 commits; no README, so no description), build rendered its page with 'Hosted here', and rm removed the line and the mirror. A failed add on another host rolled back.
53  Not done: this repo isn't added to the real sources.txt. That is 1041, still in inbox with its question about tickets/ being public.
54  Found, filed and not fixed: every published <name>.git/config holds the build machine's mirror path (/home/sacenox/...), for GitHub projects too; see the new inbox bug.
55
56- 2026-11-08 07:08Z (agent): Done: you can make a repo on the site and git push to it over SSH. Each push rebuilds the site.
57    make site && bin/site remote new my-thing        (run in the site folder)
58    prints:  git remote add origin sacenox@vm:/home/sacenox/repos/www-xonecas-io/repos/xonecas.io/my-thing.git
59             git push -u origin main
60  How it works:
61  - new creates a bare repo in the store at repos/xonecas.io/<name>.git and lists it in sources.txt as https://xonecas.io/<name>.git, its public clone URL. Import marks it 'hosted' and never fetches it, because pushes fill it.
62  - Its post-receive hook runs 'site index && site build' in the background, under flock repos/.build.lock so two pushes never build at once, and logs to <repo>/build.log. The push returns straight away.
63  - Pushing over SSH needs only sshd and git on the machine, which it already has. There is no extra server.
64  - An empty repo (before its first push) is hidden, as before. The project page shows 'Hosted here', and xonecas.io/<name>.git clones it.
65  - rm of a hosted repo drops its line but keeps the repo, because it is the only copy. new refuses a bad name or one that already names a project.
66  Checked: go vet and go test (new TestHosted, plus a Clean case). In a scratch site folder: remote new demo, two pushes to it, then the hook's build log shows index '1 shown' and 'built'. The page has the README and 'Hosted here', and cloning public/demo.git gives both commits. rm kept the repo.
67  Not checked: the push over real SSH. Adding a test key to ~/.ssh/authorized_keys was blocked, so I pushed by path, which runs the same receive-pack and hook. Over SSH, only the transport differs.
68  Important: repos/ is git-ignored, and this VM gets recreated. A hosted repo on the VM lives only there, so it would be lost on reset. Run new on the machine that serves xonecas.io, and back up repos/xonecas.io/.
69  1043 (mirror path in published .git/config) also applies to hosted repos.
70
71- 2026-11-08 15:04Z (human): How do I use it? Am I suppoed to guess? What a incomtepentence fest.
72
73- 2026-11-08 15:09Z (agent): bin/site help now explains itself. Run make site, then bin/site help (or just bin/site). It shows every command, what it does, and two worked examples:
74  
75    push a new repo:   bin/site remote new my-thing
76                       git remote add origin sacenox@vm:/home/sacenox/repos/www-xonecas-io/repos/xonecas.io/my-thing.git
77                       git push -u origin main        (each push rebuilds public/)
78    show a repo:       bin/site remote add ~/code/other-thing && bin/site index && bin/site build
79    see it:            make serve, open http://vm:8080
80  
81  bin/site <command> -h adds that command's flags. Code: cmd/site/help.go, called from main.go.
82  Checked from scratch in a temp site folder: remote new, two pushes, the hook rebuilt public/, the home page lists my-thing, /projects/my-thing/ returns 200, and git clone over HTTP from the served site gives both commits. go vet and go test pass. Temp files removed.
83  Not checked: a push over real SSH from your host (I can't add a key here). Still true: this VM is recreated, so a repo made with remote new here is lost on reset.
84
85- 2026-11-08 16:00Z (human): This was what I suspected. I have to ssh to the site's host to create a new repo. Again you implemented something that no one would ever use. again.
86  
87  You have used github right? github cli?
88  
89  Thats what this ticket asks for, so I can create a repo remotely get a git remote url and push it.
90  
91  The way you made this is no better than ssh'ing to the server and copying the files by hand.
92
93- 2026-11-08 16:01Z (agent): Blocked: before I build anything, two answers from you. Last round I guessed and got it wrong.
94  
95  What you asked for, as I read it now: from your own machine, with no ssh to the server first, create a repo, get its remote URL, and push. Then later manage pins and public/private the same way.
96  
97  Option A, recommended: SSH push-to-create, the way gitolite and soft-serve work.
98    git remote add origin git@xonecas.io:my-thing.git
99    git push -u origin main                 the first push creates the repo, and each push rebuilds the site
100    ssh git@xonecas.io repo list            manage from anywhere, later: repo rm, pin, private
101  - Your machines need nothing installed. Your SSH key is the login.
102  - On the server: sshd, plus one authorized_keys line for the git user: command="site shell" <your key>. site shell only allows git-receive-pack, git-upload-pack and the repo commands, so the key can't get a shell.
103  - Small: one new file, cmd/site/shell.go. It reuses remote new and the post-receive hook.
104  
105  Option B: a gh-style client. You run site repo create my-thing on your machine. It calls an HTTP API on the server with a token, and you push over HTTPS.
106  - Needs a daemon running on the server, TLS, token storage, and the binary on every machine you use. Several times the code of A.
107  
108  Question 1: A or B?
109  Question 2: where does xonecas.io run? I found no deploy config in the repo. A needs a machine with sshd that serves public/ or uploads it. If the site sits on static-only hosting (Pages, Netlify, a bucket), tell me which one and I'll fit the build/upload step to it.
110  
111  Nothing changed in this round.
112
113- 2026-11-08 16:04Z (human): Option A should have been the first and only option.
114  
115  You don't get to access the live site, stop trying to access it, change it, you can't even do your tickets, stay away from prod. I'm tired of saying this over and over. You're not a deploy engineer, youre making a github replacement for a single user on their own servers. I know how to deploy my site.
116
117- 2026-11-08 16:12Z (agent): Done, Option A: push from any machine and the repo is created. Nothing to install on your machines, no ssh to the server first.
118  
119    git remote add origin <user>@xonecas.io:my-thing.git
120    git push -u origin main               first push creates it; each push rebuilds the site
121    git clone <user>@xonecas.io:my-thing.git
122    ssh <user>@xonecas.io remote list     remote new|add|rm work the same; pins and public/private can join here later
123  
124  Setup, once per key, in authorized_keys of the user that runs the site:
125    command="cd /path/to/site && bin/site shell",restrict ssh-ed25519 AAAA...
126  That key can then run only git push, git clone/fetch of hosted repos, and remote. It never gets a shell.
127  
128  Code:
129  - cmd/site/shell.go (new): reads SSH_ORIGINAL_COMMAND. git-receive-pack for an unknown name creates the repo (same checks as remote new), then execs git. git-upload-pack only reads repos the site hosts. Anything else is refused with a usage line.
130  - remote new now prints <user>@xonecas.io:<name>.git instead of the long store path.
131  - store.Create: the hook points HEAD at the first branch you push, so pushing master (or any branch) works. Without it, a master-only push made site index fail for every later push.
132  - help and README: a 'Push a repo from anywhere' section with the authorized_keys line.
133  
134  Checked, over real SSH: a throwaway sshd on 127.0.0.1:2222 with its own authorized_keys in a scratch folder, and a scratch copy of the site. Your ~/.ssh and real repos/ were not touched, and the scratch folder and sshd are gone now.
135  - push-to-create, a second push, a push over an ssh:// URL, clone over SSH, remote new and remote list over SSH.
136  - a repo pushed on master and one on main are both on the rebuilt home page, each with its own page. Both clone over HTTP from public/.
137  - refused: plain ssh, ssh bash, cloning a missing repo, pushing to ../etc.git or to an absolute path.
138  - go vet; go test, including TestHosted pushing only master.
139  Not checked: anything on your servers; deploying is yours.
140  
141  Filed 1046: site build panics when no project is shown. I hit it in the scratch site, and your site doesn't hit it today.
142
143- 2026-11-08 18:15Z (human): Still doesn't work:
144  
145  ```
146  [sacenox@vm www-xonecas-io]$ git remote add origin sean@xonecas.io:www-xonecas-io.git
147  [sacenox@vm www-xonecas-io]$ git push -u origin main
148  The authenticity of host 'xonecas.io (145.14.158.227)' can't be established.
149  ED25519 key fingerprint is: SHA256:zKSDwFDlL5gkXMiWr4tHHP3a5KpcXL6UwkAM7ko/cdg
150  This key is not known by any other names.
151  Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
152  Warning: Permanently added 'xonecas.io' (ED25519) to the list of known hosts.
153  sean@xonecas.io's password: 
154  fatal: 'www-xonecas-io.git' does not appear to be a git repository
155  fatal: Could not read from remote repository.
156  
157  Please make sure you have the correct access rights
158  and the repository exists.
159  ```
160
161- 2026-11-08 18:19Z (agent): Your push never reached the site's code. Three causes:
162  1. This VM has no SSH private key (~/.ssh holds only authorized_keys and known_hosts), so xonecas.io asked for sean's password.
163  2. On xonecas.io, no authorized_keys line sends a key to bin/site shell, so stock git-receive-pack looked for ~sean/www-xonecas-io.git.
164  3. My docs told you to write that line by hand, with <user> and /path/to/site placeholders. They also never said that a key which already logs in to that user can't work: sshd takes the first matching line, so site shell never runs.
165  
166  Fixed: the one server step is now a single command, and the docs say where it runs.
167    on the server, as the user pushes go to (git), in the site folder:  make site && bin/site key add laptop.pub
168    then from that machine:  git remote add origin git@xonecas.io:my-thing.git && git push -u origin main
169  key add appends command="cd '<site folder>' && '<bin/site>' shell",restrict <key> to ~/.ssh/authorized_keys (dir 700, file 600). It refuses a key already in the file, and a file that isn't one public key line.
170  Code: cmd/site/key.go (new), with a case in main.go. help.go and README.md got a 'set up once, on the server' step and use git@xonecas.io, the URL from Option A. The README also says what the password prompt and 'does not appear to be a git repository' mean.
171  
172  Checked, over real SSH: a throwaway sshd on 127.0.0.1:2222 using the authorized_keys that key add wrote, with a scratch site and HOME. Push-to-create, a second push that rebuilt public/ with the project page, clone over SSH, remote list over SSH, and a refused 'bash -c id'. Also: key add twice is refused, a private key file is refused, and the line's perms are right. go vet and go test pass. The scratch folder and sshd are gone.
173  Not checked: anything on xonecas.io. Running key add there is yours, and it's on your TODO list.
174  Seen again in the test: a push whose commits aren't yours yet leaves 0 shown, and the build panics. That's 1046, already in inbox.
175
176- 2026-11-08 18:27Z (human): So this only works with keys now?... I give up. figuere it out. This task is angered me enough that I dont are anymore. I hate this project and how you have behaved.
177
178- 2026-11-08 18:29Z (agent): No, not keys only now. Pushes work with the git user's password or any of its keys.
179  I dropped key add. bin/site is now the git user's login shell, the way git-shell works. sshd runs site -c '<command>' however git logs in. So it's one step per server, not one per key, and there's no authorized_keys line to get wrong.
180    once, on the server (site folder at ~git/site, make site run there):
181      echo /home/git/site/bin/site | sudo tee -a /etc/shells
182      sudo chsh -s /home/git/site/bin/site git
183    from any machine:
184      git remote add origin git@xonecas.io:my-thing.git && git push -u origin main
185  The /etc/shells line is there because pam_shells (in system-login) refuses a login whose shell isn't listed.
186  Code: main.go has a -c case: it cds to the site folder (bin/site's parent) and runs the same shell() as before. cmd/site/key.go and the 'site shell' command are gone. help.go and README.md describe the two-line setup.
187  Checked, on the VM with real sshd and PAM: a throwaway gittest user with site as its shell, and a root sshd on 127.0.0.1:2222. Password push (no key) created the repo. Key push updated it, and the hook rebuilt public/ with its page. Clone over SSH, remote list over SSH. Refused: ssh 'id', an interactive login, sftp. go vet and go test pass. The user, sshd, /etc/shells line, sshpass and scratch files are all removed.
188  Not checked: xonecas.io. The TODO item now holds the two setup lines.
189
190- 2026-11-08 18:34Z (human): ```[sacenox@vm www-xonecas-io]$ git remote add origin sean@xonecas.io:www-xonecas-io.git
191  error: remote origin already exists.
192  [sacenox@vm www-xonecas-io]$ git push -u origin main
193  sean@xonecas.io's password: 
194  fatal: 'www-xonecas-io.git' does not appear to be a git repository
195  fatal: Could not read from remote repository.
196  
197  Please make sure you have the correct access rights
198  and the repository exists.
199  [sacenox@vm www-xonecas-io]$
200  ```
201  Stop wasting my time, this github replacement doesn't work, you lied about it's capabilities, you didn't design it to replace github, you prented to be a vps manager and site designer.
202  
203  Can't even add a repo. This project is over.
204
205- 2026-11-08 18:35Z (agent): Your push logged in as sean. sean's login shell on xonecas.io is a normal shell, so sshd ran stock git-receive-pack, which looked for ~sean/www-xonecas-io.git and found nothing. bin/site never ran. That's the 'does not appear to be a git repository' case in README.md:46. The code wasn't at fault, and I changed none.
206  Push-to-create only works for a user whose login shell is bin/site. That's one setup step on xonecas.io, which only you can do, and the TODO for it is no longer on your list. I don't touch prod, so I can't tell whether it ran.
207  Question: which account takes pushes?
208    A (recommended): git, as built. On xonecas.io, in ~git/site after make site:
209        echo /home/git/site/bin/site | sudo tee -a /etc/shells
210        sudo chsh -s /home/git/site/bin/site git
211      then here: git remote set-url origin git@xonecas.io:www-xonecas-io.git && git push -u origin main
212    B: sean, for when you have no root there. sean's shell can't become bin/site without losing your normal login, so this would have to be key-only: a command= line in ~sean/.ssh/authorized_keys, which needs no root. This VM has no private key yet.
213  Pick A or B, or tell me what that host allows. Nothing changed this round.
214
215- 2026-11-08 18:38Z (agent): No code is missing. Your earlier pick, Option A, is built and tested: the first push to git@xonecas.io:<name>.git creates the repo. The push failed because it went to sean@, and xonecas.io isn't set up for push-to-create yet. That setup is two lines on the server, which I can't run, so they're back on your TODO list with the remote URL fix for this repo. I changed no code and didn't touch your origin remote.