@@ -1,45 +1,18 @@
1-# Provider Rate Limiting Plan
1+# Provider Rate Limiting
22
3-## Goal
3+## Overview
44
5-Prevent provider overload and external API rate limiting by enforcing a shared request rate per provider. All myses using the same provider should share a single limiter.
5+Zoea Nova enforces shared request limits per provider to prevent GPU overload (Ollama) and reduce the risk of upstream throttling (OpenCode Zen). Every mysis created with the same provider shares a single limiter.
66
7-## Research Findings
7+## Implementation
88
9-### Ollama (local)
9+- Each provider factory owns a `golang.org/x/time/rate` limiter created at startup.
10+- Providers created by the factory reuse that limiter, so the rate is shared across all myses for that provider.
11+- Requests in `Chat`, `ChatWithTools`, and `Stream` call `limiter.Wait(ctx)` before executing.
1012
11-- No built-in API rate limits for local deployments.
12-- Bottleneck is hardware throughput and concurrency.
13-- We should cap request rate to avoid GPU overload and reduce tail latency.
13+## Configuration
1414
15-### OpenCode Zen (remote)
16-
17-- Public documentation does not specify explicit request rate limits.
18-- Users report stalled requests that are consistent with server-side rate limiting.
19-- We should enforce a conservative request rate and handle 429 responses.
20-
21-### Implementation Approach
22-
23-Use `golang.org/x/time/rate` token buckets. The provider factory owns the limiter and injects it into each provider instance. This makes rate limiting shared per provider rather than per mysis.
24-
25-## Proposed Design
26-
27-### Shared Limiter Per Provider Factory
28-
29-- Each provider factory is instantiated once at startup.
30-- Each factory owns a `rate.Limiter` configured from `config.toml`.
31-- Providers created by a factory share the same limiter reference.
32-
33-### Request Flow
34-
35-1. Mysis prepares a request.
36-2. Provider waits on limiter (`limiter.Wait(ctx)`).
37-3. Provider executes the request.
38-4. If the request fails with 429, emit a rate limit event and return error.
39-
40-## Configuration Changes
41-
42-Add two optional fields to `ProviderConfig`:
15+Rate limits are configured per provider in `config.toml`.
4316
4417 ```toml
4518 [providers.ollama]
@@ -57,60 +30,20 @@ rate_limit = 10.0
5730 rate_burst = 5
5831 ```
5932
60-Defaults if omitted:
61-
62-- `rate_limit`: 2.0 for Ollama, 10.0 for OpenCode Zen
63-- `rate_burst`: 3 for Ollama, 5 for OpenCode Zen
33+## Environment Overrides
6434
65-## Implementation Steps
35+- `ZOEA_OLLAMA_RATE_LIMIT`
36+- `ZOEA_OLLAMA_RATE_BURST`
37+- `ZOEA_OPENCODE_RATE_LIMIT`
38+- `ZOEA_OPENCODE_RATE_BURST`
6639
67-### Phase 1: Config Layer
40+## Request Flow
6841
69-- Add `RateLimit` and `RateBurst` fields to `ProviderConfig`.
70-- Add environment overrides:
71- - `ZOEA_OLLAMA_RATE_LIMIT`
72- - `ZOEA_OLLAMA_RATE_BURST`
73- - `ZOEA_OPENCODE_RATE_LIMIT`
74- - `ZOEA_OPENCODE_RATE_BURST`
75-
76-### Phase 2: Provider Factory
77-
78-- Update `OllamaFactory` and `OpenCodeFactory` to own a limiter.
79-- Add constructors that accept `rate_limit` and `rate_burst`.
80-- Factory `Create()` injects the limiter into the provider instance.
81-
82-### Phase 3: Provider Calls
83-
84-- Add limiter field to `OllamaProvider` and `OpenCodeProvider`.
85-- Wrap all `Chat`, `ChatWithTools`, and `Stream` entry points with `limiter.Wait(ctx)`.
86-- If `Wait` returns an error, return immediately to caller.
87-
88-### Phase 4: Error Reporting
89-
90-- Add a dedicated event type (for UI/telemetry) when rate limit triggers.
91-- On HTTP 429, emit that event with provider name and model.
92-
93-### Phase 5: Tests
94-
95-- Config tests: verify parsing of `rate_limit` and `rate_burst`.
96-- Provider tests: ensure limiter blocks or delays requests.
97-- Commander tests: ensure myses sharing a provider are rate limited together.
98-
99-## Files To Change
100-
101-- `internal/config/config.go`
102-- `config.toml`
103-- `internal/provider/factory.go`
104-- `internal/provider/ollama.go`
105-- `internal/provider/opencode.go`
106-- `internal/provider/provider.go`
107-- `internal/core/events.go`
108-- `internal/core/commander.go`
109-- `documentation/KNOWN_ISSUES.md`
110-- Tests: `internal/config/config_test.go`, `internal/provider/*_test.go`, `internal/core/commander_test.go`
42+1. Mysis prepares a request.
43+2. Provider waits on the shared limiter with `limiter.Wait(ctx)`.
44+3. Provider executes the request if the limiter allows it.
11145
112-## Verification
46+## Behavior Notes
11347
114-1. `make test`
115-2. `make build`
116-3. Manual test: start two myses on the same provider, send rapid messages, observe throttling
48+- When the limiter blocks, the call waits until tokens are available or the context is canceled.
49+- Limiter errors are returned to the caller without additional event emission.