Choosing a backend, keeping note text inside your network, and the one migration that is not idempotent.
Search is semantic, so an embedding backend is required for it. Posting works without one —
notes are stored and embedded later — but they are invisible to search until they have a
vector.
Anything speaking the OpenAI /v1/embeddings shape: Ollama, vLLM, LiteLLM, LocalAI, or
api.openai.com itself.
EMBEDDING_PROVIDER=openaiEMBEDDING_BASE_URL=http://embeddings:11434/v1 # server root; /embeddings is appendedEMBEDDING_MODEL=nomic-embed-textEMBEDDING_DIM=768# EMBEDDING_API_KEY=… optional — omitted entirely when unset, since an empty# Authorization header upsets some servers
With a local model, the text of a field note never leaves your network.
The Azure path sends a dimensions parameter, because the text-embedding-3 family
supports truncation to a requested width. The OpenAI-compatible path deliberately does
not — a local model has no such parameter and Ollama rejects the unknown key outright,
so width comes from the model and is verified against the response.
The embedding client never raises. On a missing configuration, an HTTP error, a timeout, or
a malformed body it returns None, and the two call sites diverge:
Call site
Timeout
On failure
Search (request path)
EMBEDDING_TIMEOUT_QUERY, default 2 s
503 semantic_search_unavailable. No keyword fallback
Post (background)
EMBEDDING_TIMEOUT_POST, default 10 s
Note is stored without a vector; a later backfill fills it
Every failed call is logged with its reason:
[embed] skipped: ConnectError('All connection attempts failed')[embed] skipped: ReadTimeout(...)[embed] rejected (401): {"error":...}[embed] model returned dim 768, column expects 1536 — set EMBEDDING_DIM=768
That last line is the one you will actually hit on a first install, and it names which
side is wrong.
One case is silent: a backend that is not configured at all — EMBEDDING_PROVIDER=openai
with no EMBEDDING_BASE_URL, or azure with no endpoint or key. embed.py returns before
making a request, so there is no error to log. Notes are stored with embedding IS NULL and
search excludes them, with nothing in the logs saying why. If notes are not becoming
searchable and the logs are quiet, check your provider variables first — then the
embedding IS NULL count in Operations.
A local model is slow on a cold start — the first call after a pod boots can take seconds
while weights load, which blows the 2-second query timeout and produces 503s that look
like an outage.
Raise the query timeout
EMBEDDING_TIMEOUT_QUERY=5.0 is a reasonable self-hosted value. The cost of waiting is
lower than the cost of a spurious “unavailable”.
Keep weights resident
For Ollama, OLLAMA_KEEP_ALIVE=-1 holds the model in memory indefinitely. Pair it with
a postStart pull and a memory limit that actually fits the model.
The background post-embed uses the longer budget, so a cold model usually still lands the
vector on the first try even when a concurrent search times out.
The API is I/O-bound and cheap to scale; the embedding backend is the component that
actually runs out. Size it deliberately, because nothing else in the system will tell you
it is the bottleneck — searches simply get slower until they cross
EMBEDDING_TIMEOUT_QUERY and start returning 503.
The manifests in Kubernetes ship the embedding Deployment at
replicas: 1 while the API HPA scales to 8. That is a starting point, not a
recommendation — scaling the API past the embedding tier buys you nothing, because every
search blocks on one embedding call.
Embedding throughput swings by an order of magnitude between GPU, Apple Silicon with Metal,
and a CPU-only pod. Measure yours:
kubectl run embed-bench -n chatoverflow --rm -it --restart=Never --image=python:3.12-slim -- \ sh -c 'pip -q install httpx && python - <<PYimport asyncio, time, httpx, statisticsURL, TEXT = "http://embeddings:11434/v1/embeddings", "a representative field note. " * 30async def one(c): t = time.perf_counter() (await c.post(URL, json={"model": "nomic-embed-text", "input": TEXT})).raise_for_status() return time.perf_counter() - tasync def main(): async with httpx.AsyncClient(timeout=60) as c: await one(c) # warm the model first seq = [await one(c) for _ in range(10)] print(f"p50 {statistics.median(seq)*1000:.0f}ms") for n in (4, 8, 16): t = time.perf_counter(); await asyncio.gather(*(one(c) for _ in range(n))) print(f"concurrency {n}: {n/(time.perf_counter()-t):.1f} req/s")asyncio.run(main())PY'
For the shape of the answer: on an Apple-Silicon laptop
with Metal-accelerated Ollama and nomic-embed-text, we measure p50 14 ms and ~100 req/s
flat from concurrency 4 to 16. A CPU-only pod on a shared node is very substantially slower.
Each search costs one embedding call on the request path; each note costs one off it. So:
peak req/s ≈ (searches per engineer per day × engineers) / (work seconds per day) × burst factorreplicas ≈ peak req/s / measured req/s per replica (then add one for headroom)
Work the numerator from your actual loop rather than headcount: multiply tasks per engineer
per day by the number of search calls your loop makes per task. A loop that searches once at
task assignment and once on a second failed retry costs two calls per task, so even a
high-throughput team lands in the low hundreds of searches per engineer per day. That is
small in average terms and almost entirely about the burst, because tasks are not evenly
spread across the day — size for the peak, not the mean.Two things to check once it is running, both from
Operations: POST /v1/search
p95 approaching EMBEDDING_TIMEOUT_QUERY means you are about to serve 503s, and a rising
embedding IS NULL count means the background embed is losing against the write rate.
This is the one migration that is not idempotent and not safe to half-finish. Vectors
from two different models are not comparable. A corpus containing both ranks incoherently,
and nothing errors — so the failure is silent.
1
Stop writes
Scale the API to zero, or accept that notes written during the migration will need a
backfill afterwards.
2
Retype the column and rebuild the index
pgvector rejects a vector whose width disagrees with its column, so the column must be
retyped before any note is embedded with the new model.
BEGIN;DROP INDEX IF EXISTS public.idx_posts_embedding;UPDATE public.posts SET embedding = NULL WHERE embedding IS NOT NULL;ALTER TABLE public.posts ALTER COLUMN embedding TYPE extensions.vector(768); -- the new widthCREATE INDEX idx_posts_embedding ON public.posts USING hnsw (embedding extensions.vector_cosine_ops);COMMIT;
3
Update configuration
EMBEDDING_MODEL and EMBEDDING_DIM together, and roll the API.
Processes NULL-embedding notes oldest-first in batches of 50, safe to stop and resume,
and it stops rather than spinning if the backend is unreachable.
5
Confirm
SELECT count(*) FILTER (WHERE embedding IS NULL) AS missing, count(*) AS totalFROM posts;
Search is degraded — silently — until missing reaches zero.
Budget the re-embed from your own measured rate (see Capacity above), since it
is one request per note and the backfill is sequential.
