ADR-0012: Vacancy upsert stage

Status: Proposed. Implementation available for review.

Date: 2026-09-23

Context

The last vacancy stage, finalize, sent the pipeline result to the Engine API over tRPC. The API validated the source quotes again and stored the vacancy in one transaction. The embed stage built that request: besides the embeddings, it mapped the classification to stored vacancy fields. Classify already validates the same quotes against the same Markdown.

Decision

This ADR renames finalize to upsert in the ADR-0011 sequence:

harvest -> classify -> geocode -> embed -> upsert

Embed only adds embedding: the model and the requirements and context vectors, or null. The upsert worker maps the embed result to the stored vacancy in utils/transform.ts and writes it with the engine-db upsertVacancy query. The query keeps the finalize transaction: it locks the source, checks that the harvest belongs to the same company, reports unchanged content, stores a source version, closes the vacancy of an expired job, and inserts or updates the vacancy with its locations, statements, and embedding. The quotes are not validated a second time.

A missing source or a harvest of another company fails without a retry. A stored job page requests a source noise-filter review for its configuration. When that request fails, only the upsert job retries. A repeated upsert reports the content as unchanged and requests the review again.

Consequences

The vacancy-finalize queue becomes vacancy-upsert, and the worker image becomes engine-worker-vacancy-upsert. The publisher rejects scheduled ingestion requests that still name the vacancy-finalize root. Cancel them with VacancyClient.cancelIngestion; the next harvest submits the sources again.

The Engine API lists flows by root queue from @gigflow/engine-schema. Its flow views do not show vacancy-upsert flows until the API reads the root queue from @gigflow/engine-jobs. The vacancy.finalize route and queries/vacancy-finalize.ts stay until the API restructure removes them.

Verification

Set TEST_DATABASE_URL to a migrated local database whose name contains test. From the repository root, with Redis and PostgreSQL from dev/cli/dev up engine, run:

bun --env-file=apps/engine/apps/workers/vacancy/upsert/.env \
  test --timeout 30000 apps/engine/apps/workers/vacancy/upsert/e2e
bun --env-file=apps/engine/apps/workers/vacancy/embed/.env \
  test --timeout 30000 apps/engine/apps/workers/vacancy/embed/e2e

The upsert test runs the real worker and database without replacements. The embed test replaces only the embedding provider call. Reports are written to .context/verification/vacancy-upsert/ and .context/verification/vacancy-embed/.