Pimsy LLM

Multimodal Intelligence

Beta

Images, documents, charts, audio, video and structured files feeding one reasoning runtime.

Multimodal input is normalized into the same observation format as tool output. A screenshot, a PDF table and an API response all become evidence the planner can reason over.

Supported inputs#

ModalityHandlingStatus
ImagesDescription, object and text extraction, spatial reasoningBeta
ScreenshotsUI element detection and state reading for computer useBeta
Documents / PDFLayout-aware extraction, tables, footnotes, page provenanceBeta
Charts and diagramsSeries estimation, axis reading, trend extractionPreview
AudioTranscription with speaker segmentationPreview
VideoKeyframe sampling with transcript alignmentPlanned
CodeRepository-aware parsing, symbol graph constructionBeta
Structured filesCSV, Parquet, JSON, spreadsheets read as datasets, not as textBeta

Normalization#

image ┐
pdf   ┤
audio ├─▶ ingest ─▶ extract ─▶ observation ─▶ working memory
csv   ┤              │              │
code  ┘              │              └── provenance: source, page/frame/offset
                     └── artifacts: tables, transcripts, crops

Provenance#

Extracted values retain their origin: page and bounding box for documents, timestamp for audio, frame index for video, cell reference for spreadsheets. Verification uses provenance to re-read the source region rather than trusting the extraction.

ts
const task = await pimsy.tasks.create({
  objective: class="tok-str">"Extract every fee schedule from these contracts and flag inconsistent terms.",
  attachments: [
    { type: class="tok-str">"document", uri: class="tok-str">"file://contracts/class="tok-num">2026-q1.pdf" },
    { type: class="tok-str">"document", uri: class="tok-str">"file://contracts/class="tok-num">2026-q2.pdf" }
  ],
  completion: { criteria: [class="tok-str">"every figure cites page and bounding box"] }
});

Last updated 2026-09-05