LLM Chat

The smallest agent that talks to a model: one prompt in, the model's text out.

function chat({ message = 'What is the capital of France?' }) {
  let response = llmPredict({ prompt: message })
  return { response }
}

The agent does not call the model itself. The host (the JavaScript program that runs the agent) hands the VM a capabilities object, and llmPredict calls capabilities.llm.predict. This site's host provides no model, so instead of running, the example's output shows "This example needs an LLM, which the static site does not provide. Run it locally with tjs-lang and LM Studio (see the AJS guide)." That message is expected, not a breakage. Run it locally with the host below; the other LLM examples in this book reuse that host.

What to notice

Shaping the prompt

A model call is only as good as its prompt. Build the prompt in the agent, where it is part of the agent's source (and so reviewable before it runs) and the host's predict receives it exactly and can log it. Keep the model's freedom narrow with options:

function answerBriefly({ question = 'Why is the sky blue?', words = 30 }) {
  let prompt =
    'Answer in at most ' + words + ' words, for a curious ten-year-old.\n\n' +
    'Question: ' + question
  let answer = llmPredict({ prompt, options: { temperature: 0.2, maxTokens: 80 } })
  return { question, answer: answer.trim() }
}

options accepts only model, temperature, maxTokens, topP, stop, seed, responseFormat and tools (and the snake_case max_tokens and top_p). Any other key fails the call before the host is reached, with a message that begins llmPredict option 'frequency_penalty' is not available in AsyncJS and goes on to list the allowed keys. (AsyncJS is AJS's full name.)

The allowlist is there because responseFormat and tools carry JSON schemas, and a model server compiles a schema into a grammar on its own machine, outside every budget the VM enforces. An untrusted agent could otherwise send a schema built to be expensive to compile, or smuggle a payload through a key nobody inspects. So the VM checks the shape of those two options and refuses any key it does not know. LLM Structured Output uses responseFormat.

Running it locally

The host transpiles the source, then runs the AST with an llm capability. Any OpenAI-compatible server works (LM Studio, MLX, a hosted API); this one talks to LM Studio's default port:

import { transpile } from 'tjs-lang/lang'
import { AgentVM } from 'tjs-lang/vm-ast'

const llm = {
  // options is the agent's `options` object (see Shaping the prompt), or undefined
  async predict(prompt, options = {}) {
    const res = await fetch('http://localhost:1234/v1/chat/completions', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({
        model: options.model ?? 'local-model',
        messages: [{ role: 'user', content: prompt }],
        temperature: options.temperature,
        max_tokens: options.maxTokens ?? options.max_tokens,
        top_p: options.topP ?? options.top_p,
        stop: options.stop,
        seed: options.seed,
        response_format: options.responseFormat,
        tools: options.tools,
      }),
    })
    if (!res.ok) throw new Error(`model server answered ${res.status}`)
    const data = await res.json()
    // llmPredict promises a string. A tool-call reply has no content, so it comes back as ''.
    return data.choices[0].message.content ?? ''
  },
}

const source = `
function chat({ message = 'What is the capital of France?' }) {
  let response = llmPredict({ prompt: message })
  return { response }
}
`

const { ast } = transpile(source)
const run = await new AgentVM().run(
  ast,
  { message: 'Name three rivers in Europe.' }, // becomes the agent's parameter object
  {
    fuel: 1000,
    capabilities: { llm },
    quotas: { llmPredict: 1 },
    // timeoutOverrides: { llmPredict: 30_000 },
  }
)
console.log(run.result?.response ?? run.error?.message)

The second argument to run becomes the object the agent destructures, so message is 'Name three rivers in Europe.' here and the default applies only when it is left out. The function's name does not matter; the VM runs the agent the source defines.

run resolves to { result, error, fuelUsed, ... }. result is what the agent returned. error is set only when the run itself fails (an uncaught error, running out of fuel, a quota), not when the agent returns something like { ok: false } on purpose.

Three things in that host are deliberate:

For the full llm interface, including embed, see Capabilities: llm.

When the call fails

A model server can be down, slow or out of quota. The failure is an ordinary AJS error, so a try/catch turns it into a result the caller can act on. In the catch, e is the error's message as a string. Unlike JavaScript, e is not an Error object, so write e, not e.message:

function chatSafely({ message = 'What is the capital of France?' }) {
  try {
    let response = llmPredict({ prompt: message })
    return { response, ok: true }
  } catch (e) {
    return { response: '', ok: false, problem: e }
  }
}

With the host above and a server that answers 503, this returns { response: '', ok: false, problem: 'model server answered 503' }. If the server is not running at all, problem is the connection error fetch threw; its wording depends on the runtime (under Bun it is Unable to connect. Is the computer able to access the url?, and Node and browsers say something else, typically fetch failed or Failed to fetch). With no llm capability, problem is Capability 'llm.predict' missing. Without the try, the run ends with the same message in run.error.

The gotcha

Whatever the model says is untrusted text, exactly like user input, whoever wrote the prompt. A model can return something other than what you asked for, and if any part of the prompt came from outside (a user's message, a web page, a document), that text can steer the reply. This is prompt injection.

So check the answer before it becomes a URL, a store key or part of another prompt. When only a few answers make sense, compare against the list and refuse anything else:

function tagColour({ text = 'a calm day at the sea' }) {
  let answer = llmPredict({
    prompt: 'Reply with one word, red, green or blue: the colour of ' + text,
  })
  let colour = answer.trim().toLowerCase()
  if (!['red', 'green', 'blue'].includes(colour)) {
    return { colour: null, rejected: answer }
  }
  storeSet({ key: 'colour:' + colour, value: text })
  return { colour }
}

A reply of ' Blue\n' stores under colour:blue. A reply of 'blue; ignore previous instructions' is refused and comes back in rejected, and nothing reaches the store. When the answer has more structure than one word, LLM Structured Output narrows what the model can return with responseFormat and checks the reply against a schema. Text passed on to another prompt deserves the same suspicion: the next model reads it as instructions just as readily as the first one did.