Put your adapter to work.

Your LoRA contains the small changes learned during training. To answer questions, load it alongside the same base model and tokenizer you trained with.

Have a .gpuless file? That is GPUless’s verified native format. For a regular Transformers setup, use Download for Transformers on the completed job. It gives you the same LoRA weights in PEFT format, with a ready-to-run chat script.

1. Download and unpack

Open your completed job and choose Download for Transformers. Unzip it into a new folder. You should see:

adapter_config.json
adapter_model.safetensors
gpuless_provenance.json
chat.py
README.md

The ZIP contains the adapter, not the full model. Keep the original base weights available. For an uploaded model, use the original model folder on your own machine; GPUless deletes its temporary copy after training. For an operator-provided model, use the matching base download and revision from your operator. The GPUless revision in the provenance file may be a content digest rather than a Hugging Face commit.

2. Install the inference tools

Use Python 3.10 or newer in a fresh virtual environment. These Transformers, PEFT and Accelerate versions are the tested combination:

python -m venv .venv
source .venv/bin/activate
python -m pip install torch --index-url https://download.pytorch.org/whl/cpu
python -m pip install safetensors \
  "transformers==4.57.6" "peft==0.17.1" "accelerate==1.10.1"

On Windows, activate with .venv\Scripts\activate instead. The command above installs CPU PyTorch. For GPU inference or another platform, choose the matching build from PyTorch’s installation page.

3. Ask a question

Run this from the unpacked adapter folder, replacing the model path with your original base-model directory:

python chat.py \
  --base-model /path/to/original/Qwen3-8B \
  --question "In Windows Internals, what problem does AutoBoost address?"

The script loads the model on the CPU and adds your LoRA through PEFT. It uses the tokenizer’s chat template and disables Qwen3’s thinking mode so the response budget goes to the answer. For a base model trained with plain completion prompts, add --raw.

To keep the model loaded while asking several questions, omit --question. Type /quit to exit. Each question starts a fresh exchange.

python chat.py --base-model /path/to/original/Qwen3-8B

For a Hugging Face repository, pass its exact training commit:

python chat.py --base-model Qwen/Qwen3-8B \
  --revision YOUR_TRAINING_COMMIT \
  --question "Your question"

CPU inference defaults to bfloat16 weights. Qwen3-8B needs about 16 GB for those weights, plus runtime memory; allow roughly 24 GB of RAM for a short single-user run. --dtype float32 uses more memory. Use --device cuda only when your GPU has enough memory for the base model. --threads 24 can help on a larger CPU host.

Use it in your own Transformers code

You can replace the bundled script with your own application. Load the base model first, then the unpacked adapter folder:

import accelerate
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_path = "/path/to/original/Qwen3-8B"
tokenizer = AutoTokenizer.from_pretrained(base_path)
base = AutoModelForCausalLM.from_pretrained(
    base_path, dtype=torch.bfloat16, use_safetensors=True,
).eval()
model = PeftModel.from_pretrained(base, "./my-adapter").eval()

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Explain AutoBoost briefly."}],
    tokenize=False, add_generation_prompt=True, enable_thinking=False,
)
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
with torch.inference_mode():
    output = model.generate(
        **inputs, max_new_tokens=96, do_sample=False,
        pad_token_id=tokenizer.eos_token_id,
    )
print(tokenizer.decode(
    output[0, inputs.input_ids.shape[1]:], skip_special_tokens=True,
))

Already downloaded a native file?

You can return to the job and download its Transformers export. If you have the GPUless source and native library installed locally, you can also convert an existing file without loading the base model:

python gpuless_export.py \
  --adapter my-adapter.gpuless \
  --output ./my-adapter

The converter verifies the native artifact before exporting its A/B matrices and scaling. It requires the GPUless native library (GPULESS_NATIVE_LIB if it is outside build/lib). Running the exported PEFT folder afterward only needs the inference tools above.

Return to your jobs