GAD Editorial · Published · · English study · Chinese summary · Study dated 2026-10-02
An architectural assistant may describe a photograph fluently while linking it to the wrong plan. That error matters when a practice assembles precedent studies, checks image archives or prepares a client presentation. ARCH-B, submitted on 28 September 2026, makes this relationship between representations the subject of a dedicated research benchmark. Submission record
The study evaluates 25 multimodal models using 354 four-choice questions across 11 task types. Its evidence includes photographs and architectural drawings. The human comparison uses non-expert participants, while scoring considers the selected answer rather than the accompanying explanation. Invalid answer formats and terminal inference failures count as incorrect. These details shape what the leaderboard can support. Original paper
Start with the office task
GAD recommends translating the research question into a narrowly defined local trial. For an image archive, the task might be to suggest which project record deserves a human check. For a precedent study, it might be to flag a photograph that appears inconsistent with the supplied plan. Keep those tasks separate: a useful candidate suggestion and a reliable final match require different evidence.
Prepare a small, permission-cleared set of project records with answers established by someone who knows the source material. Include ordinary cases, visually similar buildings and deliberately incomplete records. Remove filenames or captions that reveal the answer if the intended test concerns visual reasoning. Preserve them in a separate ground-truth register so the review remains auditable.
Ask for evidence that can be inspected
For each proposed match, request a few observable correspondences and the image locations that support them. Suitable examples include the position of a courtyard, an offset volume or an unusual sequence of openings. Have the reviewer check those statements independently. A correct selection accompanied by invented visual evidence should be recorded as a separate failure.
Also allow the system to say that the available evidence is insufficient. In a real archive, the correct answer may be absent, whereas a four-choice benchmark supplies a fixed set. Test a missing-answer case before using confidence language in a client-facing workflow. Treat the proposed abstention test as an office extension, not a result demonstrated by ARCH-B.
Read the score in layers
The paper's task-level variation supports examining the particular transition a team needs. A photograph-to-plan workflow should have its own test results. GAD would additionally separate wrong matches, unreadable responses and sensible refusals, because each produces a different operational problem.
Keep the selected model version, image preparation settings and review date with the trial. Recheck representative cases after a model or workflow change. Record the time needed to verify suggestions so an apparently fast first answer does not conceal a slow correction process.
ARCH-B offers a useful way to ask a sharper question about architectural AI. It does not demonstrate professional competence across design tasks or the accuracy of measurements extracted from drawings. GAD has not rerun the benchmark. The practical next step is a permission-cleared, task-specific trial whose outcome explains when automated matching saves work and when a person should inspect the original record. Evaluation scope