Guide Multi Reviewer

Working with several reviewers

An SLR is normally annotated by two or more people independently, then reconciled — that's what makes agreement between them mean something. Set up a project for multiple reviewers (see Setting up a project) and SaiLoR handles the rest.

Picking a seat

Opening a multi-reviewer project asks who you are before showing you anything — an answer that isn't attributable to a specific reviewer is worse than no answer at all.

It's asked once per project; switch seats any time from the toolbar's reviewer switcher at the top. Each paper remembers the seat you last read it in and switches back to it when you return, so if you are Reviewer 1 on some papers and Reviewer 2 on others, you only choose once per paper. A paper you haven't opened yet keeps the seat you're in.

Each numbered reviewer sees and edits only their own answers — nobody is influenced by what anyone else already wrote. There's always one extra role beyond the numbered reviewers: Consolidation.

Consolidation

Consolidation is not "one more opinion" — it's the pass where the disagreements actually get settled. Every field gets a ⇄ compare button showing every reviewer's answer side by side.

What Consolidation records is Paper.annotations — the project's final result, the one an export or analysis would actually read.

• Where every reviewer agreed, Consolidation fills the field in for you with a light-blue border until you click it — there's nothing to reconcile when everyone already agrees, and copying those across by hand is exactly the task during which a real disagreement further down gets missed. Only case and stray whitespace are forgiven automatically; a genuine difference in wording, or a field one reviewer left blank, is still your call.
• Repeatable groups are lined up for you. Opening a paper in Consolidation works out which of each reviewer's entries are the same entry — two people rarely list the same three findings in the same order — so the compare button lines up answers about the same thing instead of reporting a disagreement that was really just a difference of ordering. A group you've already answered is left alone, not re-matched underneath you.
• Every entry gets a slot, even one only one reviewer recorded. Consolidation lists the union, not just the busiest reviewer's count: if you recorded three findings and your co-reviewer recorded two, one of which you never wrote down, you get four groups — the shared one, filled in as agreement, and three that only one of you has, for you to confirm or delete. Two entries are only ever paired when they are genuinely more alike than not; entries with nothing in common each keep their own group rather than being forced together and reported as a disagreement they were never about.
• Your edits stay put when you switch away and back. The two automatic steps above — lining entries up and filling in agreed answers — only run when a reviewer's answers have actually changed since the last time you looked at that paper. Clearing a field a reviewer filled in, or deleting a group you decided doesn't belong, used to come back the next time you re-entered the seat; it no longer does.
• If a reviewer edits a paper you've already consolidated, you get asked. Their change means there may be new agreement to fill in, but re-running the automatic steps could write over what you decided. So a dialog offers Update (fold their new answers in) or Keep My Version (change nothing). Escape and clicking outside both mean Keep. You're only asked once per change they make.
• Your own answers are never rearranged. Lining the entries up is bookkeeping Consolidation keeps to itself: in your own reviewer seat your groups stay exactly as you typed them, in your order, with no blank entries inserted to match somebody else's count.
• Consolidation signs papers off too. The panel's checkbox reads Consolidation finished in this seat, and works exactly like a reviewer's (see Marking a paper finished): the paper list's dot fills as the consolidated record fills, turns green when you tick the box, and goes red if you tick it while a required field is empty. The filter dropdown and its counter work here as well, so "which papers have I still not settled" is one glance. Whether every reviewer has answered a paper yet is a separate fact: it's in each dot's tooltip and in the · 3/10 ready half of the counter, and it's what the ⇄ button waits for before it will open.

Clicking a compare button shows every reviewer's value, and whether they agree:

Two reviewers can write RCT and randomized controlled trial and mean the same thing, which no amount of text comparison can know on its own. Tick "These answers mean the same thing" and the app counts it as agreement from then on — in the badge, in the disagreement list, and in the statistics. It's worth doing before you quote a κ anywhere, since until you do, the number understates how much your reviewers actually agreed. Ticking it settles that they agreed, not what — you still click one of the answers to record the value; trying to leave without doing so pauses you rather than silently recording an empty "agreement".

⚠ Disagreements and ⚖ Agreement

Two buttons above the annotation fields, visible only in the Consolidation seat:

⚠ Disagreements lists every field the reviewers answered differently on the current paper — click one to jump straight to it. Its Overview button opens the same list project-wide, one row per paper with a disagreement, to reach any of them without hunting paper by paper. ⚖ Agreement reports the usual inter-rater coefficients — Cohen's κ, Fleiss' κ, and Krippendorff's α — tick any combination you want.

Both the per-paper list and the project-wide overview have an Export button — Copy to clipboard or Save to file… — producing plain text: each paper's ID, authors, and title, then every disagreement under it as the exact field location followed by each reviewer's value, one per line. The project-wide export covers every paper with at least one disagreement; a paper with none is left out rather than padding the file with an empty entry.

A coefficient that can't honestly be computed for your project yet (nothing overlapping between reviewers so far, or Cohen's needing exactly two raters) is greyed out and explains why on hover, rather than silently reporting a number that wouldn't mean what it appears to.

⇊ Adopt all unanimous runs the same "everyone agreed, fill it in" logic across every paper at once — aligning repeatable groups first, exactly as opening each paper individually would — instead of you clicking through a hundred papers just to trigger the auto-fill on each one. A paper you've already partly consolidated is left alone.

Reviewer-seat identity

A seat is a role on a paper, not a person. A large review often divides its papers among more people than it has seats — six reviewers, sixty papers, each paper read by two of them — so "Reviewer 1" is a different person on different papers. The seats you pick (see above) are remembered on your own machine only, keyed to that project file; nothing about who you are is written into the project.

What must not happen is two people reading the same paper in the same seat: both write the same file, and whoever merges last replaces the other's answers. If you're using Git support, SaiLoR checks for exactly that. When the paper you're on was last committed in your current seat by someone else, a notice above the form names them. It doesn't block you — a legitimate handover looks the same — but if it wasn't meant to be you, switch seats or pick another paper. It says nothing about other papers, other seats, or your own earlier commits, and it only knows what has been committed and pulled.

Without git there is no such check — see Things to know.