Rigor Score Specification

Every protocol Olto shows you carries a Rigor Score: a number from 0 to 100 reporting how many reproducibility signals the protocol’s own text contains. It is produced by one pure function, analyzeRigor in src/lib/protocol/rigor.ts. No model is called, nothing crosses a network, and no clock or random source is read, so the same protocol returns the same number on every machine, every time. This page is the specification. Read it and you can reproduce any Rigor Score on this site with a pen.

What the score is for

It answers one narrow question, before anyone runs anything: does this protocol write down the things a reviewer is going to ask about? Controls, replicates, a stated n, the statistical test, real quantities, randomization or blinding, and safety handling are the items that get questioned when a method is examined closely, and the items most often left implicit in a draft. The score puts that gap in front of the author while the protocol is still cheap to change, and in front of a reader deciding whether a published protocol is worth forking.

It is deliberately shallow, and it is published in full for the same reason it is deterministic: so that a reader who distrusts the number can take it apart instead of taking it on trust.

What the score is not

• It is not peer review. No person and no model reads the protocol to produce it.
• It does not assess whether the science is correct. It has no opinion on whether the hypothesis is sound, whether those are the right controls, whether that test suits that data, or whether the experiment will work.
• It does not check that the work was done. It reads the document, not the bench. A protocol that describes controls it never ran scores exactly the same as one that ran them.
• It is not a gate. Nothing in Olto is blocked, hidden, or ranked out of existence by a low score.

What the scorer reads

The input is the protocol’s sections object. collectProtocolText walks it up to five levels deep, collects every string it finds in order, joins them with newlines, lowercases the result, and truncates it at 60,000 characters. That single lowercased string is the entire input.

Structure is discarded on the way in. The scorer cannot tell the title from the safety section, so a term that appears in an unrelated sentence counts exactly as much as the same term in the section it belongs to. Every pattern below is then tested once against that one string, and a signal is either present or absent. There is no partial credit and no per-signal intensity: mentioning replicates once counts the same as describing them in detail.

The 7 signals

Each signal is a single regular expression tested against the lowercased text. The middle column below is a plain-English reading of that expression, written for this page; the file is the authority, and where the two ever disagree the file wins.

Every row contributes the same amount: one signal out of 7. There is no per-signal weighting, no signal worth double, and no bonus for combinations. Safety counts as much as statistics.

The weighting, and how the total is produced

The whole calculation is one line. Count the signals whose pattern matched, divide by 7, multiply by 100, and round to the nearest whole number:

score = round( passed / 7 * 100 )

Because the numerator can only be a whole number of signals, the score has exactly 8 possible values, listed in full below. It is an ordinal reading rather than a continuous measurement: a score that falls between two rows of this table cannot occur.

One signal is therefore worth 14 or 15 points depending on where the rounding lands, which is another way of saying the score is a count of 7 things wearing a percentage sign.

A worked example

Take this excerpt as the entire protocol text:

Seed E. coli at 5 × 10^5 CFU/mL into 96-well plates. Include an untreated vehicle control on every plate. Dose the compound at 0, 2, 8 and 32 µg/mL, then incubate at 37°C for 18 hours. Read OD600 in triplicate. Compare treated and control means with a two-tailed t-test at p < 0.05. Handle all cultures in a BSL-2 cabinet, in gloves and a buttoned lab coat, and autoclave plates before disposal.

Lowercase it and test the 7 patterns against it in turn. The matched text below is shown as the engine sees it, after lowercasing:

Each absent signal is reported back to the author in these words, verbatim:

• Sample size / power: No sample-size justification — a power analysis or stated n strengthens the design.
• Randomization / blinding: If your design assigns subjects to groups or scores outcomes subjectively, add randomization/blinding to reduce bias.

Now the arithmetic. 5 of 7 signals are present, so 5 ÷ 7 × 100 = 71.43 to two decimal places, which rounds to 71. That is the number the product displays for this text, because it is the same function that produced the table above.

One sentence, two signals

Append a single sentence to the excerpt:

Assign wells to conditions from a randomized plate layout, with n = 6 biological replicates per condition.

• n = 6 turns Sample size / power from absent to present.
• randomized turns Randomization / blinding from absent to present.

So the score goes from 71 to 100: 7 ÷ 7 × 100 = 100.00. Note that n = 6 would have satisfied the replication signal as well, had triplicate not already done so. The patterns overlap, and one phrase can move two signals at once.

Where the score is used

• On the generation result and on a protocol’s own page, as a drafting checklist.
• On every public protocol in the open library, rendered on the server so a reader can judge reproducibility before forking.
• On /try, where it is computed in your browser, from your own text, with no account.
• Inside a Reproducibility Passport, where it is the methodological rigor dimension and carries the largest single weight of the passport’s scored dimensions, at 34% of the completeness grade.

The limits

This is a text-pattern instrument. It matches vocabulary, not meaning, and everything below follows from that one fact. Each example is checkable against the specification above.

• It does not read negation. The sentence “no controls were used” satisfies the controls signal, because the pattern looks for the word control and finds it. So does “temperature control” in a sentence about an incubator.
• Punctuation changes the result. p<0.05 satisfies the statistics signal; p < 0.05 on its own does not, because the pattern requires the number to follow the operator with nothing in between. In the worked example above, the statistics signal is carried by t-test, not by the p threshold sitting right beside it. Name your test.
• Near synonyms miss. randomly selected does not satisfy the randomization signal, while random assignment, randomized and blinded all do.
• Patterns overlap. n = 3 appears in both the replication and the sample size patterns, so writing it once can move the score by two signals.
• Presence is binary. Naming a control once and specifying three of them across a paragraph look identical to this instrument.
• It can be gamed. Pasting the vocabulary from the table above into a protocol scores 100 without changing a single thing about the experiment.

We publish the patterns knowing they can be worked around. An instrument you can inspect is one you can argue with; an instrument you cannot inspect has to be believed. Everything on this page exists so that the Rigor Score is the first kind.

Recompute it yourself

• By hand. Lowercase the text, run the 7 descriptions in the signals table over it, count the hits, and read the total off the lookup table.
• In your browser. Generate a protocol at /try, with no account, and the score is computed client-side from the text you get back.
• Against ours. Open any protocol in the public library, score its text by hand, and compare. If your number and ours differ, the source is src/lib/protocol/rigor.ts and we would like to hear about it.

The signal list, the wording, the worked example and the lookup table on this page are rendered by that same function when the page is built, so this specification cannot fall out of step with the engine it describes.