CommentRake keeps a record of what has already been removed by your decision. This is a record of fact, not a proposal to remove more.
Three quantities are kept apart and are never added together:
| Quantity | What it means | Where it comes from |
|---|---|---|
| Possible | the upper bound of what is worth looking at: comments flagged by rules | candidate metrics |
| Selected | the exact consequence of your decisions before Apply | the EditPlan |
| Done | what actually left the files | the archive of applied changes |
Mixing them up means telling a lie. A comment flagged by a rule is not promised for removal; one selected in a plan has not been removed yet; only what is recorded in the archive as applied counts as removed.
A measurement is what was counted: how many characters, of what composition, how many comments and physical lines. A measurement asserts nothing and does not go stale.
A methodology is the way of turning a measurement into a number a person reads. Methodologies are refined over time; measurements are not.
That is why the measurement is what gets stored, and next to every number shown the methodology it came from is named. A refined methodology recomputes the past rather than cancelling it.
More than that: the archive keeps the exact original text of every removed comment, so a recomputation is always possible from the primary source rather than from a summary. The numbers in a cleanup session are a speed-up snapshot, not the only copy of the truth.
For every applied change, the text that was in the file and the text that took its place are taken. What counts as removed is the difference: a replacement with shorter text removes the difference, and a replacement with longer text removes nothing.
The measurement breaks the text down into:
- the total number of characters;
- ASCII characters;
- non-ASCII characters;
- combining marks (diacritics), which are not standalone characters.
This breakdown was chosen because every reasonable estimate of volume depends on it: in any tokenizer a non-ASCII character costs more than an ASCII one, and a combining mark costs nothing.
Five units for every ASCII character, eight for every non-ASCII character, zero for a combining mark; twenty units make one estimated token. That is roughly four ASCII characters, or two and a half non-ASCII characters, per token. It is computed in integers, so the same text gives the same answer on any machine.
No tokenizer is used. The result is an estimate, and it is always shown as an estimate: "about N". The exact number of tokens depends on the specific model and its vocabulary, and stating it without running that model's tokenizer would be untrue.
The measurement itself, without conclusions: how many characters were removed, and nothing about tokens. It fits wherever an estimate would claim more precision than the question deserves.
The counting methodology is a subject for discussion, not a dogma. If you think the coefficients were chosen badly, that ideographic writing or emoji need to be accounted for separately, that the character breakdown should be finer, or that a methodology for a specific tokenizer is needed — say so. A new methodology is added alongside the old ones, not instead of them: the numbers shown earlier stay explainable, because the methodology that produced them does not go anywhere.
Changing an existing methodology without a new identifier is not permitted: it would silently rewrite the meaning of numbers that have already been shown.
They do not set a target and do not compete. A large "N saved" figure turns a careful tool into a
scoreboard, and removing a comment into an achievement. The rule from
docs/en/comment-analysis.md applies here as well: an estimate must not become an inducement to
remove.