Ditch Distance

The Australian Government Information Security Manual (AUISM), published by the Australian Signals Directorate, and the New Zealand Information Security Manual (NZISM), published by the Government Communications Security Bureau, both set out baseline security controls for government systems in their respective jurisdictions.

Australia and New Zealand are close neighbours "across the ditch", with tightly integrated economies and a shared cyber-security heritage, so the two ISMs cover much of the same ground. Where they part company is in style and pace. The NZISM leans prescriptive, written in MUST/SHOULD compliance language. The AUISM prefers outcome statements and, in places, gets quite specific about vendor technology (Microsoft being a common example). AUISM ships a new release roughly every quarter; the NZISM cycle is closer to 15–18 months. The gap that produces is easiest to see at the frontier: AUISM has been iterating on AI system risks for a while now, and NZISM v3.9 doesn't yet cover most of that ground.

Ditch Distance measures how far apart the two ISMs are, control by control. For each control on one side it finds the closest match on the other side using any of five scoring methods; the mechanics are described in Statistical Methods.

Same data, one fixed lens. Every AUISM release is scored against the current NZISM using RRF fusion at threshold 0.75, and each tier is drawn as its own line so you can watch it move from version to version. The dashed line is the median rank-1 RRF score for that release (right axis). When the median climbs and the gap band shrinks, that AUISM release has drifted into ground the NZISM already covers. When the median drops and the gap band widens, the AUISM has moved into new territory the NZISM hasn't caught up on. The second chart flips the direction and asks the mirror question: how well does each AUISM release actually cover the NZISM?

This is not a substitute for the official ISM releases. It's an analytical aid for security and compliance practitioners doing mapping, alignment, and crosswalk work between the two frameworks. If you need the authoritative text, go to the issuing agencies directly:

• AUISM — cyber.gov.au/business-government/asds-cyber-security-frameworks/ism
• NZISM — nzism.gcsb.govt.nz/ism-document

The question this tool is trying to answer is narrow: how much overlap is there at the control level? Not: how philosophically aligned are the two documents in general. So the comparison is scoped to the actual controls, and the surrounding framing material is deliberately left out.

The two source formats aren't symmetric on this front. The AUISM OSCAL catalog contains nothing but control statements — no preface, no principles, no governance narrative. The NZISM CSV/XML export bundles the controls together with broader framing chapters covering information security fundamentals, governance and roles, certification and accreditation.

Leaving those NZISM framing chapters in would make the AUISM look like it doesn't address material that it does, in fact, cover elsewhere in its published materials. The reverse would happen if the AUISM preface and principles were bundled in on the AU side. So the following NZISM chapters are dropped from the corpus before matching begins:

On the Australian side, the corresponding preface material — the Using the Information Security Manual section (Executive Summary, Applying a Risk-based Approach to Cybersecurity) and the Cyber Security Principles — is not part of the comparison either. It never was: the OSCAL catalog only carries control statements, so those sections simply don't reach the corpus. The list is here for symmetry with the NZ side above.

Flip the perspective (AUISM → NZISM or the other way), change the scoring method, or move the match threshold — the tables below update live. The dual score-range slider is useful when you want to focus on one tier at a time. Dragging both handles down into the borderline / gap zone, for instance, hides the strong matches and leaves only the candidates worth investigating for genuine coverage gaps.

Under the hood, each source-side control is matched to its closest counterpart on the other side. There are five ways to do that matching, each with different strengths:

• RRF fusion (default) — Reciprocal Rank Fusion combines the ranks (not the raw scores) of the four methods below using sum(1/(60 + rank)). Consensus picks float to the top; outliers from any one method are damped.
• TF-IDF cosine — fast lexical overlap over 1–2 word n-grams. Rewards shared vocabulary; weak on paraphrase.
• Sentence-transformer embedding — pooled semantic vector (all-MiniLM-L6-v2), cosine similarity. Handles paraphrase but can over-score topically-adjacent controls that don't actually correspond.
• SecureBERT bi-encoder — same shape as the sentence-transformer method but using a cyber-security-domain BERT (ehsanaghaei/SecureBERT) with mean-pooled last-hidden-state, so domain-specific jargon (ACL, MFA, SIEM, SBOM, ...) is weighted more like a security practitioner would weight it.
• Cross-encoder rerank — the top candidates from TF-IDF and the sentence-transformer are re-scored jointly by a cross-encoder (bge-reranker-base) that reads both control statements together. Slower, but visibly reduces the false-positive "strong-solid" matches that pooled cosine sometimes produces.

Why RRF is the default. Every one of the four base methods has a weakness we already know about. TF-IDF can't see paraphrase. The MiniLM sentence-transformer over-scores anything topically nearby. SecureBERT out of the box crushes almost every pair into a narrow 0.95–0.99 window — without a proper fine-tuned sentence head it just isn't a good discriminator. And the cross-encoder is limited to whatever shortlist the cheaper methods put on its plate.

RRF sidesteps most of that by fusing on rank position rather than raw score. A target that shows up near the top for most methods rises regardless of what the absolute numbers say, and a target that only one method likes gets damped. In practice this knocks out a lot of the false-positive "strong-solid" matches you see when the bi-encoder methods run on their own. The other four are still there in the dropdown — often the more interesting signal is where they disagree with RRF, or with each other.

All five methods are precomputed in both directions and can be toggled above without regenerating the report.

Why not use an LLM for this? You can hand an LLM a pair of controls and ask it for a similarity score, and it will cheerfully give you a number that looks about right. The problem starts as soon as anyone asks why. Security and compliance work has to be defensible: an analyst needs to be able to point at a claim ("control X is equivalent to control Y", "control X is a gap") and walk through the reasoning. LLMs don't give you that. They can't faithfully report how they arrived at a score, they invent supporting detail when pressed, and they lean toward calling things a match even when the evidence is thin.

The methods used here are older, but every step is inspectable. Each score is a deterministic function of the input text and the published model weights. You can pull out the intermediate rankings. You can compare one method against another, and the places where they disagree are often the most interesting rows in the table. LLMs still have a role in this kind of work — most naturally as a downstream summariser, explaining a candidate match to a human reviewer once the shortlist is drawn — but not as the primary scoring engine.