Morphology Bridge: Explain Verb-Tense Confusion in Yellow-Mark Help Panel

1 commit in this PR

be55c77 Morphology bridge: explain inflected forms on a confused/missed mark Stian Haklev 6/3/2026

Overview

The Big Picture

When an Arabic learner taps a yellow 'missed/confused' mark for a word they've studied, the help panel shows a morphological color-band decomposition (clitics, derived forms). But an analysis of confusion-capture data revealed that ~85% of 'Hard' flags are form-recognition failures, and the existing decomposer only explained ~45% of inflected surfaces — it was silent on verb conjugations (present tense, past with gender/number) because these aren't stored in forms_json. Learners who knew the lemma "أَفْسَدَ" but failed to recognize its present-tense "يُفْسِدُ" got no help at all.

This PR introduces classify_surface_morphology() — a shared rule-based classifier that covers the full range of inflection types: verb present-tense (prefix heuristic), other verb conjugations, derived forms matched via forms_json, proclitics, enclitics, and a catch-all inflection category. For the verb-tense cases the color bands can't render, it generates a one-line explanation string (e.g. "present-tense form of «to spoil»"). The function is wired into three places: the analyze_confusion read path (exposed in ConfusionAnalysisOut), the submit-sentence write path (stored per-surface in variant_stats_json so confusion is queryable, not re-derived), and the confusion_help interaction log (records morph_category per yellow tap).

The result is a morphology field flowing from backend service to Pydantic schema to TypeScript type to a new morphBridge UI widget in WordInfoCard. The old _match_surface_form private helper in sentence_review_service is removed — its narrower logic is fully superseded by classify_surface_morphology. 11 new unit tests cover edge cases including the Form-IV past-not-present guard (the lemma "أَفْسَدَ" starts with أ, which is also a present-tense prefix — the classifier must not misidentify the dictionary form itself as present tense).

Architecture

flowchart TD subgraph Before["Before: Gap in inflection coverage"] direction TB B1["Yellow mark tapped"] --> B2["analyze_confusion()"] B2 --> B3["decompose_surface() — handles clitics + forms_json"] B3 --> B4{"form in forms_json?"} B4 -- yes --> B5["Color bands rendered"] B4 -- no --> B6["❌ Verb conjugations: silent"] B2 --> B7["variant_stats_json via\n_match_surface_form()\n(forms_json only)"] end subgraph After["After: Full inflection bridge"] direction TB A1["Yellow mark tapped"] --> A2["analyze_confusion()"] A2 --> A3["decompose_surface() — clitics + forms_json"] A2 --> A4["classify_surface_morphology()"] A4 --> A5{"category?"} A5 -- "verb_present / verb_other" --> A6["explanation:\npresent-tense form of…"] A5 -- "derived_form / proclitic / enclitic" --> A7["category + form_key\n(bands already explain)"] A5 -- "inflection" --> A8["category only"] A6 --> A9["morphology field in response"] A7 --> A9 A8 --> A9 A9 --> A10["WordInfoCard\nmorphBridge widget"] A4 --> A11["variant_stats_json\ncategory + form_key stored"] A4 --> A12["confusion_help log\nmorph_category"] end

Review Tips

✓
confusion_service.py:820-830 — Walk through the Form-IV present-tense guard: not lemma_bare.startswith(core[0]). Trace it with lemma 'افسد' and surface 'افسد' (identity, caught earlier), then surface 'يفسد' (core[0]='ي', lemma starts with 'ا' ≠ 'ي', fires correctly), then surface 'أفعل' for a Form-IV lemma starting with أ (core[0]='أ', lemma_bare starts with 'أ', guard suppresses — correct).
The guard at confusion_service.py (if core and core[0] in _PRESENT_PREFIXES and not lemma_bare.startswith(core[0])) works correctly for all three cases. The test test_form_iv_past_is_identity_not_present confirms the identity case returns None, and test_verb_present_not_in_forms confirms 'يفسد' with lemma_bare='افسد' (starts with 'ا' ≠ 'ي') fires correctly. For a Form-IV lemma starting with 'أ', core[0]='أ' and lemma_bare.startswith('أ') would be True, so the guard suppresses — correct.
✓
confusion_service.py:888-892 — Verify the new has_morph logic: decomposition is not None or bool(morphology and morphology.get('explanation')). This means verb_other also counts as morphological (it always has an explanation). Is that intentional? A past-tense conjugation of a verb the learner knows should probably count as morphological confusion, but confirm this is the desired classification.
The code at confusion_service.py sets has_morph = decomposition is not None or bool(morphology and morphology.get('explanation')). Both verb_present and verb_other always return a non-None explanation, so both count as morphological. The comment in the diff explicitly states 'A verb-tense explanation counts as morphological even when decompose_surface returned nothing', confirming this is intentional design.
⚠
frontend/lib/review/WordInfoCard.tsx:564-578 — Check whether morphBridge can render simultaneously with a non-None decomposition. If a surface has both clitics (decomposition set) and is also a verb tense (explanation set), both the color bands and the bridge text would appear. Based on the classifier logic this shouldn't happen (explanation only set when form_key is None, but decomp can still be set for clitic cases)... verify with a proclitic + verb-present surface like 'ليفسد'.
In classify_surface_morphology, when the surface is a verb with a proclitic (e.g. 'ليفسد'), the code strips the proclitic from core and then checks if core[0] is a present prefix. If it matches, it returns {category: 'verb_present', explanation: '...'}. Meanwhile, decompose_surface may also find the 'ل' proclitic and return a decomposition. In analyze_confusion, both decomposition and morphology.explanation could be non-None simultaneously, causing both the color bands AND the morphology bridge text to render in WordInfoCard. The classifier logic does not suppress explanation when a decomposition exists.
ℹ
backend/tests/test_sentence_review.py:373 — The test seeds surface_form='يُفْسِدُ' but asserts on ulk.variant_stats_json['يفسد']. Confirm strip_diacritics is called before the variant_stats key lookup in submit_sentence_review — if it's not, the key lookup will fail silently (missing key returns empty dict, tests may pass but for wrong reasons).
The diff does not show the key lookup code in submit_sentence_review where vstats[surface_bare] is set. The variable is named surface_bare suggesting diacritics are stripped before use, but the actual stripping call is not visible in the diff (it's in unchanged code). The test would need to pass with the diacritized surface 'يُفْسِدُ' being stripped to 'يفسد' before the vstats key is set.
✓
backend/app/services/sentence_review_service.py:352-363 — The new code calls classify_surface_morphology on every missed/confused surface. This runs decompose_surface() internally. Check the performance profile: is decompose_surface() called twice now (once in the read path when building confusion help, and once here in the write path)? For the submit-sentence write path this is expected, but confirm it's not accidentally called in a hot loop.
The write path in submit_sentence_review calls classify_surface_morphology once per missed/confused surface word when updating vstats. This is a separate code path from the read path (analyze_confusion in confusion_service.py). The call is inside the loop over sentence words but only executes for missed/confused surfaces, which is expected and acceptable for a write path.
⚠
frontend/lib/types.ts:1104-1110 — The category union type will cause a TypeScript error if the backend adds a new category without updating the frontend. This is a feature (schema safety), but it means the two files must be updated atomically. Consider adding a comment cross-referencing the backend constant to make this dependency explicit for future maintainers.
The SurfaceMorphology interface in frontend/lib/types.ts has a strict union type for category with no cross-reference comment pointing to the backend schemas.py or confusion_service.py. The backend comment in schemas.py says # verb_present | verb_other | derived_form | proclitic | enclitic | inflection but neither file references the other, making this a maintenance risk.
ℹ
docs/data-model.md — The updated variant_stats_json description says 'each entry also stores a category'. This is only true going forward — historical rows written by _match_surface_form have no category field. If any analytics or product code reads this field without a default, it could fail on historical data. Verify all reads of variant_stats_json['category'] use .get('category') or equivalent.
The diff only shows the write path in sentence_review_service.py:352-363 setting entry['category'] = morph['category'], which uses dict assignment (not an issue). However, any read-side code consuming variant_stats_json entries and accessing ['category'] directly (without .get('category')) would fail on historical rows — but no read-side code is visible in this diff to verify.

Section 01 · 1 file

Introduce classify_surface_morphology() as the single source of truth

The heart of this PR is a new 70-line function in confusion_service.py. Before this change, the codebase had two separate places trying to understand inflected surfaces: decompose_surface() (which handles clitics and forms_json-stored forms) and _match_surface_form() in sentence_review_service (a narrower helper that only looked in forms_json). Neither could explain verb conjugations like present-tense prefixes.

classify_surface_morphology() acts as a unifying layer above decompose_surface(). It calls the existing decomposer first to get whatever structural information is available (prefix clitics, suffix clitics, matched form key), then applies additional heuristics for the gaps:

1. Definite-article suppression: if the only prefix clitic is ال and there's no form match, that's trivial — return None so the UI doesn't show an unhelpful bridge.

2. Verb present-tense detection: for pos == "verb" with no form_key, strip any proclitic, then check if the first letter is one of the Arabic present-tense subject prefixes (ي ت ن أ). The critical guard: only fire if the lemma itself doesn't start with that letter — this prevents mis-classifying Form-IV past-tense verbs like أَفْسَدَ (which starts with أ) as their own present tense.

3. Other verb conjugations: any verb surface that didn't match forms_json and didn't trigger the present-tense heuristic gets verb_other with a generic conjugation explanation.

4. Derived form / proclitic / enclitic: when decompose_surface() succeeded, map the result to the appropriate category. Notably, explanation is None for these — the color-band UI already shows the breakdown, so a text line would be redundant.

5. Catch-all inflection: surface differs but nothing matched (broken plurals not in forms_json, irregular feminines, etc.).

The function returns None for the dictionary form itself (no gap), and a dict in all other cases. This design — returning None rather than a neutral sentinel — lets callers do simple truthiness checks.

Diagram

flowchart TD IN["classify_surface_morphology\nsurface_bare, lemma"] --> G1{"surface == lemma_bare?"} G1 -- yes --> RN1["return None"] G1 -- no --> DS["decompose_surface()\n→ prefix_clitics, suffix_clitics, form_key"] DS --> G2{"only ال prefix,\nno form_key?"} G2 -- yes --> RN2["return None\n(trivial definite)"] G2 -- no --> G3{"pos==verb and\nno form_key?"} G3 -- yes --> G4{"core[0] in present prefixes\nAND lemma doesn't start with it?"} G4 -- yes --> R1["verb_present\nwith explanation"] G4 -- no --> R2["verb_other\nwith explanation"] G3 -- no --> G5{"form_key set?"} G5 -- yes --> R3["derived_form\nno explanation"] G5 -- no --> G6{"suffix clitics?"} G6 -- yes --> R4["enclitic"] G6 -- no --> G7{"prefix clitics?"} G7 -- yes --> R5["proclitic"] G7 -- no --> R6["inflection\n(catch-all)"]

Form-IV أ guard

Arabic Form-IV verb lemmas are stored in the dictionary as past-tense forms starting with أ (e.g. أَفْسَدَ). The present-tense prefix أ (first-person singular) would match this initial letter, causing the classifier to mis-fire on the lemma itself. The guard not lemma_bare.startswith(core[0]) is exactly what prevents this — reviewers should trace this through the test test_form_iv_past_is_identity_not_present to confirm it holds.

Heuristic vs. full parse

The present-tense detection is a prefix heuristic, not a full morphological parse. It will misclassify some noun/adjective surfaces that happen to start with ي (e.g. يد 'hand'). The tradeoff is acceptable because the classifier only runs for pos == 'verb' surfaces, and the catch-all verb_other is a safe fallback when the heuristic doesn't fire.

Null for trivial cases

Returning None for the identity case and the bare-definite case means callers can write if morph: without inspecting the category. This is a cleaner interface than returning a sentinel like {category: 'identity'} — but it means callers must be null-safe, which the code consistently is.

backend/app/services/confusion_service.py L779–784, L787–852 2 annotations important

backend/app/services/confusion_service.py CHANGED Viewed

@@ -776,6 +776,77 @@ def find_phonetically_similar(
     return results[:max_results]
+# مضارع (present-tense) subject prefixes. أ is included but only fires when the
+# lemma (stored as past) doesn't itself start with it — guards Form-IV pasts (أفسد).
+_PRESENT_PREFIXES = ("ي", "ت", "ن", "أ")
+_PROCLITICS_LONGEST_FIRST = ("وال", "بال", "فال", "كال", "لل", "و", "ف", "ب", "ل", "ك")
+def classify_surface_morphology(surface_bare: str, lemma: "Lemma | None") -> dict | None:
+    """Classify how an inflected surface differs from its dictionary lemma.
+    Returns None when the surface IS the dictionary form (no gap) or carries only
+    the definite article (trivial — not worth a bridge). Otherwise returns:
+        {"category": "verb_present" | "verb_other" | "derived_form"
+                     | "proclitic" | "enclitic" | "inflection",
+         "form_key": <forms_json key matched by decompose_surface, or None>,
+         "explanation": <one-line surface->lemma bridge, or None>}
+    `explanation` is populated only for the verb-tense cases that
+    `decompose_surface` (and therefore the WordInfoCard color bands) cannot
+    render; for proclitic/enclitic/derived_form the bands already show the
+    breakdown, so we leave it None to avoid a redundant line. Used to (a) push
+    the morphology bridge in review, (b) populate `variant_stats_json`, (c) log
+    the morphological cause of a confusion.
+    """
+    if not lemma or not surface_bare:
+        return None
+    lemma_bare = lemma.lemma_ar_bare or strip_diacritics(lemma.lemma_ar or "")
+    if not lemma_bare or surface_bare == lemma_bare:
+        return None
+    decomp = decompose_surface(surface_bare, lemma_bare, getattr(lemma, "forms_json", None))
+    prefix_clitics = decomp.get("prefix_clitics") if decomp else []
+    suffix_clitics = decomp.get("suffix_clitics") if decomp else []
+    form_key = decomp.get("matched_form_key") if decomp else None
+    # Pure definite article (ال + stem == lemma) — trivial, suppress the bridge.
+    if (
+        decomp
+        and not suffix_clitics
+        and not form_key
+        and [c.get("text") for c in prefix_clitics] == ["ال"]
+    ):
+        return None
+    gloss = (lemma.gloss_en or "").strip()
+    lemma_ar = lemma.lemma_ar or lemma_bare
+    of_lemma = f"“{gloss}” ({lemma_ar})" if gloss else f"({lemma_ar})"
+    # Verb tense — only when decompose didn't already match a stored form.
+    if lemma.pos == "verb" and not form_key:
+        core = surface_bare
+        for pro in _PROCLITICS_LONGEST_FIRST:
+            if core.startswith(pro) and len(core) > len(pro) + 1:
+                core = core[len(pro):]
+                break
+        if core and core[0] in _PRESENT_PREFIXES and not lemma_bare.startswith(core[0]):
+            return {"category": "verb_present", "form_key": None,
+                    "explanation": f"present-tense form of {of_lemma}"}
+        return {"category": "verb_other", "form_key": None,
+                "explanation": f"a conjugated form of {of_lemma}"}
+    if form_key:
+        return {"category": "derived_form", "form_key": form_key, "explanation": None}
+    if suffix_clitics:
+        return {"category": "enclitic", "form_key": None, "explanation": None}
+    if prefix_clitics:
+        return {"category": "proclitic", "form_key": None, "explanation": None}
+    # Surface differs but nothing matched (irregular / broken plural not in forms_json).
+    return {"category": "inflection", "form_key": None, "explanation": None}
 def analyze_confusion(
     db: Session,
     lemma_id: int,

L779–784

New module-level constants for present-tense prefix detection. The comment explains the أ guard inline — it's documentation as much as code, because the Form-IV false-positive is the trickiest edge case.

L787–852

The new classifier function, previously absent. Most of this logic was either implicit (the present-tense heuristic) or scattered in two places (_match_surface_form + decompose_surface callers). The docstring is unusually detailed — it explains why explanation is None for certain categories (the bands already cover them), which is a non-obvious design decision that would otherwise confuse maintainers.

Section 02 · 2 files

Wire morphology into analyze_confusion() and extend the API response type

With the classifier in place, the next step is integrating it into the analyze_confusion() read path so it flows through to the API response. Before this change, analyze_confusion called decompose_surface() and used has_morph = decomposition is not None to decide whether the confusion was 'morphological'. The problem: for verb conjugations (no forms_json entry), decomposition is None, so these were silently classified as non-morphological even though the learner's confusion was entirely form-based.

The change adds a call to classify_surface_morphology() after the decomposition step, then broadens the has_morph logic: a verb-tense explanation (even without a decomposition) now counts as morphological. This matters for the confusion_type field — 'morphological', 'visual', 'both', or None — which controls how the UI frames the analysis.

The morphology dict is also added to the returned result dict, and the Pydantic schema ConfusionAnalysisOut gains a corresponding SurfaceMorphology | None field. The new SurfaceMorphology schema is a proper class (not an inline dict), which means Pydantic validates category values and the OpenAPI spec documents the shape correctly.

confusion_type correctness

The broadened has_morph means some confusions previously classified as None or visual will now become morphological or both. Verify that the frontend handles a confusion_type of morphological when decomposition is still None (only morphology.explanation is set) — the WordInfoCard might render an empty decomposition section.

Schema-first API contract

By giving SurfaceMorphology its own Pydantic model rather than leaving it as a raw dict in the response, the OpenAPI schema now documents the exact shape. This is the correct pattern for any data structure that crosses the API boundary — it makes the frontend TypeScript types authoritative rather than inferred.

backend/app/services/confusion_service.py L886–898, L911–912 2 annotations important

backend/app/services/confusion_service.py CHANGED Viewed

@@ -815,8 +886,13 @@ def analyze_confusion(
     # 3. Prefix disambiguation hint
     prefix_hint = _build_prefix_hint(surface_bare, lemma_bare, lemma.root, decomposition)
+    # 4. Morphology classification — coarse category + a one-line surface->lemma
+    #    explanation for the verb-tense cases the color-band decomposition can't show.
+    morphology = classify_surface_morphology(surface_bare, lemma)
-    # Determine confusion type
+    # Determine confusion type. A verb-tense explanation counts as morphological
+    # even when decompose_surface returned nothing (so the bridge data still flows).
-    has_morph = decomposition is not None
+    has_morph = decomposition is not None or bool(morphology and morphology.get("explanation"))
     has_visual = len(similar_words) > 0
     if has_morph and has_visual:
@@ -835,6 +911,7 @@ def analyze_confusion(
         "lemma_ar": lemma.lemma_ar,
         "gloss_en": lemma.gloss_en,
         "decomposition": decomposition,
+        "morphology": morphology,
         "similar_words": similar_words,
         "phonetic_similar": phonetic_similar,
         "prefix_hint": prefix_hint,

L886–898

Replaces the two-line 'Determine confusion type' block. Previously has_morph was purely decomposition is not None. Now it also considers whether morphology carries an explanation — bridging the verb-conjugation gap where decompose_surface returns nothing but the morphology classifier still identifies the form type.

L911–912

Adds morphology to the result dict returned by analyze_confusion. Previously this key was absent entirely — any caller inspecting result.get('morphology') would get None.

backend/app/schemas.py L917–921, L927–928 2 annotations important

backend/app/schemas.py CHANGED Viewed

@@ -914,6 +914,12 @@ class PrefixHint(BaseModel):
     root_meaning: str | None = None
     hint_text: str
+class SurfaceMorphology(BaseModel):
+    category: str  # verb_present | verb_other | derived_form | proclitic | enclitic | inflection
+    form_key: str | None = None
+    explanation: str | None = None
 class ConfusionAnalysisOut(BaseModel):
     confusion_type: str | None  # "morphological" | "visual" | "both" | None
     surface_form: str
@@ -921,6 +927,7 @@ class ConfusionAnalysisOut(BaseModel):
     lemma_ar: str
     gloss_en: str | None = None
     decomposition: MorphDecomposition | None = None
+    morphology: SurfaceMorphology | None = None
     similar_words: list[SimilarWord] = []
     phonetic_similar: list[PhoneticSimilarWord] = []
     prefix_hint: PrefixHint | None = None

L917–921

New Pydantic model for the morphology shape. Previously this data didn't exist in the schema layer at all — it was an ad-hoc dict inside confusion_service. Promoting it to a named schema makes the API contract explicit and enables TypeScript code-gen.

L927–928

Adds the morphology field to ConfusionAnalysisOut. Before this line, the schema serialized the result dict but silently dropped the morphology key — Pydantic's default behavior for unexpected keys. Now it's an explicit optional field.

Section 03 · 1 file

Replace _match_surface_form() with the richer classifier on the submission write path

The submit_sentence_review() function in sentence_review_service.py maintains variant_stats_json on each UserLemmaKnowledge row — a per-surface-form accounting of how many times a word was seen, missed, or confused. Before this PR, when a surface was missed or confused, the code called _match_surface_form() to find if that surface was a known forms_json entry, and if so stored form_key and form_label for future querying.

_match_surface_form() was a 20-line private helper that did essentially the same forms_json lookup as part of classify_surface_morphology(), but without the verb-tense heuristic, without clitics, and without the inflection catch-all. It represented a partial solution to the same problem.

This PR deletes _match_surface_form() entirely and replaces the call site with classify_surface_morphology(). The replacement stores the richer category field on every entry (not just forms_json matches), and stores form_key/form_label when available (derived_form case). The result: per-form confusion data is now queryable at the category level — you can ask 'how many present-tense confusions does this user have for أَفْسَدَ?' without re-running the classifier on read.

Existing variant_stats rows

Existing variant_stats_json entries in the database were written by the old _match_surface_form() logic — they may have form_key/form_label but no category. Any analytics query on category must handle NULL/missing for historical rows. This is expected but worth documenting in the migration runbook.

Queryable confusion categories

Storing category at write time is a meaningful data model improvement. Previously, understanding why a word was confused required re-running classifier logic on read. Now the category is an indexed fact on the JSON column — research queries like 'group confused surfaces by morphological category' become straightforward.

Cross-service import

sentence_review_service now imports from confusion_service. This couples the two services — a change to classify_surface_morphology could affect the write path. The alternative (duplicating the logic or keeping _match_surface_form) was worse, but reviewers should be aware the classifier is now on a hot path (every sentence submission that involves a missed/confused word).

backend/app/services/sentence_review_service.py L20–22, L352–363 2 annotations supporting

backend/app/services/sentence_review_service.py CHANGED Viewed

@@ -20,30 +20,11 @@
     SentenceWord,
     UserLemmaKnowledge,
 )
+from app.services.confusion_service import classify_surface_morphology
 from app.services.fsrs_service import STATE_MAP, parse_json_column, submit_review
 from app.services.grammar_service import record_grammar_exposure
 from app.services.sentence_validator import strip_diacritics, _is_function_word
-_FORM_METADATA_KEYS = {"gender", "verb_form", "pattern", "notes"}
-def _match_surface_form(surface_bare: str, lemma: Lemma | None) -> dict | None:
-    """Return the forms_json key matching a tracked surface, when known."""
-    if not lemma:
-        return None
-    forms = parse_json_column(lemma.forms_json)
-    if not isinstance(forms, dict):
-        return None
-    surface_no_al = surface_bare[2:] if surface_bare.startswith("ال") else surface_bare
-    for key, value in forms.items():
-        if key in _FORM_METADATA_KEYS or not isinstance(value, str) or not value:
-            continue
-        form_bare = strip_diacritics(value)
-        form_no_al = form_bare[2:] if form_bare.startswith("ال") else form_bare
-        if surface_bare in (form_bare, form_no_al) or surface_no_al in (form_bare, form_no_al):
-            return {"form_key": key, "form_label": key.replace("_", " ")}
-    return None
 def submit_sentence_review(
     db: Session,
@@ -371,9 +352,12 @@ def submit_sentence_review(
                         entry["missed"] = entry.get("missed", 0) + 1
                     elif is_confused:
                         entry["confused"] = entry.get("confused", 0) + 1
-                    form_match = _match_surface_form(surface_bare, canonical_lemma_obj)
+                    morph = classify_surface_morphology(surface_bare, canonical_lemma_obj)
-                    if form_match:
+                    if morph:
+                        entry["category"] = morph["category"]
-                        entry.update(form_match)
+                        if morph.get("form_key"):
+                            entry["form_key"] = morph["form_key"]
+                            entry["form_label"] = morph["form_key"].replace("_", " ")
                     vstats[surface_bare] = entry
                     knowledge.variant_stats_json = vstats

L20–22

Replaces the import of nothing (the deleted _match_surface_form was local) with an import of classify_surface_morphology from confusion_service. This creates a new inter-service dependency — sentence_review_service now imports from confusion_service.

L352–363

Replaces the 3-line _match_surface_form call with a call to classify_surface_morphology. The old code stored form_key/form_label only when forms_json matched. The new code always stores category (every inflected form now gets a classification) and conditionally stores form_key/form_label only for derived_form. This is a superset — existing form_key/form_label storage is preserved, and verb conjugation entries now gain a category field.

Section 04 · 1 file

Record morph_category in the confusion-help interaction log

The confusion_help router endpoint already logs interaction telemetry when a user opens the help panel for a yellow mark. This PR adds morph_category to that log entry — drawn from result.get('morphology', {}).get('category'), which is the category string from the new classifier, or None if the surface is the dictionary form or trivially definite.

This is a small change (one line) but it closes a data loop: the original confusion-capture analysis that motivated this PR was done by parsing interaction logs. Adding morph_category to future logs means subsequent analyses will be able to directly group 'what kinds of inflected forms are causing confusion help to be opened' without re-running the classifier retroactively. The .get('morphology') or {} guard is defensive — it handles the case where morphology is None (the classifier returned None) without raising an AttributeError.

Closing the analysis loop

The PR was motivated by an analysis of confusion captures. Logging morph_category means the next analysis can directly measure whether the bridge is reducing confusion rates per morphological category — without needing to re-run the classifier on historical interaction data.

backend/app/routers/review.py L988–989 1 annotation important

backend/app/routers/review.py CHANGED Viewed

@@ -985,6 +985,7 @@ def confusion_help(
         ],
         phonetic_lemma_ids=[w.get("lemma_id") for w in result.get("phonetic_similar", [])],
         has_decomposition=result.get("decomposition") is not None,
+        morph_category=(result.get("morphology") or {}).get("category"),
     )
     return result

L988–989

Adds morph_category to the confusion_help interaction log call. Previously absent — this field was not logged at all. The (result.get('morphology') or {}).get('category') pattern handles both the case where the key is missing and where its value is None.

Section 05 · 2 files

Propagate morphology type to TypeScript and render the explanation in WordInfoCard

The final leg of the pipeline is the frontend. The ConfusionAnalysis interface in frontend/lib/types.ts mirrors the Pydantic ConfusionAnalysisOut schema — it's the TypeScript type that WordInfoCard receives as confusionData. Before this PR, the interface had no morphology field, so even if the backend sent it, TypeScript would reject any access to it.

The SurfaceMorphology interface is added as a proper TypeScript union type for category (not a bare string), providing compile-time safety that the backend's string constants are exhaustively matched. ConfusionAnalysis gains morphology: SurfaceMorphology | null.

In WordInfoCard.tsx, the RevealedView component already destructures confusionData to pull out decomp, similarWords, etc. This PR adds morphExplanation = confusionData?.morphology?.explanation. The UI renders a new morphBridge view only when morphExplanation is non-null — a teal-tinted pill with a git-compare icon and the explanation text. Importantly, this widget appears between the color-band decomposition and the forms strip, so it's visible in exactly the gap where the bands were previously silent.

Placement in revealed view

The morphBridge renders after the decomposition bands and before the FormsStrip. Verify this placement makes sense when BOTH decomposition AND morphology.explanation are present — can a surface form simultaneously have color-band decomposition AND a verb-tense explanation? The classifier logic suggests not (explanation is only set for verb cases where form_key is None, so decomposition would be None too), but it's worth confirming the two never render together.

Category union type

Typing the category as a TypeScript string literal union (rather than string) is the right pattern here. It means a future refactor that adds or renames a category will produce a compile error on the frontend, making the schema boundary explicit and safe.

frontend/lib/types.ts L1101–1113, L1120–1121 2 annotations important

frontend/lib/types.ts CHANGED Viewed

@@ -1101,6 +1101,18 @@ export interface PrefixHint {
   hint_text: string;
 }
+export interface SurfaceMorphology {
+  category:
+    | "verb_present"
+    | "verb_other"
+    | "derived_form"
+    | "proclitic"
+    | "enclitic"
+    | "inflection";
+  form_key: string | null;
+  explanation: string | null;
+}
 export interface ConfusionAnalysis {
   confusion_type: "morphological" | "visual" | "both" | null;
   surface_form: string;
@@ -1108,6 +1120,7 @@ export interface ConfusionAnalysis {
   lemma_ar: string;
   gloss_en: string | null;
   decomposition: MorphDecomposition | null;
+  morphology: SurfaceMorphology | null;
   similar_words: SimilarWordItem[];
   phonetic_similar: PhoneticSimilarItem[];
   prefix_hint: PrefixHint | null;

L1101–1113

New SurfaceMorphology interface with a union-typed category field. Previously this type did not exist — the morphology data would have been any or ignored. The union type means TypeScript will error if a new category is added to the backend without updating the frontend.

L1120–1121

Adds morphology to ConfusionAnalysis. Previously absent — accessing confusionData?.morphology would have been a TypeScript error (or typed as any if the interface was loose).

frontend/lib/review/WordInfoCard.tsx L337–340, L564–578, L1233–1253 3 annotations supporting

frontend/lib/review/WordInfoCard.tsx CHANGED Viewed

@@ -337,6 +337,7 @@ function RevealedView({
   });
   const decomp = confusionData?.decomposition;
+  const morphExplanation = confusionData?.morphology?.explanation;
   const similarWords = confusionData?.similar_words;
   const phoneticSimilar = confusionData?.phonetic_similar;
   const prefixHint = confusionData?.prefix_hint;
@@ -563,6 +564,15 @@ function RevealedView({
         );
       })()}
+      {/* Morphology bridge — one-line surface->lemma link for verb-tense forms the
+          color bands can't decompose (e.g. "present-tense form of «to spoil»"). */}
+      {morphExplanation && (
+        <View style={styles.morphBridge}>
+          <Ionicons name="git-compare-outline" size={13} color={colors.accent} />
+          <Text style={styles.morphBridgeText}>{morphExplanation}</Text>
+        </View>
+      )}
       {/* Forms strip */}
       <FormsStrip
         pos={result.pos}
@@ -1223,6 +1233,21 @@ const styles = StyleSheet.create({
     fontSize: 11,
     color: colors.textSecondary,
   },
+  morphBridge: {
+    flexDirection: "row",
+    alignItems: "center",
+    gap: 6,
+    backgroundColor: "rgba(100, 140, 180, 0.06)",
+    borderRadius: 10,
+    paddingHorizontal: 12,
+    paddingVertical: 8,
+  },
+  morphBridgeText: {
+    flex: 1,
+    fontSize: 13,
+    color: colors.text,
+    fontFamily: fontFamily.translitRegular,
+  },
 });
 const cfStyles = StyleSheet.create({

L337–340

Extracts morphExplanation from confusionData alongside the existing decomp/similarWords destructuring. Previously this variable didn't exist. The optional-chaining pattern matches the nullable field type.

L564–578

New conditional morphBridge widget inserted between the decomposition color bands and the FormsStrip. Previously this area showed nothing for verb conjugations — the learner who failed to recognize يُفْسِدُ got no bridge to أَفْسَدَ. Now they see 'present-tense form of "to spoil" (أَفْسَدَ)'. The git-compare-outline icon is a deliberate choice — it visually suggests a transformation/mapping relationship.

L1233–1253

New StyleSheet entries for the morphBridge widget. The rgba(100, 140, 180, 0.06) background is a very subtle blue tint — distinct from the decomposition bands but not competing for attention. Using fontFamily.translitRegular is appropriate since the explanation text is Latin-script transliteration context.

Section 06 · 2 files

Validate the classifier with 11 targeted unit tests and 2 integration tests

The test suite grows in two places. TestClassifySurfaceMorphology in test_confusion_service.py tests the new function in isolation using a _mk_lemma MagicMock factory — this avoids needing a database and keeps the tests fast. The 11 cases cover: identity (returns None), bare definite (suppressed), present-tense detection, present-tense after a proclitic (li + present), the critical Form-IV false-positive guard, past-tense conjugation (verb_other), derived form from forms_json, broken plural from forms_json, proclitic noun, unknown inflection catch-all, and None-safety.

TestVariantStatsMorphology in test_sentence_review.py tests the write path end-to-end: it seeds a real verb lemma with a present-tense surface in a sentence, runs submit_sentence_review with that verb as confused, and asserts that variant_stats_json on the resulting UserLemmaKnowledge record contains category == 'verb_present'. A second test covers the derived_form + form_key case with a plural. These integration tests give confidence that the classifier is correctly called (not just correct in isolation) and that the database serialization round-trip preserves the data.

Test surface diacritics

test_confused_present_verb_records_category seeds the sentence with surface_form='يُفْسِدُ' (diacritized) but the variant_stats_json key is looked up as 'يفسد' (bare). Verify that the surface_bare stripping in submit_sentence_review handles this correctly — this is an implicit dependency on strip_diacritics being called before the variant_stats lookup.

MagicMock vs. DB fixtures

Using MagicMock for the unit tests and real DB fixtures for the integration tests is a good layered strategy. The unit tests run fast and cover combinatorial cases; the integration tests prove the wiring is correct. Reviewers should check that the MagicMock's getattr(lemma, 'forms_json', None) path is exercised — it's used inside classify_surface_morphology to safely access forms_json on mock objects.

backend/tests/test_confusion_service.py L511–522, L524–590 2 annotations supporting

backend/tests/test_confusion_service.py CHANGED Viewed

@@ -509,3 +511,80 @@ def test_no_phonetic_for_visual_match(self):
             db, 1, "كلب", {10}, candidates=[(word, "known")],
         )
         assert len(results) == 0
+def _mk_lemma(lemma_ar, bare, pos, gloss="x", forms=None):
+    m = MagicMock()
+    m.lemma_ar = lemma_ar
+    m.lemma_ar_bare = bare
+    m.pos = pos
+    m.gloss_en = gloss
+    m.forms_json = forms
+    return m
+class TestClassifySurfaceMorphology:
+    def test_identity_returns_none(self):
+        lem = _mk_lemma("سَيّارة", "سيارة", "noun", "car")
+        assert classify_surface_morphology("سيارة", lem) is None
+    def test_definite_only_suppressed(self):
+        # ال + stem == lemma is trivial; no bridge.
+        lem = _mk_lemma("سَيّارة", "سيارة", "noun", "car")
+        assert classify_surface_morphology("السيارة", lem) is None
+    def test_verb_present_not_in_forms(self):
+        # يفسد is the present of أفسد; forms_json lacks it -> still classified.
+        lem = _mk_lemma("أَفْسَدَ", "افسد", "verb", "to spoil")
+        out = classify_surface_morphology("يفسد", lem)
+        assert out["category"] == "verb_present"
+        assert "present-tense" in out["explanation"]
+        assert "to spoil" in out["explanation"]
+    def test_verb_present_after_proclitic(self):
+        # لِيُعْطِيَ -> li + present of أعطى
+        lem = _mk_lemma("أَعْطَى", "اعطى", "verb", "to give")
+        out = classify_surface_morphology("ليعطي", lem)
+        assert out["category"] == "verb_present"
+    def test_form_iv_past_is_identity_not_present(self):
+        # The lemma itself (أفسد) must never classify as its own present tense.
+        lem = _mk_lemma("أَفْسَدَ", "افسد", "verb", "to spoil")
+        assert classify_surface_morphology("افسد", lem) is None  # identity
+    def test_verb_other_conjugation(self):
+        lem = _mk_lemma("وَقَعَ", "وقع", "verb", "to happen")
+        out = classify_surface_morphology("وقعت", lem)  # past 3fs
+        assert out["category"] == "verb_other"
+        assert out["explanation"] is not None
+    def test_derived_form_matches_forms_json(self):
+        lem = _mk_lemma("خَطَّط", "خطط", "verb", "to plan", {"masdar": "تَخْطِيط"})
+        out = classify_surface_morphology("التخطيط", lem)
+        assert out["category"] == "derived_form"
+        assert out["form_key"] == "masdar"
+        # bands render this case, so no redundant explanation line
+        assert out["explanation"] is None
+    def test_plural_derived_form(self):
+        lem = _mk_lemma("وَرَقَة", "ورقة", "noun", "paper", {"plural": "أَوْرَاق"})
+        # surface bare keeps hamza (strip_diacritics does not normalize it), matching forms_json
+        out = classify_surface_morphology(strip_diacritics("أَوْرَاق"), lem)
+        assert out["category"] == "derived_form"
+        assert out["form_key"] == "plural"
+    def test_proclitic_noun(self):
+        lem = _mk_lemma("نُور", "نور", "noun", "light")
+        out = classify_surface_morphology("بنور", lem)  # bi- + light
+        assert out["category"] == "proclitic"
+        assert out["explanation"] is None
+    def test_inflection_unknown(self):
+        # Surface differs, decompose can't explain, not a verb -> bridge-less category.
+        lem = _mk_lemma("أَبْيَض", "ابيض", "adj", "white")
+        out = classify_surface_morphology("بيضاء", lem)
+        assert out["category"] == "inflection"
+        assert out["explanation"] is None
+    def test_none_lemma_safe(self):
+        assert classify_surface_morphology("xyz", None) is None

L511–522

New _mk_lemma helper — creates a MagicMock Lemma with controllable fields. Previously absent; the existing test class used real DB fixtures. This lightweight factory enables fast in-memory unit tests for the new classifier.

L524–590

TestClassifySurfaceMorphology — 11 test cases covering the full decision tree of the new classifier. The test_form_iv_past_is_identity_not_present case is particularly important: it's the edge case the PR description calls out explicitly, and it validates the not-lemma-starts-with guard.

backend/tests/test_sentence_review.py L345–422 1 annotation important

backend/tests/test_sentence_review.py CHANGED Viewed

@@ -345,6 +345,78 @@ def capture_log(**kwargs):
         assert events[-1]["confusion_candidate_lemma_ids"] == {2: [1]}
+class TestVariantStatsMorphology:
+    """The confused/missed write path classifies the surface form and stores
+    category + form_key on the canonical ULK's variant_stats_json (PR: morphology bridge)."""
+    def _seed_verb_sentence(self, db, surface):
+        # primary noun + a verb whose sentence surface is an inflected form
+        _seed_word(db, 1, "كتاب", "book")
+        verb = Lemma(
+            lemma_id=2, lemma_ar="أَفْسَدَ", lemma_ar_bare="أفسد",
+            pos="verb", gloss_en="to spoil",
+        )
+        db.add(verb)
+        db.flush()
+        db.add(UserLemmaKnowledge(
+            lemma_id=2, knowledge_state="learning", fsrs_card_json=_make_card(),
+            introduced_at=datetime.now(timezone.utc) - timedelta(days=10),
+            last_reviewed=datetime.now(timezone.utc) - timedelta(hours=1),
+            source="study",
+        ))
+        sent = Sentence(id=1, arabic_text="x", english_translation="x",
+                        target_lemma_id=1, mappings_verified_at=datetime.now(timezone.utc))
+        db.add(sent)
+        db.flush()
+        db.add(SentenceWord(sentence_id=1, position=0, surface_form=surface, lemma_id=2))
+        db.add(SentenceWord(sentence_id=1, position=1, surface_form="كتاب", lemma_id=1))
+        db.flush()
+        db.commit()
+    def test_confused_present_verb_records_category(self, db_session):
+        self._seed_verb_sentence(db_session, "يُفْسِدُ")  # present tense of أفسد
+        submit_sentence_review(
+            db_session, sentence_id=1, primary_lemma_id=1,
+            comprehension_signal="partial", confused_lemma_ids=[2], session_id="t",
+        )
+        ulk = db_session.query(UserLemmaKnowledge).filter_by(lemma_id=2).first()
+        entry = ulk.variant_stats_json["يفسد"]
+        assert entry["confused"] == 1
+        assert entry["category"] == "verb_present"
+    def test_derived_form_records_form_key(self, db_session):
+        # plural in forms_json -> derived_form with form_key
+        _seed_word(db_session, 1, "كتاب", "book")
+        noun = Lemma(
+            lemma_id=2, lemma_ar="وَرَقَة", lemma_ar_bare="ورقة", pos="noun",
+            gloss_en="paper", forms_json={"plural": "أَوْرَاق"},
+        )
+        db_session.add(noun)
+        db_session.flush()
+        db_session.add(UserLemmaKnowledge(
+            lemma_id=2, knowledge_state="learning", fsrs_card_json=_make_card(),
+            introduced_at=datetime.now(timezone.utc) - timedelta(days=10),
+            source="study",
+        ))
+        sent = Sentence(id=1, arabic_text="x", english_translation="x",
+                        target_lemma_id=1, mappings_verified_at=datetime.now(timezone.utc))
+        db_session.add(sent)
+        db_session.flush()
+        db_session.add(SentenceWord(sentence_id=1, position=0, surface_form="أَوْرَاق", lemma_id=2))
+        db_session.add(SentenceWord(sentence_id=1, position=1, surface_form="كتاب", lemma_id=1))
+        db_session.flush()
+        db_session.commit()
+        submit_sentence_review(
+            db_session, sentence_id=1, primary_lemma_id=1,
+            comprehension_signal="partial", confused_lemma_ids=[2], session_id="t",
+        )
+        ulk = db_session.query(UserLemmaKnowledge).filter_by(lemma_id=2).first()
+        entry = ulk.variant_stats_json["أوراق"]
+        assert entry["category"] == "derived_form"
+        assert entry["form_key"] == "plural"
 class TestNoIdea:
     def test_all_words_get_rating_1(self, db_session):
         _seed_word(db_session, 1, "كتاب", "book")

L345–422

TestVariantStatsMorphology — two integration tests for the write path. These test that classify_surface_morphology is wired in correctly (not just that it returns the right value in isolation) and that variant_stats_json is persisted with the category. Previously there were no tests for the _match_surface_form logic at this integration level.

Remaining Changes

3 files not in walkthrough

These files were changed in the PR but not featured in the walkthrough above.

docs/ 3 files

api-reference.md +1 −1

docs/api-reference.md CHANGED Viewed

@@ -20,7 +20,7 @@ Full endpoint list. See `backend/app/routers/` for implementation.
 | POST | `/api/review/submit-sentence` | Submit sentence review. Schedulable content lemmas get FSRS/acquisition credit; function words and proper-name lemmas are lookup-only and ignored for scheduling/review credit. Accepts `missed_lemma_ids`, `confused_lemma_ids`, optional `confusion_candidate_lemma_ids` telemetry from the yellow-tap help panel, and optional `confusion_captures` (array of `{failed_lemma_id, capture_method: 'suggested_pick' \| 'free_text', confused_with_lemma_id?, confused_with_text?, candidates_shown}`) — explicit user-picked confusions persisted to the `confusion_captures` table for later analysis. Optional `parent_card_type` (`"passage"`/`"sentence"`/`"wrapup"`) tags the review with its parent card so analytics can split passage-internal reviews from standalone ones. |
 | POST | `/api/review/undo-sentence` | Undo a sentence review — restores pre-review FSRS state, deletes logs |
 | GET | `/api/review/word-lookup/{lemma_id}` | Word detail + root family + forms_translit (computed on-the-fly if not stored) + pattern_examples + etymology_json for review lookup |
-| GET | `/api/review/confusion-help/{lemma_id}?surface_form=...` | Confusion analysis for "did not recognize" words — morphological decomposition (clitics/forms) + form-aware visual similarity (surface/form edit distance, rasm, short-verb ranking) + phonetic similarity |
+| GET | `/api/review/confusion-help/{lemma_id}?surface_form=...` | Confusion analysis for "did not recognize" words — morphological decomposition (clitics/forms) + `morphology` `{category, form_key, explanation}` surface→lemma bridge (incl. verb-tense forms the band decomposition can't show) + form-aware visual similarity (surface/form edit distance, rasm, short-verb ranking) + phonetic similarity |
 | POST | `/api/review/sync` | Bulk sync offline reviews |
 | POST | `/api/review/reintro-result` | Submit re-introduction quiz result |
 | POST | `/api/review/experiment-intro-ack` | Acknowledge experiment intro card was shown (sets `experiment_intro_shown_at` for dedup + rescue cooldown) |

backend-services.md +1 −1

docs/backend-services.md CHANGED Viewed

@@ -60,7 +60,7 @@ All services in `backend/app/services/`.
 - `morphology.py` — CAMeL Tools analyzer. Hamza normalized at comparison time only (preserved in storage). Falls back to stub if not installed.
 - `transliteration.py` — Deterministic Arabic→ALA-LC romanization from diacritized text. Handles long vowels, shadda, hamza carriers, alif madda/wasla, sun letter assimilation, tāʾ marbūṭa, nisba ending. **Uthmani diacritics**: recognizes U+06E1 (small high dotless head of khaa / Uthmani sukun), U+06DF (small high rounded zero), U+06E2 (small high meem) so Quranic text transliterates correctly. **Long-vowel inference for partially-vocalized text** (fixed 2026-05-04): bare ya/waw following a vowelless consonant infers long ī/ū (e.g. `حَديقة` → `ḥadīqa`, `إيجار` → `ījār`), mirroring the existing bare-alif → long ā logic. Word-initial hamza-carriers (إ ا أ ٱ) handle long ī/ū the same way. **Consonant-glide disambiguation**: a ya/waw is treated as a consonant — not a long-vowel marker — when (a) it carries its own short vowel (e.g. `سِيَاسَة` → `siyāsa`, not `sīāsa`) or (b) it's immediately followed by alif/maqsura (e.g. `حَالِياً` → `ḥāliyā`, not `ḥālīā`), since Arabic phonotactics disallow two adjacent long vowels. `transliterate_lemma()` for dictionary form (strips tanwīn + case vowels). `transliterate_forms()` iterates forms_json values and produces parallel ALA-LC transliterations (skips metadata keys like "gender", "verb_form").
 - `variant_detection.py` — Three-layer variant detection: (1) CAMeL candidates with root_id validation (rejects different-root pairs), (2) Gemini Flash LLM confirmation with VariantDecision cache, (3) display fix in sentence_selector uses original lemma_id. Used by ALL import paths. Graceful fallback if LLM unavailable.
-- `confusion_service.py` — Rule-based confusion analysis for "did not recognize" (yellow) words. Four analysis types: (1) **morphological** — decomposes surface form into prefix clitics + stem + suffix clitics using PROCLITICS/ENCLITICS lists, matches stem against lemma and forms_json entries; (2) **visual/form-aware** — finds similar-looking words in user's vocabulary (including encountered and suspended leech words) by comparing the target dictionary form and exposed surface form against candidate dictionary forms and `forms_json` entries, then ranks by edit distance, rasm skeleton distance, same-root signal, short-verb priority, **adjacent transposition** (metathesis, e.g. جرح↔جحر — same letters reordered, which plain Levenshtein scores as distance 2; reason "letters swapped"), and **shared rime** (same final letters, different onset — e.g. نام/صام, حرث/ورث; reason "rhymes" — pulls the rhyme cohort above equidistant dot-variants so the user's near-miss isn't truncated; added 2026-06-01 after free-text capture analysis showed these confusions were in vocab but ranked out of the list). Rasm groups map letters differing only by dots to same skeleton (ب/ت/ث/ن → same base). The response includes `match_reason`, `matched_form`, and matched form key for diagnostics; (3) **phonetic** — finds words that sound similar to learners but look different via `PHONETIC_MAP` (emphatic→plain: ص→س, ض→د, ط→ت, ظ→ذ; pharyngeal: ح→ه, ع→ا; interdental: ث→س, ذ→ز; uvular: غ→خ). Catches confusions like سبع↔صباح. Only surfaces words NOT already in visual results; (4) **prefix disambiguation** — when a word starts with و/ف/ب/ل/ك, hints whether it's a proclitic prefix or part of the root (uses `lemma.root` relationship). All rule-based, no LLM. Endpoint: `GET /api/review/confusion-help/{lemma_id}?surface_form=...`.
+- `confusion_service.py` — Rule-based confusion analysis for "did not recognize" (yellow) words. Four analysis types: (1) **morphological** — decomposes surface form into prefix clitics + stem + suffix clitics using PROCLITICS/ENCLITICS lists, matches stem against lemma and forms_json entries; (2) **visual/form-aware** — finds similar-looking words in user's vocabulary (including encountered and suspended leech words) by comparing the target dictionary form and exposed surface form against candidate dictionary forms and `forms_json` entries, then ranks by edit distance, rasm skeleton distance, same-root signal, short-verb priority, **adjacent transposition** (metathesis, e.g. جرح↔جحر — same letters reordered, which plain Levenshtein scores as distance 2; reason "letters swapped"), and **shared rime** (same final letters, different onset — e.g. نام/صام, حرث/ورث; reason "rhymes" — pulls the rhyme cohort above equidistant dot-variants so the user's near-miss isn't truncated; added 2026-06-01 after free-text capture analysis showed these confusions were in vocab but ranked out of the list). Rasm groups map letters differing only by dots to same skeleton (ب/ت/ث/ن → same base). The response includes `match_reason`, `matched_form`, and matched form key for diagnostics; (3) **phonetic** — finds words that sound similar to learners but look different via `PHONETIC_MAP` (emphatic→plain: ص→س, ض→د, ط→ت, ظ→ذ; pharyngeal: ح→ه, ع→ا; interdental: ث→س, ذ→ز; uvular: غ→خ). Catches confusions like سبع↔صباح. Only surfaces words NOT already in visual results; (4) **prefix disambiguation** — when a word starts with و/ف/ب/ل/ك, hints whether it's a proclitic prefix or part of the root (uses `lemma.root` relationship). All rule-based, no LLM. Endpoint: `GET /api/review/confusion-help/{lemma_id}?surface_form=...`. **`classify_surface_morphology(surface_bare, lemma)`** (2026-06-03) is the shared classifier behind the morphology bridge: returns `{category, form_key, explanation}` (None for the dictionary form or a bare definite article). `category` ∈ verb_present/verb_other/derived_form/proclitic/enclitic/inflection. `explanation` is a one-line surface→lemma bridge ("present-tense form of «to spoil»") populated only for the verb-tense cases `decompose_surface` can't render as color bands — closing the ~55% of inflected confusions (esp. conjugations absent from `forms_json`) the bands missed. `analyze_confusion` returns it under `morphology`, the `submit-sentence` write path stores `category`/`form_key` on `variant_stats_json`, and `WordInfoCard` renders the `explanation` line on a yellow mark.
 - `grammar_service.py` — 49 features, 8 tiers. Comfort score: 60% log-exposure + 40% accuracy, decayed by recency.
 - `grammar_tagger.py` — LLM-based grammar feature tagging.
 - `grammar_lesson_service.py` — LLM-generated grammar lessons, cached in DB.

data-model.md +1 −1

docs/data-model.md CHANGED Viewed

@@ -7,7 +7,7 @@ SQLAlchemy models in `backend/app/models.py`. Pydantic schemas in `backend/app/s
 - `pattern_info` — Morphological pattern metadata: wazn (PK, e.g. "fa'il"), wazn_meaning, enrichment_json (LLM-generated: explanation, how_to_recognize, semantic_fields, example_derivations, register_notes, fun_facts, related_patterns)
 - `lemmas` — Dictionary forms: root FK, pos, gloss, frequency_rank, cefr_level, grammar_features_json, forms_json, example_ar/en, transliteration, audio_url, canonical_lemma_id (variant FK), source_story_id, word_category (NULL=standard, proper_name, onomatopoeia), thematic_domain, etymology_json, memory_hooks_json, wazn (morphological pattern e.g. "fa'il", "maf'ul", "form_2", indexed), wazn_meaning (human-readable pattern description), forms_translit_json (ALA-LC transliteration per forms_json key, e.g. {"present": "yaktub", "plural": "kutub"}), gates_completed_at (timestamp set by `run_quality_gates()` — NULL means ungated, session builder rejects), decomposition_note (nullable JSON audit metadata from lemma-decomposition audit: `{mle_misanalysis: bool, reason, source_artifact, tagged_at, phase}` — stamped by Step 4b+ on orphan compounds whose CAMeL MLE decomposition proved wrong; query: `json_extract(decomposition_note, '$.mle_misanalysis') = 1`)
 - `frequency_core_entries` — Weighted high-frequency curriculum ranks. `core_rank` is a continuous teachable-content rank; `lemma_id` links to an Alif lemma when mapped and stays NULL for honest missing-from-DB gaps. Stores source evidence (`camel_rank/count`, `buckwalter_rank`, `artenten_rank`, `kelly_rank/cefr`, `hindawi_rank`, `news_rank`, `islamic_rank`, `broad_source_count`, `confidence_tier`, `gap_status`, `source_flags_json`) plus display/gloss fields for stats.
-- `user_lemma_knowledge` — Per-lemma SRS state: knowledge_state (encountered/acquiring/new/learning/known/lapsed/suspended), fsrs_card_json, times_seen, times_correct, times_heard (passive listening count, incremented by mark-story-heard), total_encounters, source (study/duolingo/textbook_scan/book/story_import/frequency_core/auto_intro/collateral/leech_reintro — preserved through acquisition, not overwritten), variant_stats_json (diagnostic per-surface seen/missed/confused counts; may include `form_key`/`form_label` when the surface matches `forms_json`; never an independent scheduling unit), acquisition_box (1/2/3), acquisition_next_due, entered_acquiring_at (when word entered Leitner pipeline), graduated_at, leech_suspended_at, leech_count, experiment_group (nullable, `intro_ab_card` for standard card-first acquisition; legacy `textbook_preserve_intro` rows may exist but no longer generate cards), experiment_intro_shown_at (nullable, timestamp when intro card was shown — prevents re-showing)
+- `user_lemma_knowledge` — Per-lemma SRS state: knowledge_state (encountered/acquiring/new/learning/known/lapsed/suspended), fsrs_card_json, times_seen, times_correct, times_heard (passive listening count, incremented by mark-story-heard), total_encounters, source (study/duolingo/textbook_scan/book/story_import/frequency_core/auto_intro/collateral/leech_reintro — preserved through acquisition, not overwritten), variant_stats_json (diagnostic per-surface seen/missed/confused counts; each entry also stores a `category` — verb_present/verb_other/derived_form/proclitic/enclitic/inflection, from `confusion_service.classify_surface_morphology` — plus `form_key`/`form_label` when the surface matches a `forms_json` form; lets per-form confusion be queried instead of re-decomposed; never an independent scheduling unit), acquisition_box (1/2/3), acquisition_next_due, entered_acquiring_at (when word entered Leitner pipeline), graduated_at, leech_suspended_at, leech_count, experiment_group (nullable, `intro_ab_card` for standard card-first acquisition; legacy `textbook_preserve_intro` rows may exist but no longer generate cards), experiment_intro_shown_at (nullable, timestamp when intro card was shown — prevents re-showing)
 ## Sentences & Reviews
 - `sentences` — Generated/imported: arabic_text (fully diacritized — all pipelines store the voweled form; callers needing plain text strip diacritics at query time), english_translation, transliteration, target_lemma_id, story_id (FK to stories, for book-extracted sentences), source (llm/book/corpus/michel_thomas/tatoeba/manual), times_shown, last_reading_shown_at/last_listening_shown_at, last_reading_comprehension/last_listening_comprehension, is_active, max_word_count, created_at, page_number (for book sentences), mappings_verified_at (nullable DateTime — NULL=never verified, timestamp=when last verified by batch LLM check)

Appendix

File Map

backend/app/services/confusion_service.py — Core change: adds classify_surface_morphology() (the new shared classifier), integrates it into analyze_confusion() to populate a new 'morphology' field, and broadens has_morph to include verb-tense explanations.

backend/app/services/sentence_review_service.py — Deletes the narrow _match_surface_form() private helper and replaces its call site with classify_surface_morphology(), storing richer category data on variant_stats_json.

backend/app/schemas.py — Adds SurfaceMorphology Pydantic model and a morphology field to ConfusionAnalysisOut, making the new field part of the validated API contract.

backend/app/routers/review.py — Adds morph_category to the confusion_help interaction log, enabling future analysis of which morphological categories drive yellow-mark help opens.

frontend/lib/types.ts — Adds SurfaceMorphology TypeScript interface with a union-typed category field, and adds morphology: SurfaceMorphology | null to ConfusionAnalysis.

frontend/lib/review/WordInfoCard.tsx — Extracts morphExplanation from confusionData and renders a new morphBridge UI widget (pill with git-compare icon + explanation text) for verb-tense forms the color bands can't show.

backend/tests/test_confusion_service.py — Adds TestClassifySurfaceMorphology with 11 unit tests covering the full decision tree of the new classifier, including the Form-IV past-not-present guard and None-safety.

backend/tests/test_sentence_review.py — Adds TestVariantStatsMorphology with 2 integration tests verifying that submit_sentence_review stores category and form_key on variant_stats_json via the new classifier.

docs/api-reference.md — Updates confusion-help endpoint description to document the new morphology {category, form_key, explanation} field in the response.

docs/backend-services.md — Extends the confusion_service.py entry to document classify_surface_morphology(), its categories, and where it is consumed.

docs/data-model.md — Updates the user_lemma_knowledge variant_stats_json description to reflect that each entry now stores a category field from classify_surface_morphology.