BenchCouncil Transactions on Benchmarks, Standards and

Evaluations, 2026

DOI: https://doi.org/10.66834/9a7y0c13

Research Article

RESEARCH ARTICLE

ABWS: The Arabic Boundary-aware Word

Segmentation Benchmark for Reproducible Evaluation

Huda AlShuhayeb

1,∗

and Behrouz Minaei-Bidgoli

1,∗

School of Computer Engineering, Iran University of Science and Technology (IUST), Tehran, Iran

∗

Corresponding author: hudaalshuhayeb@gmail.com; b_minae@iust.ac.ir

Received on 27 January 2026; Accepted on 7 April 2026

Abstract

With the rapid adoption of natural language processing (NLP) systems for morphologically rich languages, it has

b ecome increasingly imperative to standardize a common set of measures and evaluation practices to ensure reproducibil-

ity and fair comparison. Arabic word segmentation serves as a foundational layer in the NLP software stack; however,

the ﬁeld remains fragmented due to inconsistent datasets and an overreliance on opaque, aggregate metrics that mask

systemic architectural biases.

We present ABWS (Arabic Boundary-aware Word Segmentation), a scalable and publicly available benchmarking

system designed for the rigorous, reproducible evaluation of diverse segmentation paradigms. To enable paradigm-agnostic

comparison across rule-based, statistical, and neural models, ABWS introduces a canonical boundary vector abstraction

that normalizes disparate system outputs into a uniﬁed evaluation interface. The benchmarking harness includes a manu-

ally veriﬁed gold-standard workload of 212,873 words across diverse genres and integrates seven widely used segmentation

systems as reproducible baselines.

Our systematic evaluation reveals that while neural subword-based models are robust for vocabulary compression,

they exhibit extreme Over-Segmentation Ratios (OSR > 0.58), leading to a signiﬁcant drop in word-level exact match ac-

curacy compared to rule-based engines. We further introduce Critical Boundary Accuracy (CBA), a linguistically weighted

metric that prioritizes high-impact morphological boundaries. Our cross-layer analysis demonstrates that CBA is highly

predictive of downstream performance in Machine Translation and Named Entity Recognition (ρ > 0.88 ), whereas tradi-

tional token-level F

scores often obscure these performance bottlenecks.

By providing a containerized evaluation pipeline and versioned system artifacts, ABWS establishes a new standard

for methodological rigor in Arabic NLP research, oﬀering a template for benchmarking other morphologically complex

languages within the broader computational ecosystem.

Key words: Arabic NLP, Morphological Segmentation, Benchmarking, Reproducibility, Boundary Errors, Error

Taxonomy, Benchmark Traceability, Evaluation Conditions

1. Introduction

With the rapid proliferation and deployment of natural language

processing (NLP) systems across global industries, it has become in-

creasingly imperative to standardize a common set of measures and

evaluation practices to ensure reproducibility and fair comparison.

For morphologically rich languages (MRLs) such as Arabic, word

segmentation serves as a foundational preprocessing layer in the

NLP software stack. Despite its critical role, the ﬁeld remains frag-

mented, lacking a uniﬁed benchmarking infrastructure capable of

systematically evaluating the diverse array of rule-based, statistical,

and neural segmentation paradigms.

To illustrate the unique complexity of Arabic word segmenta-

tion compared to languages like English, consider the single Arabic

word token ’fabi-iltizmi-him’. In English, this is expressed as a multi-

word phrase: ’and by their commitment’. While English maintains

clear whitespace boundaries between the conjunction (’and’), prepo-

sition (’by’), noun (’commitment’), and possessive pronoun (’their’),

Arabic merges these distinct functional morphemes into a single or-

thographic unit. This ’clitic stacking’ creates a signiﬁcant challenge

for NLP systems, as a single segmentation errorsuch as failing to

isolate the proclitic ’fa-’ (and) or the preposition ’bi-’ (by)can lead

to a complete misinterpretation of the word’s syntactic role. Unlike

English, where tokenization is largely a trivial whitespace-splitting

task, Arabic segmentation requires a sophisticated boundary-aware

analysis to recover these latent grammatical structures, making it a

critical pre-processing bottleneck.

Arabic, spoken by over 400 million people, presents unique chal-

lenges for system evaluation due to its complex morphology, where

a single space-delimited string can represent multiple concatenated

morphemes (roots, patterns, and aﬃxes) [1]. The performance of a

AlShuhayeb et al.

segmentation system directly dictates the eﬃciency and accuracy of

downstream tasks, including machine translation [2] and information

retrieval [3]. However, the absence of a standardized benchmarking

harness prevents researchers from understanding how diﬀerent archi-

tectural choices. such as subword-based methods versus traditional

statistical modelsb ehave across varied data modalities and genres.

Current evaluation practices in Arabic NLP suﬀer from three

critical methodological gaps that hinder the development of high-

performance standards:

1. Lack of a Standardized Benchmark Suite: Many evalu-

ations rely on non-public or inconsistently annotated datasets,

making it impossible to replicate results or perform “apples-

to-apples” comparisons between emerging neural models and

established baselines [4].

2. Metric Opacity and Coarse Granularity: Most systems

report aggregate token-level F

scores. These “black-box” met-

rics mask qualitative diﬀerences in boundary placement errors,

such as the over-segmentation of stems versus the under-

segmentation of clitic clusters, which have vastly diﬀerent

impacts on system usability [5].

3. Isolation from Downstream Impact: There is a lack of

empirical evidence linking speciﬁc segmentation error types to

performance degradation in full-stack NLP pipelines. This lim-

its the ability of systems engineers to perform task-aware model

selection.

To address these challenges, we introduce ABWS (Arabic

Boundary-aware Word Segmentation), a scalable and publicly avail-

able benchmarking system designed for the rigorous and repro-

ducible evaluation of Arabic segmentation. Similar to benchmarking

eﬀorts in other computational domains (e.g., MLCommons), ABWS

provides a standardized framework that decouples the evaluation

logic from the underlying model implementation.

The primary contributions of this work are as follows:

• A Standardized Gold-Standard Dataset: We present a

manually veriﬁed dataset comprising 212,873 words across di-

verse genres, providing a representative workload for evaluating

system robustness and generality.

• A Uniﬁed Benchmarking Harness: We establish repro-

ducible baselines by integrating seven widely used segmentation

systemsspanning rule-based, statistical, and neural paradigm-

sunder a common evaluation protocol.

• Boundary-aware Metrics and Taxonomy: We extend tra-

ditional evaluation practices by introducing a ﬁne-grained error

taxonomy that quantiﬁes boundary placement decisions, oﬀering

deeper insights into system-level bottlenecks.

• Cross-Layer Impact Analysis: We provide a systematic

study of how segmentation errors propagate through down-

stream NLP tasks, enabling a more holistic assessment of

performance beyond simple accuracy scores.

By providing the dataset, standardized evaluation scripts, and

baseline system outputs, ABWS aims to establish a new standard

for metho dological rigor in Arabic NLP. This framework not only fa-

cilitates transparent performance tracking but also serves as a model

for benchmarking other morphologically complex languages within

the broader NLP ecosystem.

The remainder of this paper is organized as follows: Section 2

reviews existing segmentation and evaluation practices; Section 3

details the design and composition of the ABWS benchmark; Sec-

tion 4 presents the boundary-aware evaluation framework; Section 5

reports experimental results and systematic error analysis; Section 6

examines implications for downstream task performance; and Sec-

tion 7 concludes with future directions for standardization in the

ﬁeld.

2. Related Work

This section reviews prior work from a benchmark- engineering

perspective, with particular attention to three dimensions: (i) the

evolution of Arabic morphological segmentation systems, (ii) exist-

ing evaluation methodologies and benchmarks for segmentation, and

(iii) recent advances in benchmarking theory that emphasize the

explicit speciﬁcation of evaluation conditions, evaluation systems,

and standards as prerequisites for comparability and reproducibility

[6–8].

2.1. Arabic Morphological Segmentation Systems

Arabic morphological segmentation has evolved through several

methodological paradigms. Early systems were predominantly rule-

based and lexicon-driven, aiming to produce linguistically well-

formed analyses grounded in classical morphological theory. Systems

such as MADA and AlKhalil Morpho Sys exemplify this generation,

integrating rich lexical resources with hand-crafted rules and con-

textual disambiguation [9–11]. While these systems achieved high

linguistic precision, they were often constrained by limited cover-

age, sensitivity to orthographic variation, and reduced robustness

to out-of-vocabulary forms and non-canonical usage [12].

To address coverage and scalability, statistical segmentation ap-

proaches emerged. Data-driven models based on conditional random

ﬁelds and discriminative classiﬁers learned boundary decisions from

annotated corpora, notably the Penn Arabic Treebank. Farasa fur-

ther emphasized eﬃciency and deployability by introducing a fast,

deterministic segmentation pipeline with statistical ranking, en-

abling near real-time pro cessing on large corpora [13]. These systems

improved robustness but often traded linguistic interpretability for

speed and generalization.

In contemporary NLP pipelines, segmentation is frequently in-

duced implicitly through subword tokenization. Methods such as

Byte-Pair Encoding (BPE) and SentencePiece, as well as WordPiece

tokenization used in transformer pretraining, generate boundaries

optimized for vocabulary compression and language modeling ob-

jectives rather than morphological validity [14–16]. Arabic-focused

pretrained models, including AraBERT and later AraELECTRA

and MARBERT, inherit this tokenization-centric notion of segmen-

tation, which often results in boundaries that cut across morphemes

or clitic units [17, 18]. Although recent work explores explicit neu-

ral segmentation via boundary prediction or multitask learning with

orthographic processes, such approaches remain fragmented across

datasets and annotation conventions and are not yet standardized

[19].

Despite this methodological diversity, there is no consensus on

an “optimal” segmentation strategy. In practice, system selection is

frequently driven by pragmatic constraints such as speed, memory

footprint, or compatibility with downstream models rather than by

linguistic or task-aware criteria.

2.2. Evaluation of Arabic Segmentation

Early evaluations of Arabic segmentation typically relied on align-

ment with treebank-style gold annotations and reported boundary-

level precision, recall, and F

. However, treating all boundaries

as equally important obscures qualitatively diﬀerent error types,

such as under-segmentation of proclitics versus over-segmentation

of stems [? ]. Task-oriented studies demonstrated that segmentation

errors have asymmetric downstream impact: over-segmentation may

harm precision in information retrieval, while under-segmentation

may reduce recall or impair translation quality [20, 21].

More recent analyses highlight that tokenization and segmenta-

tion choices also aﬀect the eﬃciency and behavior of transformer-

based models, inﬂuencing both performance and computational cost

[22]. Nevertheless, most comparative studies still report aggregate

metrics computed under heterogeneous and often undocumented

evaluation conditions, limiting interpretability and reproducibility.

From a standards perspective, a central limitation of prior work

is the absence of a standardized protocol for comparing fundamen-

tally diﬀerent segmentation paradigms. Morphological segmenters

produce linguistically motivated morpheme boundaries, whereas

subword tokenizers generate boundaries derived from statistical vo-

cabulary construction. Without an explicit mapping between these

representations, evaluation scores across paradigms become eﬀec-

tively incomparable, even when computed on the same dataset [6].

Reproducibility is further hindered when code, data splits, normal-

ization policies, and evaluation scripts are not fully speciﬁed or

publicly available [4].

2.3. Benchmarking Practices, Standards, and

Robustness

General-purpose NLP benchmarks such as GLUE and SuperGLUE

demonstrated the value of uniﬁed tasks, datasets, and scoring

protocols for accelerating progress through comparability [23, 24].

Subsequent benchmarking research has clariﬁed, however, that a

benchmark should not be understood as a dataset alone, but as a

complete evaluation system whose conclusions depend on explicitly

deﬁned evaluation conditions (EC), a concrete evaluation system

(ES), and a value function that encodes what is being optimized

[6, 7].

Within this perspective, a dataset is only meaningful insofar as

it instantiates a representative workload. That is, benchmark data

should approximate the structural, distributional, and operational

characteristics of real-world inputs that systems are expected to

process. ABWS adopts this workload-centric view explicitly: the

curated corpus is not treated as a passive collection of labeled ex-

amples, but as a controlled workload designed to stress-test Arabic

segmentation systems under realistic linguistic conditions, including

dense clitic stacking, derivational morphology, orthographic vari-

ation, and genre-speciﬁc constructions common in formal Arabic

text.

Recent benchmark frameworks emphasize workload characteri-

zation as a prerequisite for valid measurement. For example, AICB

formalizes benchmarks around representative workloads executed

under reproducible environments and explicitly deﬁned ECs, en-

suring that performance claims reﬂect behavior under realistic

operating conditions rather than isolated test sets [7]. Similarly,

COADBench argues that benchmarks must align evaluation met-

rics with practical outcomes, demonstrating that mischaracterized

workloads can render even precise metrics misleading [8].

In the context of Arabic segmentation, workload characterization

is particularly critical. Segmentation diﬃculty varies substantially

across registers and genres, and small shifts in text composition can

induce large changes in boundary distributions and error modes.

ABWS therefore ﬁxes and documents workload propertiesinclud-

ing genre, morphological density, normalization rules, and boundary

conventionsso that reported results correspond to a clearly speciﬁed

and reproducible segmentation workload, rather than an abstract

notion of Arabic data.

Two robustness issues follow directly from this workload-centric

framing. First, domain shiftfor example b etween Classical Arabic,

Modern Standard Arabic, and informal or social media textcan sub-

stantially alter error distributions and system rankings unless ECs

such as genre selection, orthographic normalization, and boundary

deﬁnitions are ﬁxed and reported. Second, data contamina-

tion risks arise when benchmark material overlaps with resources

used during system development or pretraining, particularly for

large pretrained models, leading to inﬂated and non-generalizable

performance estimates.

These considerations motivate benchmark designs that treat

workload speciﬁcation, dataset provenance, splitting strategy, nor-

malization procedures, and evaluation scripts as ﬁrst-class artifacts.

By doing so, ABWS aligns with determinacy and equivalence as core

benchmarking standards [6], and ensures that its results reﬂect sys-

tem behavior on a well-deﬁned, representative Arabic segmentation

workload rather than incidental properties of a static dataset.

2.4. Our Position

ABWS is designed as a standards-oriented benchmark for Ara-

bic word segmentation. It explicitly speciﬁes evaluation condi-

tions, provides a reproducible evaluation system, and deﬁnes

value functions that (i) distinguish boundary types and error po-

sitions, (ii) enable comparison across rule-based, statistical, and

neural/subword paradigms via boundary harmonization, and (iii)

support downstream-aware analysis where appropriate. In doing so,

ABWS aims to move Arabic segmentation evaluation from dataset-

speciﬁc reporting toward a rigorous, comparable, and repro ducible

benchmark engineering practice [6, 7].

3. Formal Speciﬁcation and Evaluation

Conditions

This section describes the architectural design of ABWS (Arabic

Boundary-aware Word Segmentation), a benchmarking framework

engineered to address fundamental limitations in existing Arabic

segmentation evaluation practices. Empirical inspection of segmen-

tation outputs across rule-based, statistical, and neural systems

reveals that segmentation errors are not random, but systematic

and paradigm-dependent

. Subword-based models fragment stems

to minimize vocabulary entropy, neural tokenizers exhibit unstable

boundary placement, and statistical systems bias toward conserva-

tive under-segmentation in clitic-dense constructions. These failure

modes cannot be reliably captured by aggregate word-level metrics

alone.

ABWS is therefore designed not as a static dataset, but as a

uniﬁed benchmarking harness that enables repro ducible, paradigm-

agnostic, and diagnostically meaningful evaluation. Following

benchmarking principles established for large-scale computational

systems [6, 7], ABWS formalizes evaluation around standardized ex-

ecution conditions, a canonical boundary representation layer, and

a multi-dimensional metric suite explicitly aligned with observed

linguistic error behavior.

3.1. Design Principles and Standardization Goals

The design of ABWS is guided by four core principles, each directly

motivated by empirical segmentation pathologies observed across

contemporary systems.

• Boundary-Centric Granularity: Empirical analysis demon-

strates that neural and subword-based systems frequently in-

sert boundaries within morphologically atomic stems (e.g.,

istiqqan → ist + q + q + an), while other systems omit re-

quired clitic boundaries (e.g., fa + li + namad → falinahmad).

AlShuhayeb et al.

ABWS therefore formulates segmentation as a sequence of bi-

nary boundary decisions at the character level, enabling direct

diagnosis of over- and under-segmentation behavior.

• Paradigm-Agnostic Normalization: Arabic segmentation

systems produce structurally incompatible outputs, ranging

from morpho-syntactic analyses to frequency-driven subword de-

compositions. To enable fair comparison, ABWS introduces a

boundary vector abstraction that projects all outputsregardless

of underlying architectureinto a common mathematical space.

• Repro ducibility-First Engineering: To eliminate hidden

variability, all datasets, normalization rules, evaluation scripts,

and system outputs are version-controlled and containerized.

This benchmark-as-code approach ensures that rep orted results

are deterministic, auditable, and independently veriﬁable.

• Error-Aware Metric Design: Observed segmentation failures

disproportionately aﬀect certain boundary types (e.g., clitics ver-

sus stem-internal splits). ABWS metrics are therefore designed

to distinguish directional error biases and to weight linguistically

salient boundaries according to their downstream impact.

3.2. Standardized Boundary Representation Layer

A central challenge in Arabic segmentation benchmarking is output

incompatibility. For example, rule-based analyzers correctly pre-

serve clitic boundaries (li + al + wu), while subword tokenizers

may split stems (al- + h + ra) or collapse multi-clitic constructions

(wa-lil-junub). Direct comparison of such outputs is ill-deﬁned.

ABWS resolves this incompatibility by projecting all system out-

puts into a Character-Level Boundary Vector, which serves as the

canonical internal representation for evaluation.

Boundary Vector Formalization. Given an input string of n

characters, ABWS deﬁnes a binary boundary vector

B = (b

, b

, . . . , b

n−1

where







1 if a boundary exists between characters i and i + 1,

0 otherwise.

This representation ensures that all systems are evaluated

against an identical character sequence, eliminating alignment drift

caused by orthographic normalization, Unicode variation, or to-

kenization artifacts. As a result, stem-internal splits, clitic omis-

sions, and boundary displacements are measured uniformly across

paradigms.

3.3. Evaluation Engine and Value Functions

Let S and G denote the system-predicted and gold-standard bound-

ary vectors, respectively. The ABWS evaluation engine computes

a suite of value functions designed to capture complementary

dimensions of segmentation quality revealed by empirical error

analysis:

• Boundary-Level Precision, Recall, and F

: Baseline mea-

sures of boundary detection accuracy, insensitive to token length

but sensitive to boundary placement.

• Word-Level Exact Match (EM): A strict correctness crite-

rion requiring all boundary decisions within a word to match

the gold standard, penalizing even a single stem-internal split or

missed clitic.

• Boundary Distance (BD): A granular disagreement metric

quantifying average per-boundary deviation:

BD(S, G) =

n − 1

n−1

∑

i=1

(S) − b

(G)| .

This measure captures systemic boundary noise observed in

subword tokenizers.

• Directional Bias Ratios: Over-Segmentation Ratio (OSR)

and Under-Segmentation Ratio (USR) explicitly separate stem-

fragmentation errors from clitic-merging errors, reﬂecting the

asymmetric failure modes observed across architectures.

• Critical Boundary Accuracy (CBA): A weighted accuracy

metric prioritizing linguistically salient boundaries (e.g., procl-

itics and enclitics) over stem-internal positions. Fixed weights

clitic

= 2.0, w

stem

= 0.5) ensure determinism while

reﬂecting downstream sensitivity.

• CBA Formulation: The diﬀerential weighting in the Criti-

cal Boundary Accuracy (CBA) metricassigning w = 2.0

to clitic boundaries and w = 0.5 to internal stem bound-

ariesis grounded in the concept of Downstream Impact Analysis

of segmentation errors. In Arabic, clitics (proclitics and encli-

tics) frequently function as essential syntactic markers, including

conjunctions, prepositions, and pronominal suﬃxes. Failure to

correctly segment a clitic (for example, the preposition bi-) often

produces a catastrophic error in downstream tasks such as Ma-

chine Translation or Dependency Parsing, because it alters the

fundamental grammatical role of the token within the sentence.

Conversely, over-segmentation or under-segmentation within the

stem (for example, incorrectly splitting a root-derived noun)

usually produces a recoverable error, where the semantic core

remains partially identiﬁable by information retrieval systems

or embedding-based models. By assigning a higher penalty to

clitic-related segmentation errors, the CBA metric explicitly pri-

oritizes boundaries that preserve functional linguistic structure.

This weighting scheme ensures that the benchmark emphasizes

architectural precision necessary for syntactic and grammati-

cal integrity rather than treating all boundary errors as equally

consequential lexical variations.

3.4. Statistical Protocol and Robustness

To ensure that reported diﬀerences reﬂect systematic behavior

rather than sampling variance, ABWS adopts a rigorous statistical

protocol:

• Conﬁdence Estimation: 95% conﬁdence intervals estimated

via 1,000-resample bootstrap procedures.

• Pairwise Signiﬁcance Testing: McNemars test with Bonfer-

roni correction for multiple comparisons.

• Eﬀect Size Reporting: Cohens h is reported alongside

p-values to distinguish statistical signiﬁcance from practical

impact.

3.5. Implementation and Portability

ABWS is implemented in Python as a modular evaluation li-

brary. To guarantee portability and long-term reproducibility, the

entire benchmarking pipeline is containerized with pinned dependen-

cies and ﬁxed normalization rules. New segmentation systems can

be integrated by supplying raw outputs, which are automatically

normalized and projected into boundary vectors, enabling imme-

diate inclusion in the benchmarking harness without architectural

modiﬁcation.

This design positions ABWS as a stable, extensible, and diagnos-

tically expressive benchmark capable of evolving alongside Arabic

NLP systems while preserving comparability across generations of

models.

While the current evaluation focuses on a workload characterized

by high morphological densityspeciﬁcally Classical Arabic texts such

as Shari al-Islmthe ABWS framework is architecturally designed to

be extensible to Arabic dialects. The core strength of the benchmark

lies in its Canonical Boundary Vector (CBV) abstraction, which

decouples linguistic sp eciﬁcities from the technical evaluation har-

ness. In dialectal Arabic, where segmentation challenges often arise

from phonological fusion or elision, the CBV maintains its utility by

treating segmentation as a series of vocabulary-independent binary

decisions at the character level. Consequently, adapting ABWS to

various dialects only requires redeﬁning the ’Gold Vector’ to align

with the speciﬁc morphological conventions of a given dialect (e.g.,

handling the aspectual preﬁx ’bi-’ in Levantine or negation parti-

cles in Maghrebi). This ﬂexibility ensures that ABWS remains a

paradigm-agnostic system capable of evaluating model performance

across the full spectrum of the Arabic linguistic continuum without

necessitating changes to its underlying mathematical or procedural

framework.

4. Experimental Results and Performance

Analysis

The objective of this evaluation is to provide a diagnostic breakdown

of Arabic word segmentation quality beyond aggregate accuracy

scores. All reported results are computed using the canonical

boundary vector representation deﬁned by ABWS, ensuring strictly

comparable (apples-to-apples) evaluation across heterogeneous seg-

mentation paradigms, including rule-based, statistical, and neural

systems. In addition to quantitative metrics, we incorporate linguis-

tically grounded error inspection to validate that ABWS diagnostics

capture real and systematic segmentation pathologies.

4.1. Comparative Analysis of Word-Level Accuracy

Table 1 reports Word-Level Exact Match (EM) accuracy, the most

stringent metric in the ABWS evaluation suite. EM requires a sys-

tem to reproduce the complete gold morphological segmentation of

each word without any boundary insertion, deletion, or displacement

errors.

Table 1. Word-level exact match accuracy across paradigms

(N = 212,873).

Paradigm System Accuracy

Rule-based CAMeL Tools 0.817

Rule-based ALP 0.790

Statistical Farasa 0.810

Neural / Subword BERT-based 0.460

Neural / Subword SelfSeg 0.163

Neural / Subword mBART 0.122

Neural / Subword BPE 0.102

The results reveal a pronounced performance hierarchy. Rule-

based systems achieve the highest word-level reliability, followed by

statistical models, while neural and subword-based tokenizers ex-

hibit a substantial degradation in exact match accuracy. Crucially,

this degradation is explained by structural mismatches between to-

kenization objectives and Arabic morphology: subword tokenizers

optimized for vocabulary compression frequently fragment morpho-

logically atomic stems (e.g., altahra → al + h + ra in mBART;

istiba → ist + b + a in mBART), while language-agnostic neural

systems may collapse required clitic boundaries (e.g., waad + n +

hu → waadnhu in SelfSeg). Such errors are catastrophic under EM

because even a single stem-internal split or missed clitic boundary

invalidates the entire word segmentation.

4.2. Multi-Dimensional Diagnostic Metrics

To identify the structural sources of segmentation failure, we analyze

boundary-level diagnostics using ABWS metrics in Table 2. Errors

are decomposed into Boundary F

, Boundary Distance (BD), Over-

Segmentation Ratio (OSR), and Under-Segmentation Ratio (USR),

enabling ﬁne-grained characterization of systematic error behavior.

Table 2. Boundary-level diagnostic proﬁles and error distribution.

System Boundary F

BD OSR USR

CAMeL Tools 0.86 0.11 0.08 0.14

Farasa 0.78 0.19 0.15 0.23

BERT-based 0.71 0.27 0.21 0.32

SelfSeg 0.38 0.61 0.55 0.09

BPE 0.32 0.65 0.58 0.07

mBART 0.29 0.68 0.62 0.09

Table 3. Standardized Evaluation Conditions (EC) for ABWS

Benchmark

Parameter Speciﬁcation / Rule

Orthographic Normalization Alef normalization , Ya normaliza-

tion (y, alif maqsura → uniﬁed

form)

Kashida Removal All tatweel characters (U+0640)

stripped before processing

Diacritics (Tashkeel) All short vowels and shadda re-

moved for consistency

Input Format UTF-8 encoded raw text strings

(sentence-level)

Punctuation Handling Preserved in text but excluded

from boundary vector calculation

Tool Versions Farasa (v1.1), Stanza (v1.4),

MADAMIRA (v2.1), CAMeL

Tools (v1.2)

Hardware Environment Ubuntu 22.04 LTS, 32GB RAM,

NVIDIA RTX 3090 (for neural

models)

To ensure a fair and reproducible comparison, all segmentation

systems were evaluated under a uniﬁed set of Evaluation Conditions

(EC) as detailed in Table 3. Since diﬀerent Arabic NLP tools often

employ internal normalization logic, we enforced a pre-processing

layer that standardizes Alef/Ya characters and removes non-lexical

elements like Kashida and Diacritics. This prevents performance

discrepancies from arising due to orthographic variations rather than

the segmentation logic itself. Furthermore, we provide the exact

versions of each integrated tool to ensure that our results can be

replicated in future studies.

4.3. Proﬁling Systematic Failure Modes

The diagnostic metrics reveal strongly asymmetric error proﬁles

across segmentation paradigms, consistent with direct linguistic

inspection:

AlShuhayeb et al.

• Subword Tokenizers (BPE, mBART): These systems ex-

hibit extreme over-segmentation behavior (OSR > 0.58), fre-

quently inserting boundaries within stems and even within root

material. In the provided examples, mBART splits morpholog-

ically atomic forms such as istiba into ist + b + a, and frag-

ments deﬁnite-article constructions such as al-ahra into al- + h

+ ra. Such boundaries are not linguistically valid morphemes,

but artifacts of vocabulary compression objectives.

• Neural Tokenizers (SelfSeg, BERT-based): These sys-

tems demonstrate unstable boundary behavior. SelfSeg exhibits

a mixed proﬁle dominated by boundary omissions on required

clitic chains (e.g., waad + n + hu → waadnhu, fa + lan + namad

left unsegmented), while also occasionally introducing non-

morphological preﬁx splits (e.g., a + l-ahra). BERT-based

outputs are comparatively stronger than subword tokenizers but

still exhibit boundary drift, including occasional stem-internal

splits and inconsistent handling of aﬃxes (e.g., al-ahr + a

instead of al + ahra).

• Statistical Systems (Farasa): Farasa exhibits a conservative

boundary-decision strategy with elevated USR, particularly in

multi-clitic sequences and function-word attachment. This is vis-

ible in cases where clitic boundaries are merged (e.g., wa + kull

+ hu predicted as wakull + hu) and in reduced granularity for

proclitic chains.

• Rule-based Systems (CAMeL To ols, ALP):

Rule-based

analyzers maintain the most balanced error distribution and

low BD, indicating that residual errors are localized rather

than systemic. They consistently preserve canonical clitic and

article boundaries (e.g., li + al + wu, wa + al + mandb) and

avoid stem fragmentation, aligning with gold morphological

conventions.

4.4. Assessment of High-Salience Boundaries

Critical Boundary Accuracy (CBA) evaluates segmentation perfor-

mance on linguistically salient boundariessuch as proclitics (e.g.,

wa+, fa+, bi+, li+), the deﬁnite article (al+), and enclitics (e.g., +hu,

+hum)that exert disproportionate inﬂuence on downstream tasks.

Table 4 reports CBA scores across systems.

Table 4. Critical Boundary Accuracy (CBA): Performance on

high-impact segments.

System CBA

CAMeL Tools 0.89

Farasa 0.82

BERT-based 0.75

SelfSeg 0.44

BPE 0.41

mBART 0.39

The widening performance gap under CBA conﬁrms that neural

and subword-based systems not only generate more errors overall,

but disproportionately fail on boundaries that are most conse-

quential for linguistic interpretation. In the qualitative examples,

failures are concentrated in clitic chains and article attachment

(e.g., fa + al + wjib, li + al + wu, al + masjidayn), where sub-

word tokenizers fragment stems and SelfSeg often collapses required

boundaries.

4.5. Statistical Veriﬁcation and Reproducibility

All observed performance diﬀerences were validated using McNe-

mars test with Bonferroni correction for multiple comparisons.

Rule-based systems signiﬁcantly outperform neural and subword-

based approaches (p < 0.001), with large eﬀect sizes (Cohens

h > 0.5).

In accordance with TBSE reproducibility standards, the full

experimental pipelineincluding the 1,000-resample bo otstrap proce-

dure used to estimate conﬁdence intervalsis fully containerized. Each

table in this section can be regenerated via a single command within

the ABWS evaluation environment.

4.6. Summary of Benchmarking Insights

The application of ABWS yields three core conclusions:

• Architecture Dictates Boundary Precision: Segmentation

quality is primarily determined by architectural assumptions.

Rule-based systems preserve linguistically valid boundaries and

avoid stem fragmentation, yielding the strongest EM and bound-

ary diagnostics.

• Aggregate Metrics are Insuﬃcient: Word-level accuracy

alone obscures severe paradigm-speciﬁc biases. Boundary-aware

diagnostics are necessary to expose over-segmentation in sub-

word models and boundary omission in language-agnostic neural

tokenizers.

• Standardization Enables Diagnostic Insight: Canonical

boundary projection enables a comprehensive, multi-paradigm

evaluation under controlled conditions and provides explanatory

power by linking numerical scores to concrete linguistic failure

modes.

5. Discussion

The empirical results presented in Section 4 reveal a substantial per-

formance gap between segmentation architectural paradigms when

evaluated on the ABWS representative workload. As shown in Ta-

ble [ 1], rule-based and hybrid systems such as Farasa (0.81), CAMeL

Tools (0.81), and ALP (0.79) maintain relatively high boundary ﬁ-

delity, reﬂecting their explicit modeling of Arabic morphology. In

contrast, modern neural architectures and subword tokenizers ex-

hibit a catastrophic degradation in performance: BPE (0.102) and

mBART (0.122) fail to capture even basic clitic and stem boundaries,

despite their widespread use in downstream neural pipelines.

The observed performance degradation in neural subword mod-

els, such as mBART and BPE-based architectures, stems from a

fundamental misalignment between computational eﬃciency and

linguistic morphology. Unlike rule-based systems that prioritize mor-

pheme boundaries, subword tokenization algorithms are driven by

information-theoretic compression (e.g., maximizing likelihood or

frequency). Consequently, these models often ignore critical linguis-

tic boundariessuch as the junction between a proclitic (e.g., the

conjunction ’w-’) and a stemif a non-linguistic grouping provides a

more frequent statistical pattern in the training corpus. This ’mech-

anistic’ bias leads to the masking of functional particles, where a

model may treat a preﬁxed word as a single opaque unit rather than

a decomposable structure. Our CBA metric captures this failure by

penalizing these statistically-driven but linguistically-invalid merges,

which are particularly prevalent in the high-density Classical Arabic

workload of our benchmark.

Regarding the composition of the ABWS workload, the in-

clusion of high-density Classical Arabic textsspeciﬁcally legal and

jurisprudential treatises like Shari al-Islmis a deliberate design choice

rather than a limitation. These texts exhibit a signiﬁcantly higher

morphological density and a more complex clitic-stacking behavior

compared to modern news or technical documents. By evaluating

systems on this corpus, ABWS functions as a rigorous ’stress-test’

for segmentation models. We argue that a system capable of ac-

curately navigating the intricate boundary decisions of Classical

Arabic is inherently more robust and better prepared for the lin-

guistic variations of Modern Standard Arabic (MSA). Thus, this

workload serves as a high-water mark for evaluating the precision

and diagnostic limits of current Arabic NLP architectures.

5.1. The Failure of Subword Tokenization

The output analysis in Section 4.1 exposes a pronounced reality gap

between subword-based segmentation models and linguistically valid

Arabic morphology. In BPE and mBART, segmentation decisions

are driven primarily by statistical frequency and vocabulary com-

pression rather than morphemic structure. For example, the word

fa-al-wjib (so the obligation) is correctly decomposed by ALP and

Farasa into the clitic-aware sequence [fa, al, wjib]. By contrast,

mBART produces fragmented outputs such as [fal, w, jib], which

do not correspond to any valid morphological units in Arabic.

This behavior conﬁrms that subword-based neural models, de-

spite their apparent ﬂuency in downstream tasks, operate on

a predominantly surface-level representation that lacks structural

awareness of Arabic clitic attachment and stem integrity. From a

benchmarking perspective concerned with traceability and linguis-

tic correctness, these ﬁndings indicate that subword-level metrics

are poor proxies for morphological truth and can substantially

misrepresent actual segmentation quality.

5.2. Robustness to Domain-Speciﬁc Morphology

The evaluated workload is dominated by Classical Arabic jurispru-

dential (Fiqh) terminology, including morphologically dense and

derivationally complex forms such as al-istiba and al-mustaa. Tra-

ditional segmentation systems (Farasa and CAMeL Tools) demon-

strate robustness in this setting due to their reliance on explicit

morphological analyzers and lexicons. These systems consistently

preserve canonical preﬁx, stem, and suﬃx boundaries even in

specialized domains.

Neural models, however, exhibit marked performance degra-

dation. The BERT-based segmenter achieves moderate overall

accuracy (0.46) but still struggles with complex preﬁx–suﬃx com-

binations. For instance, forms such as wa-al-mandb are segmented

as [wal-man, db], indicating partial boundary drift and loss of mor-

phemic coherence. This behavior suggests a high evaluation risk

when deploying neural segmentation models in specialized or low-

frequency domains, where memorized subword statistics fail to

generalize underlying morphological rules.

5.3. Impact on Downstream Tasks

To address the correlation between ABWS metrics and downstream

NLP performance, we conducted a pilot study focusing on Part-

of-Speech (POS) tagging—a critical downstream task sensitive to

segmentation quality. Our experiments, involving multiple archi-

tectures (including BiLSTM and Stanza), demonstrate a strong

positive correlation (ρ > 0.88) between Critical Boundary Accu-

racy (CBA) and tagging macro-F1 scores. Speciﬁcally, we observed

that errors identiﬁed by ABWS as ‘Under-segmentation of Procli-

tics’ (high USR) lead to a disproportionate drop in POS accuracy

compared to simple stem boundary shifts. For instance, when the

CBA score fell below 0.85, the downstream POS tagger’s ability to

correctly identify functional markers (e.g., particles and conjunc-

tions) degraded by over 12%. These ﬁndings empirically validate

that the diagnostic metrics provided by ABWS are not merely in-

trinsic measures but are reliable predictors of a model’s utility in

complex Arabic NLP pipelines.

5.4. Implications for Standardization and Evaluation

Theory

From a workload characterization perspective, these results strongly

justify the design choices underlying the ABWS framework. Con-

ventional evaluation practices often mask the observed failures by

relying on aggregate metrics (e.g., BLEU or token-level F

) com-

puted over overlapping subwords, thereby conﬂating surface overlap

with linguistic correctness. By enforcing a Canonical Boundary Vec-

tor (CBV) representation, ABWS exposes fundamental limitations

that remain invisible under traditional evaluation regimes.

Speciﬁcally, the results demonstrate that:

• Neural and subword-based segmenters are not yet standard-ready

for high-precision linguistic tasks that require reliable boundary

placement.

• Evaluation equivalence between rule-based and neural systems

is unattainable without a paradigm-agnostic representation and

metric suite, such as those proposed in this work.

In summary, the current reality of Arabic NLP benchmarking

reﬂects a trade-oﬀ between the scalability and ﬂexibility of neural

models and the boundary precision of rule-based systems. For crit-

ical applications such as legal, religious, or scholarly text analysis,

the high error rates observed for SelfSeg (0.163), BPE (0.102), and

mBART (0.122) render these approaches unsuitable in their current

form. These ﬁndings underscore the urgent need for boundary-aware

training objectives and evaluation frameworks in the next generation

of large language models for Arabic.

6. Conclusion and Future Work

In this work, we introduced the Arabic Boundary Word Segmenta-

tion (ABWS) framework, a multi-paradigm benchmark designed to

address the lack of standardization in Arabic morphological evalu-

ation. By formalizing the Canonical Boundary Vector (CBV), we

provided a methodology to evaluate systems ranging from tradi-

tional rule-based analyzers to modern neural subword tokenizers

within a uniﬁed, equivalent evaluation condition (EC).

Our empirical results, based on a representative workload of

212,873 words, reveal a profound "reality gap" in current Arabic

NLP. While rule-based systems like Farasa and Camel achieve high

boundary accuracy (0.81), state-of-the-art neural models and sta-

tistical tokenizers such as mBART (0.122) and BPE (0.102) show

catastrophic failure in capturing linguistically valid boundaries.

This disparity highlights a signiﬁcant evaluation risk: conventional

metrics used in downstream tasks often mask a systemic lack of

morphological awareness in Large Language Models (LLMs).

ABWS contributes to the engineering of evaluation by provid-

ing a containerized, reproducible pipeline that ensures benchmark

traceability. By treating dataset provenance and workload character-

ization as ﬁrst-class artifacts, this benchmark allows for the rigorous

comparison of diverse architectures, ensuring that progress in Arabic

NLP is measured against a ground-truth linguistic standard rather

than surface-level statistical frequency.

While ABWS is speciﬁcally designed for Arabic, its core method-

ological contributions are language-agnostic. The Canonical Bound-

ary Vector (CBV) abstraction provides a general solution for com-

paring outputs from disparate segmentation paradigms (rule-based,

statistical, neural) in any language. The boundary-aware metrics

(e.g., OSR, USR, CBA) are deﬁned at the character level and do not

rely on Arabic-speciﬁc features, making them transferable to other

morphologically rich languages (MRLs) such as Hebrew, Turkish, or

Finnish. However, the empirical ﬁndings reported in this papersuch

as the extreme over-segmentation of subword tokenizersare directly

AlShuhayeb et al.

tied to Arabic’s unique morphological structure (e.g., concatenative

cliticization). While similar phenomena may occur in other MRLs,

further experiments are needed to conﬁrm cross-lingual patterns.

Future work will focus on expanding the ABWS workload to

include more diverse dialects and low-resource historical texts. Fur-

thermore, we intend to integrate automated artifact evaluation tools

to further streamline the repro ducibility of results across diﬀerent

hardware testbeds. Ultimately, ABWS oﬀers a template for how

complex, multi-layered NLP tasks can be standardized to support

cumulative scientiﬁc progress and reliable real-world deployment.

Ethical Statement

No ethical approval was required for this study, as it did not involve

human or animal subjects.

Funding

This research received no speciﬁc grant from any funding agency in

the public, commercial, or not-for-proﬁt sectors.

Declaration of competing interests

The authors declare that they have no known competing ﬁnan-

cial interests or personal relationships that could have appeared to

inﬂuence the work reported in this paper.

Data Availability Statements

The data supporting the ﬁndings of this study are openly available

in zenodo at https://zenodo.org/records/18138582 or https://

doi.org/10.5281/zenodo.18138582.

Credit authorship contribution statement

Behrouz Minaei-Bidgoli: Supervision; Methodology; Validation;

Writing Review & Editing. Huda AlShuhayeb: Conceptualiza-

tion; Methodology; Formal Analysis; Investigation; Visualization;

Writing Original Draft.

References

1. N. Y. Habash, Introduction to Arabic Natural Language

Processing, ser. Synthesis Lectures on Human Language

Technologies. Morgan & Claypool Publishers, 2010, doi:

https://doi.org/10.2200/S00277ED1V01Y201008HLT010.

2. R. Zbib et al., “Machine translation of arabic dialects,” in

Proceedings of the 2012 conference of the north american

chapter of the association for computational linguistics:

Human language technologies, 2012, pp. 49–59, doi:

https://doi.org/10.5555/2382029.2382037. [Online]. Available:

https://aclanthology.org/N12-1006/

3. K. Darwish, “Building a shallow arabic morphological analyser

in one day,” in Proceedings of the ACL-02 workshop on

Computational approaches to semitic languages, 2002, doi:

https://doi.org/10.3115/1118637.1118643. [Online]. Available:

https://aclanthology.org/W02-0506/

4. K. Gorman and S. Bedrick, “We need to talk about

standard splits,” in Proceedings of the 57th Annual Meeting

of the Association for Computational Linguistics, 2019, pp.

2786–2791, doi: https://doi.org/10.18653/v1/P19-1267.

5. N. Habash and O. Rambow, “Arabic tokenization, part-

of-speech tagging and morphological disambiguation in one

fell swoop,” in Proceedings of the 43rd annual meeting of

the association for computational linguistics (ACL05), 2005,

pp. 573–580, doi: https://doi.org/10.3115/1219840.1219911.

[Online]. Available: https://aclanthology.org/P05-1071/

6. F. Han et al., “Open source evaluatology: A theoreti-

cal framework for open-source evaluation,” BenchCouncil

Transactions on Benchmarks, Standards and Evalua-

tions, vol. 4, p. 100190, 2024. [Online]. Available:

https://doi.org/10.1016/j.tbench.2025.100190

7. X. Li, H. Zhou, Q. Li, S. Zhang, and G. Lu, “Aicb:

A benchmark for evaluating the communication subsystem

of LLM training clusters,” BenchCouncil Transactions on

Benchmarks, Standards and Evaluations, vol. 5, p. 100212,

2025, doi: https://doi.org/10.1016/j.tbench.2025.100212.

8. J. Xie et al., “COADBench: A benchmark for revealing

the relationship between AI models and clinical outcomes,”

BenchCouncil Transactions on Benchmarks, Standards and

Evaluations, vol. 4, p. 100198, 2025, tBSE paper (uploaded

PDF: S2772485925000110). doi: https://doi.org/10.1016/j.

tbench.2025.100198.

9. M. Maamouri, A. Bies, T. Buckwalter, and W. Mekki,

“The penn arabic treebank: Building a large-scale an-

notated arabic corpus,” in

NEMLAR Conference on

Arabic Language Resources and Tools, 2004. [Online].

Available: https://www.marefa.org/images/e/e8/The_penn_

arabic_treebank_Building_a_large-scale_an_%281%29.pdf

10. R. Roth, O. Rambow, N. Habash, M. Diab, and

C. Rudin, “Arabic morphological tagging, diacritiza-

tion, and lemmatization using lexeme models and feature

ranking,” in Proceedings of ACL-08: HLT, 2008, doi:

https://doi.org/10.3115/1557690.1557721. [Online]. Available:

https://aclanthology.org/P08-2030.pdf

11. M. Boudchiche, A. Mazroui, M. Bebah, A. Lakhouaja,

and A. Boudlal, “Alkhalil morpho sys 2: A robust arabic

morpho-syntactic analyzer,” Journal of King Saud University –

Computer and Information Sciences, vol. 29, no. 2, pp. 141–

146, 2017, doi: https://doi.org/10.1016/j.jksuci.2016.08.003.

[Online]. Available: https://www.sciencedirect.com/science/

article/pii/S131915781630026X

12. W. Zaghouani, “Critical survey of the freely available arabic

corpora,” in Proceedings of LREC, 2014. [Online]. Avail-

able: https://www.researchgate.net/proﬁle/Wajdi-Zaghouani/

publication/263215246_Critical_Survey_of_the_Freely_

Available_Arabic_Corpora/links/0046353a53977808fa000000/

Critical-Survey-of-the-Freely-Available-Arabic-Corpora.pdf

13. A. Abdelali, K. Darwish, N. Durrani, and H. Mubarak,

“Farasa: A fast and furious segmenter for arabic,” in

Proceedings of NAACL-HLT, 2016, doi: https://doi.org/10.

18653/v1/N16-3003. [Online]. Available: https://aclanthology.

org/N16-3003.pdf

14. R. Sennrich, B. Haddow, and A. Birch, “Neural machine

translation of rare words with subword units,” in Proceedings

of ACL, 2016, pp. 1715–1725, doi: https://doi.org/10.18653/

v1/P16-1162. [Online]. Available: https://aclanthology.org/

P16-1162.pdf

15. T. Kudo and J. Richardson, “Sentencepiece: A simple and

language independent subword tokenizer and detokenizer for

neural text processing,” in Proceedings of EMNLP, 2018,

doi: https://doi.org/10.18653/v1/D18-2012. [Online]. Avail-

able: https://aclanthology.org/anthology-ﬁles/anthology-ﬁles/

pdf/D/D18/D18-2.pdf#page=78

16. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova,

“BERT: Pre-training of deep bidirectional transformers for

language understanding,” in Proceedings of NAACL-HLT, 2019,

doi: https://doi.org/10.18653/v1/N19-1423. [Online]. Available:

https://aclanthology.org/N19-1423.pdf

17. W. Antoun, F. Baly, and H. Hajj, “AraELECTRA: Pre-training

text discriminators for arabic language understanding,” in

Proceedings of WANLP, 2020. [Online]. Available: https:

//aclanthology.org/2021.wanlp-1.20.pdf

18. M. Abdul-Mageed, A. Elmadany, and E. M. B. Nagoudi,

“ARBERT & MARBERT: Deep bidirectional transformers for

arabic,” in Proceedings of ACL-IJCNLP, 2021, doi: https:

//doi.org/10.18653/v1/2021.acl-long.551. [Online]. Available:

https://aclanthology.org/2021.acl-long.551.pdf

19. N. Zalmout and N. Habash, “Joint diacritization, lemmatization,

normalization, and ﬁne-grained morphological tagging,” in

Proceedings of the 58th Annual Meeting of the Association for

Computational Linguistics, 2020, pp. 8297–8307, doi: https:

//doi.org/10.18653/v1/2020.acl-main.736. [Online]. Available:

https://aclanthology.org/2020.acl-main.736/

20. N. Habash and F. Sadat, “Arabic preprocessing schemes for

statistical machine translation,” in Proceedings of NAACL-HLT,

2006, pp. 49–52. [Online]. Available: https://aclanthology.org/

N06-2013.pdf

21. K. Darwish and D. W. Oard, “Term selection for searching

printed arabic,” in Proceedings of SIGIR, 2003, doi:

https://doi.org/10.1145/564376.564423. [Online]. Available:

https://dl.acm.org/doi/pdf/10.1145/564376.564423

22. Y. Wu et al., “Google’s neural machine translation

system: Bridging the gap between human and machine

translation,” in arXiv preprint arXiv:1609.08144, 2016, doi:

https://doi.org/10.48550/arXiv.1609.08144. [Online]. Available:

https://www.researchgate.net/publication/308646556_Google’

s_Neural_Machine_Translation_System_ Bridging_the_

Gap_between_Human_and_Machine_Translation

23. A. Wang et al., “GLUE: A multi-task benchmark and

analysis platform for natural language understanding,” in

Proceedings of EMNLP Workshop, 2018, doi: https://

doi.org/10.18653/v1/W18-5446. [Online]. Available: https:

//aclanthology.org/W18-5446.pdf

24. ——, “Sup erGLUE: A stickier benchmark for general-purpose

language understanding systems,” in Proceedings of NeurIPS,

2019, doi: https://doi.org/10.48550/arXiv.1905.00537. [On-

line]. Available: https://proceedings.neurips.cc/paper_ﬁles/

paper/2019/ﬁle/4496bf24afe7fab6f046bf4923da8de6-Paper.pdf