ABWS: The Arabic Boundary-aware Word Segmentation Benchmarkfor Reproducible Evaluation
DOI:
https://doi.org/10.66834/9a7y0c13Keywords:
Arabic NLP, Morphological Segmentation, Benchmarking, Reproducibility, Boundary Errors, Error Taxonomy, Benchmark Traceability, Evaluation ConditionsAbstract
With the rapid adoption of natural language processing (NLP) systems for morphologically rich languages, it has become increasingly imperative to standardize a common set of measures and evaluation practices to ensure reproducibility and fair comparison. Arabic word segmentation serves as a foundational layer in the NLP software stack; however, the field remains fragmented due to inconsistent datasets and an overreliance on opaque, aggregate metrics that mask systemic architectural biases.
We present ABWS (Arabic Boundary-aware Word Segmentation), a scalable and publicly available benchmarking system designed for the rigorous, reproducible evaluation of diverse segmentation paradigms. To enable paradigm-agnostic comparison across rule-based, statistical, and neural models, ABWS introduces a canonical boundary vector abstraction that normalizes disparate system outputs into a unified evaluation interface. The benchmarking harness includes a manually verified gold-standard workload of 212{,}873 words across diverse genres and integrates seven widely used segmentation systems as reproducible baselines.
Our systematic evaluation reveals that while neural subword-based models are robust for vocabulary compression, they exhibit extreme Over-Segmentation Ratios (OSR $> 0.58$), leading to a significant drop in word-level exact match accuracy compared to rule-based engines. We further introduce Critical Boundary Accuracy (CBA), a linguistically weighted metric that prioritizes high-impact morphological boundaries. Our cross-layer analysis demonstrates that CBA is highly predictive of downstream performance in Machine Translation and Named Entity Recognition ($\rho > 0.88$), whereas traditional token-level $F_1$ scores often obscure these performance bottlenecks.
By providing a containerized evaluation pipeline and versioned system artifacts, ABWS establishes a new standard for methodological rigor in Arabic NLP research, offering a template for benchmarking other morphologically complex languages within the broader computational ecosystem.
References
1. N. Y. Habash, Introduction to Arabic Natural Language Processing, ser. Synthesis Lectures on Human Language Technologies. Morgan & Claypool Publishers, 2010, doi: https://doi.org/10.2200/S00277ED1V01Y201008HLT010. DOI: https://doi.org/10.2200/S00277ED1V01Y201008HLT010
2. R. Zbib et al., “Machine translation of arabic dialects,” in Proceedings of the 2012 conference of the north american chapter of the association for computational linguistics: Human language technologies, 2012, pp. 49–59, doi: https://doi.org/10.5555/2382029.2382037. [Online]. Available: https://aclanthology.org/N12-1006/
3. K. Darwish, “Building a shallow arabic morphological analyser in one day,” in Proceedings of the ACL-02 workshop on Computational approaches to semitic languages, 2002, doi:https://doi.org/10.3115/1118637.1118643. [Online]. Available: https://aclanthology.org/W02-0506/ DOI: https://doi.org/10.3115/1118637.1118643
4. K. Gorman and S. Bedrick, “We need to talk about standard splits,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019, pp. 2786–2791, doi:https://doi.org/10.18653/v1/P19-1267. DOI: https://doi.org/10.18653/v1/P19-1267
5. N. Habash and O. Rambow, “Arabic tokenization, part-of-speech tagging and morphological disambiguation in one fell swoop,” in Proceedings of the 43rd annual meeting of the association for computational linguistics (ACL05), 2005, pp. 573–580, doi:https://doi.org/10.3115/1219840.1219911. [Online]. Available: https://aclanthology.org/P05-1071/ DOI: https://doi.org/10.3115/1219840.1219911
6. F. Han et al., “Open source evaluatology: A theoretical framework for open-source evaluation,” BenchCouncil Transactions on Benchmarks, Standards and Evaluations, vol. 4, p. 100190, 2024. [Online]. Available: https://doi.org/10.1016/j.tbench.2025.100190 DOI: https://doi.org/10.1016/j.tbench.2025.100190
7. X. Li, H. Zhou, Q. Li, S. Zhang, and G. Lu, “Aicb: A benchmark for evaluating the communication subsystem of LLM training clusters,” BenchCouncil Transactions on Benchmarks, Standards and Evaluations, vol. 5, p. 100212, 2025, doi:https://doi.org/10.1016/j.tbench.2025.100212. DOI: https://doi.org/10.1016/j.tbench.2025.100212
8. J. Xie et al., “COADBench: A benchmark for revealing the relationship between AI models and clinical outcomes,” BenchCouncil Transactions on Benchmarks, Standards and Evaluations, vol. 4, p. 100198, 2025, tBSE paper (uploaded PDF: S2772485925000110). doi:https://doi.org/10.1016/j.tbench.2025.100198. DOI: https://doi.org/10.1016/j.tbench.2025.100198
9. M. Maamouri, A. Bies, T. Buckwalter, and W. Mekki, “The penn arabic treebank: Building a large-scale annotated arabic corpus,” in NEMLAR Conference on Arabic Language Resources and Tools, 2004. [Online]. Available: https://www.marefa.org/images/e/e8/The_penn_arabic_treebank_Building_a_large-scale_an_%281%29.pdf
10. R. Roth, O. Rambow, N. Habash, M. Diab, and C. Rudin, “Arabic morphological tagging, diacritization, and lemmatization using lexeme models and feature ranking,” in Proceedings of ACL-08: HLT, 2008, doi:https://doi.org/10.3115/1557690.1557721. [Online]. Available: https://aclanthology.org/P08-2030.pdf DOI: https://doi.org/10.3115/1557690.1557721
11. M. Boudchiche, A. Mazroui, M. Bebah, A. Lakhouaja, and A. Boudlal, “Alkhalil morpho sys 2: A robust arabic morpho-syntactic analyzer,” Journal of King Saud University–Computer and Information Sciences, vol. 29, no. 2, pp. 141–146, 2017, doi: https://doi.org/10.1016/j.jksuci.2016.08.003. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S131915781630026X DOI: https://doi.org/10.1016/j.jksuci.2016.05.002
12. W. Zaghouani, “Critical survey of the freely available arabic corpora,” in Proceedings of LREC, 2014. [Online]. Available: https://www.researchgate.net/profile/Wajdi-Zaghouani/publication/263215246_Critical_Survey_of_the_Freely_Available_Arabic_Corpora/links/0046353a53977808fa000000/Critical-Survey-of-the-Freely-Available-Arabic-Corpora.pdf
13. A. Abdelali, K. Darwish, N. Durrani, and H. Mubarak, “Farasa: A fast and furious segmenter for arabic,” in Proceedings of NAACL-HLT, 2016, doi:https://doi.org/10.18653/v1/N16-3003. [Online]. Available: https://aclanthology.org/N16-3003.pdf DOI: https://doi.org/10.18653/v1/N16-3003
14. R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” in Proceedings of ACL, 2016, pp. 1715–1725, doi:https://doi.org/10.18653/v1/P16-1162. [Online]. Available: https://aclanthology.org/P16-1162.pdf DOI: https://doi.org/10.18653/v1/P16-1162
15. T. Kudo and J. Richardson, “Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing,” in Proceedings of EMNLP, 2018, doi:https://doi.org/10.18653/v1/D18-2012. [Online]. Available: https://aclanthology.org/anthology-files/anthology-files/pdf/D/D18/D18-2.pdf#page=78
16. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of NAACL-HLT, 2019, doi:https://doi.org/10.18653/v1/N19-1423. [Online]. Available: https://aclanthology.org/N19-1423.pdf
17. W. Antoun, F. Baly, and H. Hajj, “AraELECTRA: Pre-training text discriminators for arabic language understanding,” in Proceedings of WANLP, 2020. [Online]. Available: https://aclanthology.org/2021.wanlp-1.20.pdf
18. M. Abdul-Mageed, A. Elmadany, and E. M. B. Nagoudi, “ARBERT & MARBERT: Deep bidirectional transformers for arabic,” in Proceedings of ACL-IJCNLP, 2021, doi:https://doi.org/10.18653/v1/2021.acl-long.551. [Online]. Available: https://aclanthology.org/2021.acl-long.551.pdf
19. N. Zalmout and N. Habash, “Joint diacritization, lemmatization, normalization, and fine-grained morphological tagging,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 8297–8307, doi:https://doi.org/10.18653/v1/2020.acl-main.736. [Online]. Available: https://aclanthology.org/2020.acl-main.736/ DOI: https://doi.org/10.18653/v1/2020.acl-main.736
20. N. Habash and F. Sadat, “Arabic preprocessing schemes for statistical machine translation,” in Proceedings of NAACL-HLT, 2006, pp. 49–52. [Online]. Available: https://aclanthology.org/N06-2013.pdf DOI: https://doi.org/10.3115/1614049.1614062
21. K. Darwish and D. W. Oard, “Term selection for searching printed arabic,” in Proceedings of SIGIR, 2003, doi:https://doi.org/10.1145/564376.564423. [Online]. Available: https://dl.acm.org/doi/pdf/10.1145/564376.564423
22. Y. Wu et al., “Google’s neural machine translation system: Bridging the gap between human and machine translation,” in arXiv preprint arXiv:1609.08144, 2016, doi:https://doi.org/10.48550/arXiv.1609.08144. [Online]. Available: https://www.researchgate.net/publication/308646556_Google’s_Neural_Machine_Translation_System_Bridging_the_Gap_between_Human_and_Machine_Translation
23. A. Wang et al., “GLUE: A multi-task benchmark and analysis platform for natural language understanding,” in Proceedings of EMNLP Workshop, 2018, doi:https://doi.org/10.18653/v1/W18-5446. [Online]. Available: DOI: https://doi.org/10.18653/v1/W18-5446
https://aclanthology.org/W18-5446.pdf
24. ——, “SuperGLUE: A stickier benchmark for general-purpose language understanding systems,” in Proceedings of NeurIPS, 2019, doi:https://doi.org/10.48550/arXiv.1905.00537. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2019/file/4496bf24afe7fab6f046bf4923da8de6-Paper.pdf
Published
Issue
Section
License
Copyright (c) 2026 The Authors. Published by BenchCouncil Press on Behalf of International Open Benchmark Council
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
