Dialectal Filtering

Matching dialectal markers in large noisy corpora to synthesise novel dialect-specific text corpora for underrepresented Kurdish varieties.

Kurdish NLP faces a fundamental data problem: available text corpora are large but extremely noisy, mixing multiple dialects, scripts, and registers without labelling. A model trained on such data learns an undifferentiated blend rather than the properties of any specific variety. Dialectal Filtering addresses this by using lexicographic features - dialect-specific markers identified from linguistic resources - to filter and partition large raw corpora into variety-specific sub-corpora.

The approach treats dialect identification as a lightweight, lexicon-driven filtering step rather than a full classification task. For each target dialect, a set of discriminative lexical and morphological markers is identified. Documents containing a sufficient density of these markers are assigned to the corresponding dialect sub-corpus. This keeps the pipeline fast, interpretable, and applicable even when only small amounts of labelled data are available.

Dialectal Filtering is one of the core data-creation threads of the Dialect-Aware NLP PhD project. The sub-corpora it produces feed directly into downstream tasks such as dialect-aware machine translation and ASR fine-tuning (see: Kurdish Voices).