DKSplit – Fast Word Segmentation for Python

Fast character-level segmentation for web-style concatenated strings — domain names, hashtags, usernames, slugs. 9 MB ONNX model, CPU-only.

pip install dksplit Requires Python >= 3.8. Dependencies: numpy, onnxruntime.

import dksplit # Single best segmentation dksplit . split ( "kubernetescluster" ) # ['kubernetes', 'cluster'] # Batch (faster for large volumes; results identical to split()) dksplit . split_batch ([ "openaikey" , "microsoftoffice" , "bitcoinprice" ]) # [['openai', 'key'], ['microsoft', 'office'], ['bitcoin', 'price']] # Ranked candidates for ambiguous inputs dksplit . split3 ( "noranite" ) # top-3, best first # [['nora', 'nite'], ['noranite'], ['nor', 'anite']] dksplit . split5 ( "pikahug" ) # top-5 # [['pikahug'], ['pika', 'hug'], ['pik', 'ahug'], ['pikah', 'ug'], ['pi', 'kahug']] dksplit . split_topk ( "chatgptlogin" , k = 3 ) # any k # [['chatgpt', 'login'], ['chatgptlogin'], ['chatgpt', 'log', 'in']] What can you do with it Typical uses: spotting brands and lookalikes in newly registered domains ( yourbrandlogin , getyourbrand ), extracting keywords from domains, hashtags, and URLs, normalizing concatenated identifiers before matching and dedup, understanding spaceless search queries.

Bugfix: split_batch() could differ from split() on rare inputs; results are now guaranteed identical. Pass exact=False to keep the old ~2x faster behavior.

Dataset 1,000 hand-audited domain prefixes drawn from the Newly Registered Domains Database (NRDS) (.com feed). No filtering or cherry-picking on segmentation difficulty. Ground truth was established through multi-model cross-validation (BiLSTM, Qwen 9B LoRA, Gemma 31B) and human audit. Each row provides a primary truth and an optional might_right field for genuinely ambiguous cases (e.g. brand-versus-compound).

Both benchmark sets ship in this repo's /benchmark directory: sample_1000.csv and benchmark_5000.csv , a larger set built the same way (also on Hugging Face as ABTdomain/dksplit-benchmark ). To explore domain data yourself, register at domainkits.com — fresh .com NRD downloads are free.

Model Strict EM Lenient EM DKSplit v1.0.2 86.5% 91.5% WordSegment 65.2% 69.5% WordNinja 51.0% 54.0% Strict EM counts only exact matches against truth . Lenient EM also accepts the might_right alternative when present.

Top-k coverage (an acceptable segmentation is present within the candidates):

git clone https://github.com/ABTdomain/dksplit.git cd dksplit/benchmark pip install dksplit wordsegment wordninja python run_benchmark.py # 1,000-sample set python run_benchmark.py benchmark_5000.csv # 5,000-sample set Adding your own segmenter to the comparison is a one-line change in run_benchmark.py . Pull requests for ambiguous samples are welcome.