§ Arabic language infrastructure

Arabic is hard.
Your pipeline should not be.

One API and two SDKs for the unglamorous layer under your Arabic product: diacritization, dialect handling, normalization, morphological segmentation, entity extraction, and an evaluation harness that tells you whether any of it actually got better.

Get an API key Read the docs

endpoints7 — diacritize, normalize, dialect, segment, entities, evaluate, batch
sdksPython 3.9+ · Node 18+ · plain HTTP
hostingManaged API or self-hosted container
free tier50,000 characters / month, no card

# POST a bare MSA string, get it back fully vowelled
curl -X POST https://api.harfscript.com/v1/diacritize \
  -H "Authorization: Bearer hs_live_9f2c4a…" \
  -H "Content-Type: application/json" \
  -d '{
        "text": "كتب الطالب الدرس في المكتبة",
        "mode": "full"
      }'

● 200 OK·api.harfscript.com·142 ms

{
  "object": "diacritization",
  "model": "harf-diac-2",
  "text": "كَتَبَ الطَّالِبُ الدَّرْسَ فِي الْمَكْتَبَةِ",
  "tokens": [
    { "raw": "كتب",   "out": "كَتَبَ",   "pos": "VERB" },
    { "raw": "الطالب", "out": "الطَّالِبُ", "pos": "NOUN" },
    { "raw": "الدرس",  "out": "الدَّرْسَ",  "pos": "NOUN" }
  ],
  "chars_billed": 27
}

from harfscript import Harfscript

hs = Harfscript(api_key=os.environ["HARFSCRIPT_API_KEY"])

# 1. which dialect am I even looking at?
tag = hs.dialect.identify("وينك يا زلمة؟ صرلي ساعة بستناك")

# 2. normalise it into MSA before it hits your model
msa = hs.dialect.to_msa("شو بدك تاكل اليوم؟")
print(msa.text)  # → "ماذا تريد أن تأكل اليوم؟"

● 200 OK·2 requests·207 ms

{
  "object": "dialect_identification",
  "top": "lev",
  "scores": { "lev": 0.94, "egy": 0.03, "glf": 0.02, "msa": 0.01 },
  "region_hint": "south_levantine",
  "code_switching": false
}
{
  "object": "msa_conversion",
  "source_dialect": "lev",
  "text": "ماذا تريد أن تأكل اليوم؟",
  "preserved_spans": []
}

import { Harfscript } from "@harfscript/sdk";

const hs = new Harfscript({ apiKey: process.env.HARFSCRIPT_API_KEY });

const { entities } = await hs.entities.extract({
  text: "سافر أحمد الخالدي من غزة إلى عمّان لحضور مؤتمر جامعة النجاح الوطنية.",
  types: ["PER", "LOC", "ORG"],
  normalize: true,
});

● 200 OK·api.harfscript.com·118 ms

[
  { "text": "أحمد الخالدي", "type": "PER", "start": 5,  "end": 17 },
  { "text": "غزة",          "type": "LOC", "start": 21, "end": 24 },
  { "text": "عمّان",        "type": "LOC", "start": 29, "end": 34,
    "note": "shadda-tolerant match; folds to عمان" },
  { "text": "جامعة النجاح الوطنية", "type": "ORG", "start": 47, "end": 67 }
]

sample requests and responses · latency and confidence values are illustrative, not published benchmark results — see what we publish

§01 The problem

Arabic breaks most NLP pipelines quietly.

Nothing throws an exception. The text goes in, something comes out, and the quality loss only shows up in production — in search results that miss, in a transcript that reads wrong, in a model that never learned the difference between two words spelled identically.

Diacritics are missing

Written Arabic normally drops short vowels. The same four consonants can be a noun, a verb, or a causative verb — and a reader resolves it from context that your tokenizer does not have. Text-to-speech guesses. Search conflates. Training data is ambiguous by construction.

rasm علم
reads as عِلْم · عَلَم · عَلَّمَ · عَلِمَ

Dialects are not a rounding error

Almost everything people actually type is dialectal. Levantine, Egyptian, Gulf and Maghrebi diverge from Modern Standard Arabic in vocabulary, morphology and spelling convention — often further than some European languages diverge from each other. Models trained on MSA news corpora meet none of it.

lev شو بدك تاكل اليوم؟
egy إنت عايز تاكل إيه النهارده؟
msa ماذا تريد أن تأكل اليوم؟

Right-to-left text mangles on the way through

Mixed Arabic and Latin runs reorder unpredictably once a string passes through a template, a log line, a PDF renderer or a CSV. Offsets drift. Highlight spans land on the wrong characters. Presentation forms and tatweel sneak into your index and silently split what should be one token.

in أهلاً وسهلاً بكم فى مِصــر
out اهلا وسهلا بكم في مصر
ops alef-fold · ya-fix · strip-tatweel

English-shaped tokenizers shred Arabic morphology

Arabic packs conjunctions, prepositions, articles and pronouns into a single orthographic word. Subword vocabularies built on English corpora cut those clitics in the wrong places, so a stem never gets seen often enough to be learned properly.

word وبالمكتبات
segs و + ب + ال + مكتب + ات
gloss and + in + the + librar- + -ies

Teams hit this and take one of two bad options: ship weak Arabic support, or spend two or three months rebuilding preprocessing that is not their product.

§02 What Harfscript is

Infrastructure, deliberately below the model layer.

Harfscript is not another chatbot and does not want to be your model. It is the preprocessing, normalization and evaluation layer that everything above it depends on — the part every Arabic team writes badly, once, under deadline.

input · raw MSAbefore

كتب الطالب الدرس في المكتبة

ambiguity: kutiba / kataba / kattaba
tts: vowel guessed at synthesis time
search: collides with unrelated stems

output · POST /v1/diacritizeafter

كَتَبَ الطَّالِبُ الدَّرْسَ فِي الْمَكْتَبَةِ

reading: kataba ṭ-ṭālibu d-darsa fī l-maktabati
gloss: the student wrote the lesson in the library
case: nominative subject, accusative object, resolved

عِلْم

ʿilm — knowledge

عَلَم

ʿalam — flag

عَلَّمَ

ʿallama — he taught

عَلِمَ

ʿalima — he knew

Four words, one undiacritized spelling: علم. This is the single decision that most Arabic pipelines never make explicitly.

§03 Capabilities

Seven things, each of them done properly.

Every capability is one endpoint, one SDK method, and one line in the evaluation harness. No orchestration layer, no agent framework, no prompt to tune.

POST /v1/diacritize

Diacritization

Full or partial tashkīl over MSA, with per-token alternatives and confidence, plus a case_endings switch for when you only need internal vowels.

ذهب → ذَهَبَ | ذَهَبٌ

POST /v1/normalize

Normalization

Deterministic, reversible, configurable: alef and hamza folding, ya and ta-marbuta handling, tatweel and presentation-form stripping, Arabic-Indic digit mapping.

فى → في · ٢٠٢٦ → 2026

POST /v1/dialect

Dialect ID & MSA conversion

Labels Levantine, Egyptian, Gulf, Maghrebi, Iraqi, Sudanese and MSA with a region hint and a code-switching flag — then converts to MSA when your downstream model needs it.

شلونك؟ → glf · كيداير؟ → mgr

POST /v1/segment

Arabic-aware tokenization

Morphological segmentation that splits clitics where Arabic actually splits them, with character offsets that survive the round trip back into your original string.

يكتبونها → ي+كتب+ون+ها

POST /v1/entities

Arabic-tuned NER

Person, location and organisation extraction that understands kunya and nasab name structures, definite-article prefixes on place names, and spelling variance across transliterated forms.

رام الله → LOC · أبو خالد → PER

POST /v1/evaluate

Evaluation harness

Arabic-specific metrics — diacritic error rate, word error rate with and without case endings, dialect-stratified accuracy — so you can measure a change instead of arguing about it.

DER · WER · WER-noCE · per-dialect

SDK python · node

Two SDKs, no surprises

Typed clients for Python 3.9+ and Node 18+, with sync and async calls, automatic retry with backoff, batching, and streaming for long documents.

pip install harfscript
npm i @harfscript/sdk

SELF hosted

Runs inside your network

The same API as a container image for teams whose text cannot leave their infrastructure — legal, health, government and publishing archives. Offline licence, no callback.

docker run harfscript/server:0.4

POST /v1/batch

Corpus-scale batch

Submit a JSONL corpus, poll a job, collect results. Built for the archive migration and the training-set rebuild, not just the single request.

jsonl in → jsonl out · resumable

§04 How it works

Three steps, then it is just a function call.

There is no console to learn and no pipeline to configure. Install the SDK, export a key, call the endpoint you need. Everything else is documentation.

01

Install

pip install harfscript or npm i @harfscript/sdk. Plain HTTP works too — every endpoint is JSON in, JSON out.
02

Authenticate

Bearer token in the Authorization header. Keys are scoped per environment and rotate without downtime.
03

Call, then measure

Run the endpoint over your text, then run /v1/evaluate over the result against your own gold set. Billing is by character, counted after normalization.

# pip install harfscript
import os
from harfscript import Harfscript

hs = Harfscript(api_key=os.environ["HARFSCRIPT_API_KEY"])

# clean → segment → diacritize, in the order that actually works
clean = hs.normalize(raw, fold_alef=True, strip_tatweel=True)
segs  = hs.segment(clean.text, scheme="atb")
out   = hs.diacritize(clean.text, mode="full")

print(out.text, out.chars_billed)

// npm i @harfscript/sdk
import { Harfscript } from "@harfscript/sdk";

const hs = new Harfscript({ apiKey: process.env.HARFSCRIPT_API_KEY });

const clean = await hs.normalize({ text: raw, foldAlef: true });
const out   = await hs.diacritize({ text: clean.text, mode: "full" });

console.log(out.text);
// → "كَتَبَ الطَّالِبُ الدَّرْسَ فِي الْمَكْتَبَةِ"

# measure the change instead of trusting it
report = hs.evaluate(
    task="diacritization",
    predictions="runs/2026-08-04.jsonl",
    references="gold/msa_news_500.jsonl",
    metrics=["der", "wer", "wer_no_case"],
    stratify_by="dialect",
)

report.to_markdown("eval/report.md")
# the harness is the product too — you get the numbers, not our word for them

§04.1 API reference

base url · https://api.harfscript.com — all endpoints accept and return application/json
Endpoint	Does	Key parameters	Billed
POST /v1/diacritize	Restores short vowels and case endings on MSA text.	mode, case_endings, alternatives	chars
POST /v1/normalize	Deterministic cleanup: alef, hamza, ya, tatweel, digits, presentation forms.	fold_alef, strip_tatweel, digits	free
POST /v1/dialect	Identifies dialect and optionally converts the text to MSA.	action, preserve_spans, region_hint	chars
POST /v1/segment	Morphological segmentation with offsets that map back to the source.	scheme, keep_offsets	chars
POST /v1/entities	Named entity extraction tuned for Arabic names and places.	types, normalize, link	chars
POST /v1/evaluate	Scores predictions against references with Arabic-specific metrics.	task, metrics, stratify_by	free
POST /v1/batch	Queues a JSONL corpus for any of the above; resumable, pollable.	endpoint, input_url, webhook	chars

errors follow one shape · { "error": { "type", "code", "message", "param" } } · 429 carries Retry-After · every response carries x-harf-request-id

§05 Pricing

Metered by character. No seats.

Characters are counted after normalization, so you are not billed for tatweel and whitespace. Normalization and evaluation calls are free on every plan — measuring should never cost you.

Free

no card

$050,000 characters / month

Enough to evaluate every endpoint against your own corpus before you decide anything.

All seven endpoints
Python and Node SDKs
Evaluation harness, unmetered
Community support

Get an API key

Usage

most teams

$0.20per 100,000 characters, from

Pay for what you send. Volume steps down automatically at 10M and 100M characters per month.

Everything in Free
Batch jobs and webhooks
Per-environment keys and usage caps
Email support, one business day

Get an API key

Self-hosted

annual

$6,000per year, from

The same API as a container inside your network, for text that is not allowed to leave it.

Air-gapped licence, no callback
Unlimited characters on your hardware
Model updates twice a year
Named contact and deployment review

Talk about licensing

free tier resets monthly · normalize and evaluate are never metered · no per-seat pricing, ever · academic and non-profit rates on request

§06 What we publish

Evidence, not adjectives.

Harfscript is in public beta and we are not going to print customer logos we have not earned or benchmark numbers we cannot reproduce. Here is exactly what we commit to publishing, and the shape it will take when it lands.

An open evaluation harness

The same /v1/evaluate endpoint we score ourselves with is the one you call. Test sets, splits and scoring code are published so a result can be reproduced or disputed.

Volume reported after the fact

Requests served per month will be published on the changelog once the numbers are worth reporting. Until then the honest figure is: early, and growing from a small base.

Named teams, only with consent

No logo goes on this page without a signed reference agreement and a real deployment behind it. If you see a name here later, it is because someone chose to put it here.

illustrative structure only — every cell is a placeholder until the harness is public
Task	Test set	Metric	Harfscript	Open baseline
diacritization	MSA news, held-out	DER ↓	—	—
diacritization	MSA news, held-out	WER (no case) ↓	—	—
dialect id	6-way, social text	macro-F1 ↑	—	—
segmentation	treebank-style	token F1 ↑	—	—
entities	mixed MSA + dialect	span F1 ↑	—	—

This table is a layout for numbers that do not exist yet. We are not claiming a score against any named system, and we will not fill these cells until the test sets and the scoring code are downloadable.

Start with the free tier and your own corpus.

50,000 characters a month, all seven endpoints, no card. Point it at text you already have and read the diff yourself — that is the only demo worth trusting.