hi

Hindi рд╣рд┐рдиреНрджреА

ISO 639-1: hi ISO 639-3: hin I L
503,387 Words in vocabulary
1 Country
1 Continent
4.25x Best compression
0.8141 Best isotropy

Asia

Spoken in

* primary

India* (hi)

Sample text

Excerpts from Hindi Wikipedia articles.

рдЬреЗрд░реЛрдо рдЗрд╕рд╛рдХ рдлреНрд░реАрдбрдорди рдЕрдореЗрд░рд┐рдХрд╛ рдХреЗ рдкреНрд░рд╕рд┐рджреНрдж рд╡реИрдЬреНрдЮрд╛рдирд┐рдХ рд╣реИрдВред рдореЗрдВ рдЗрдиреНрд╣реЗрдВ рднреМрддрд┐рдХ рд╡рд┐рдЬреНрдЮрд╛рди рдо...
рдорд╡реИрдпрд╛ рд╣рдВрдбрд┐рдпрд╛, рдЗрд▓рд╛рд╣рд╛рдмрд╛рдж, рдЙрддреНрддрд░ рдкреНрд░рджреЗрд╢ рд╕реНрдерд┐рдд рдПрдХ рдЧрд╛рдБрд╡ рд╣реИред рднреВрдЧреЛрд▓ рдЬрдирд╕рд╛рдВрдЦреНрдпрд┐рдХреА рдпрд╛рддрд╛рдпрд╛рдд...
рдорд╛рдзрд╡реА рд╣рд┐рдиреНрджреА рдлрд┐рд▓реНрдореЛрдВ рдХреА рдПрдХ рдкреНрд░рд╕рд┐рджреНрдз рдЕрднрд┐рдиреЗрддреНрд░реА рд╣реИрдВред рд╡реНрдпрдХреНрддрд┐рдЧрдд рдЬреАрд╡рди рдлрд┐рд▓реНрдореА рд╕рдлрд░ рдкреНрд░...

Most common words

The 20 most frequently used words in Hindi Wikipedia.

Top 20 words in Hindi

Performance dashboard

Key metrics for all model types at a glance.

Performance dashboard for Hindi

Quick start

Tokenizer

from wikilangs import tokenizer
tok = tokenizer('latest', 'hi', 32000)
tokens = tok.tokenize("Your text here")

N-gram

from wikilangs import ngram
ng = ngram('latest', 'hi', gram_size=3)
score = ng.score("Your text here")

Markov chain

from wikilangs import markov
mc = markov('latest', 'hi', depth=3)
text = mc.generate(length=50)

Vocabulary

from wikilangs import vocabulary
vocab = vocabulary('latest', 'hi')
info = vocab.lookup("word")

Embeddings

from wikilangs import embeddings
emb = embeddings('latest', 'hi', dimension=64)
vec = emb.embed_word("word")

Available models

Model Type Variants Description
Tokenizers8k, 16k, 32k, 64kBPE tokenizers with different vocabulary sizes
N-gram (Word)2, 3, 4, 5-gramWord-level language models
N-gram (Subword)2, 3, 4, 5-gramSubword-level language models
Markov (Word)Depth 1тАУ5Word-level text generation
Markov (Subword)Depth 1тАУ5Subword-level text generation
VocabularyтАФWord dictionary with frequency and IDF
Embeddings32d, 64d, 128dPosition-aware word embeddings

Model evaluation

Tokenizer performance

Compression ratios and token statistics across vocabulary sizes.

Tokenizer compression

N-gram evaluation

Perplexity and entropy metrics across n-gram sizes.

N-gram perplexity

Markov chain evaluation

Entropy and branching factor by context depth.

Markov entropy

Vocabulary analysis

Word frequency distribution and Zipf's law analysis.

Zipf's law
Top 20 words

Embeddings evaluation

Isotropy and vector space quality metrics.

Embedding isotropy

Full research report

Access the complete ablation study with all metrics, visualizations, and generated text samples on HuggingFace.

View on HuggingFace тЖТ