Bihari भोजपुरी

38,630 Words in vocabulary

4.11x Best compression

0.8673 Best isotropy

Sample text

Excerpts from Bihari Wikipedia articles.

नेल्सन मंडेला दक्खिन अफिरका के पहिला करिया राष्ट्रपति आ पहिला चुनल गइल राष्ट्रपत...

बबुआ कलां भारत के झारखंड राज्य में एक ठो कसबा बाटे। के शहर‏‎ आ कस्बा

घटना जनम - मन्मथनाथ गुप्त - भारतीय स्वतन्त्रता संग्राम क एगो प्रमुख क्रान्तिकारी...

The 20 most frequently used words in Bihari Wikipedia.

Explore Bihari interactively with browser-based demos.

Key metrics for all model types at a glance.

from wikilangs import tokenizer
tok = tokenizer('latest', 'bh', 32000)
tokens = tok.tokenize("Your text here")

from wikilangs import ngram
ng = ngram('latest', 'bh', gram_size=3)
score = ng.score("Your text here")

from wikilangs import markov
mc = markov('latest', 'bh', depth=3)
text = mc.generate(length=50)

from wikilangs import vocabulary
vocab = vocabulary('latest', 'bh')
info = vocab.lookup("word")

from wikilangs import embeddings
emb = embeddings('latest', 'bh', dimension=64)
vec = emb.embed_word("word")

Model Type	Variants	Description
Tokenizers	8k, 16k, 32k, 64k	BPE tokenizers with different vocabulary sizes
N-gram (Word)	2, 3, 4, 5-gram	Word-level language models
N-gram (Subword)	2, 3, 4, 5-gram	Subword-level language models
Markov (Word)	Depth 1–5	Word-level text generation
Markov (Subword)	Depth 1–5	Subword-level text generation
Vocabulary	—	Word dictionary with frequency and IDF
Embeddings	32d, 64d, 128d	Position-aware word embeddings