Clone this repo:

Branches

Tags

  1. 6b15e31 Merge pull request #38 from vaxelrod/patch-1 by RAMI ALRFOU · 4 weeks ago master
  2. cbd65d5 Add language list to README by Vera Axelrod · 4 weeks ago
  3. 0864b14 Merge pull request #37 from aboSamoor/master by RAMI ALRFOU · 5 weeks ago 3.0.13
  4. 4751b37 rename action by Rami Al-Rfou · 5 weeks ago
  5. 534c686 Add pypi to workflow by Rami Al-Rfou · 5 weeks ago

Compact Language Detector v3 (CLD3)

Model

CLD3 is a neural network model for language identification. This package contains the inference code and a trained model. The inference code extracts character ngrams from the input text and computes the fraction of times each of them appears. For example, as shown in the figure below, if the input text is “banana”, then one of the extracted trigrams is “ana” and the corresponding fraction is 2/4. The ngrams are hashed down to an id within a small range, and each id is represented by a dense embedding vector estimated during training.

The model averages the embeddings corresponding to each ngram type according to the fractions, and the averaged embeddings are concatenated to produce the embedding layer. The remaining components of the network are a hidden (Rectified linear) layer and a softmax layer.

To get a language prediction for the input text, we simply perform a forward pass through the network.

Figure

Supported Languages

The model outputs BCP-47-style language codes, shown in the table below. For some languages, output is differentiated by script. Language and script names from Unicode CLDR.

Output CodeLanguage NameScript Name
afAfrikaansLatin
amAmharicEthiopic
arArabicArabic
bgBulgarianCyrillic
bg-LatnBulgarianLatin
bnBanglaBangla
bsBosnianLatin
caCatalanLatin
cebCebuanoLatin
coCorsicanLatin
csCzechLatin
cyWelshLatin
daDanishLatin
deGermanLatin
elGreekGreek
el-LatnGreekLatin
enEnglishLatin
eoEsperantoLatin
esSpanishLatin
etEstonianLatin
euBasqueLatin
faPersianArabic
fiFinnishLatin
filFilipinoLatin
frFrenchLatin
fyWestern FrisianLatin
gaIrishLatin
gdScottish GaelicLatin
glGalicianLatin
guGujaratiGujarati
haHausaLatin
hawHawaiianLatin
hiHindiDevanagari
hi-LatnHindiLatin
hmnHmongLatin
hrCroatianLatin
htHaitian CreoleLatin
huHungarianLatin
hyArmenianArmenian
idIndonesianLatin
igIgboLatin
isIcelandicLatin
itItalianLatin
iwHebrewHebrew
jaJapaneseJapanese
ja-LatnJapaneseLatin
jvJavaneseLatin
kaGeorgianGeorgian
kkKazakhCyrillic
kmKhmerKhmer
knKannadaKannada
koKoreanKorean
kuKurdishLatin
kyKyrgyzCyrillic
laLatinLatin
lbLuxembourgishLatin
loLaoLao
ltLithuanianLatin
lvLatvianLatin
mgMalagasyLatin
miMaoriLatin
mkMacedonianCyrillic
mlMalayalamMalayalam
mnMongolianCyrillic
mrMarathiDevanagari
msMalayLatin
mtMalteseLatin
myBurmeseMyanmar
neNepaliDevanagari
nlDutchLatin
noNorwegianLatin
nyNyanjaLatin
paPunjabiGurmukhi
plPolishLatin
psPashtoArabic
ptPortugueseLatin
roRomanianLatin
ruRussianCyrillic
ru-LatnRussianEnglish
sdSindhiArabic
siSinhalaSinhala
skSlovakLatin
slSlovenianLatin
smSamoanLatin
snShonaLatin
soSomaliLatin
sqAlbanianLatin
srSerbianCyrillic
stSouthern SothoLatin
suSundaneseLatin
svSwedishLatin
swSwahiliLatin
taTamilTamil
teTeluguTelugu
tgTajikCyrillic
thThaiThai
trTurkishLatin
ukUkrainianCyrillic
urUrduArabic
uzUzbekLatin
viVietnameseLatin
xhXhosaLatin
yiYiddishHebrew
yoYorubaLatin
zhChineseHan (including Simplified and Traditional)
zh-LatnChineseLatin
zuZuluLatin

Installation

CLD3 is designed to run in the Chrome browser, so it relies on code in Chromium. The steps for building and running the demo of the language detection model are:

  • check out the Chromium repository.
  • copy the code to //third_party/cld_3
  • Uncomment language_identifier_main executable in src/BUILD.gn.
  • build and run the model using the commands:
gn gen out/Default
ninja -C out/Default third_party/cld_3/src/src:language_identifier_main
out/Default/language_identifier_main

Bugs and Feature Requests

Open a GitHub issue for this repository to file bugs and feature requests.

Announcements and Discussion

For announcements regarding major updates as well as general discussion list, please subscribe to: cld3-users@googlegroups.com

Credits

Original authors of the code in this package include (in alphabetical order):

  • Alex Salcianu
  • Andy Golding
  • Anton Bakalov
  • Chris Alberti
  • Daniel Andor
  • David Weiss
  • Emily Pitler
  • Greg Coppola
  • Jason Riesa
  • Kuzman Ganchev
  • Michael Ringgaard
  • Nan Hua
  • Ryan McDonald
  • Slav Petrov
  • Stefan Istrate
  • Terry Koo