Fixing Bijoy to Unicode: 17,000 Broken Bangla Words

Why offSchool needed this text

offSchool teaches the class 9 and 10 (SSC) science syllabus of Bangladesh, and every lesson has to match what the National Curriculum and Textbook Board (NCTB) asks for. The board's curriculum documents list each chapter's learning outcomes (শিখনফল) beside its content (বিষয়বস্তু). offSchool extracts them to check its 683 topics against the syllabus and to guide how each chapter is written.

The PDFs have a real text layer, which looks like good news until you read it. The Bangla in them was typed in SutonnyMJ, a Bijoy font. Bijoy predates Unicode Bangla: the file stores the codes of an English keyboard layout, and the font draws Bangla shapes over them. Copy the subject name পদার্থবিজ্ঞান out of one of these PDFs and you get c`v_©weÁvb.

How the conversion works

The extractor reads every text span with PyMuPDF and checks its font. Spans in SutonnyMJ, KalindiMJ or ChandrabatiMJ go through bijoy2unicode. Spans in Arial, Times or Vrinda are already Unicode and pass through untouched, which matters because English terms sit in the middle of Bangla sentences. Symbol-font spans (Greek letters, arrows, ≤ and ≥) go through a small table of their own.

The first run produced readable text with a steady rash of wrong words. Counted across the curriculum text, the SSC and HSC chapter files and the normalized syllabus, there were about 17,000 of them. They were not random. Almost all came from four causes.

Why the reph breaks things

In Bijoy, a vowel sign that is drawn on the left, such as ি or ে, is typed before its consonant, the way it looks. Unicode stores it after the consonant, the way it is said. So a converter has to move each pre-kar across exactly one consonant cluster. The reph, a র্ written as a hook above the next letter, is typed after its cluster in Bijoy too. When the library met a reph followed by a vowel sign and then the next syllable's pre-kar, it lost track of where one syllable ended, and the ে of দের ended up after the wrong letter.

The fix is a regular expression on the Bijoy text before conversion: end the piece after the reph code © and any trailing post-kars, when a pre-kar comes next. Each piece then converts correctly on its own, and a second pass moves the reph to the front of its cluster, where Unicode wants it.

The source has typos too

Not every wrong word is the converter's fault. The PDFs carry the typesetter's own slips: a doubled vowel sign, হ্নদপিণ্ড for হৃদপিণ্ড, প্লাকং for প্লাঙ্ক in Max Planck's name, and one ণ্ড typed as a dash in লঞ্চ বা স্টিমার. No glyph rule can fix a word that was typed wrong, so these live in a short word list of three dozen entries, matched after NFC normalization.

Some printed spellings were kept on purpose. Loanwords such as সিলিণ্ডার use a separate ন্ড glyph in the font, and দূর্যোগ is how the document prints it. Each of those calls went to three independent checks and followed the majority. A converter that silently "corrects" the source is no longer a faithful copy of it.

Normalization is part of the fix

Unicode allows more than one byte sequence for some Bangla letters. ো can be stored as ে followed by া, or as the single code point U+09CB, and য় can be one code point or য plus a nukta. Text that looks identical can therefore fail to match. Every comparison here runs on NFC, the converter composes ে plus া into ো, and the word list is keyed on NFC so a fix cannot miss its target because of an invisible difference.

A validator so it cannot come back

The last step was the one that matters a year from now. offSchool's content check, which runs on every change to the curriculum, simulations and practice questions, now rejects any Bangla word that a broken conversion would produce:

• a word that starts with a vowel sign;
• two vowel signs in a row;
• a vowel sign straight after a hasanta (্);
• two hasantas in a row;
• স্ন anywhere but the start of a word, which is the garbled ্ল.

None of these occurs in correctly typed Bangla, so the check has no false alarms to tune away. Before the fix, the existing content failed it 76 times. One implementation detail is worth knowing: a JavaScript regular-expression character class that mixes consonants and vowel signs trips ESLint's no-misleading-character-class rule, so the patterns are built as alternations with new RegExp.

What students saw

Less than the numbers suggest. Most of the garbling sat in source and reference files that students never read. In the syllabus data the app actually serves, 78 strings changed, nearly all of them chapter guidance used only behind the scenes, and two learning outcomes got their missing spaces back. No chapter or topic title changed. The lesson is about where the damage would have gone next: every new note, question and search index built from that text would have inherited it.

If you are converting Bijoy PDFs yourself

Check which font each span uses and convert only the Bijoy ones. Find out which version of the font your documents use, because older SutonnyMJ assigns some codes differently from what converters expect. Split at the reph before converting, convert touching Bangla spans together, normalize to NFC, and finish with a garble check like the one above. Then read a sample of the output yourself, because a pattern check cannot tell you that a correctly formed word is the wrong word.