Lost in Transliteration: Bridging the Script Gap in Neural IR

chapter 1

The question in one picture

Many languages are not written in Latin letters. Chinese and Russian are two of them. §1 People who speak them often type searches in Latin letters anyway. It saves switching keyboards on a phone. Some also use it to hide spelling mistakes. §1

The documents they search for are still written the usual way. §1 So the query and the documents use different letters, even though they are in the same language.

The picture shows the paper's own example. A Chinese query about peanut allergy finds documents about allergies. Written in Latin letters, the same query finds general documents about medicine. §4.3

The answer, in short

The search score of BGE-M3, a model for many languages, fell by 97% on Chinese queries written in Latin letters. §4.1 Chapter 2 explains the score. The paper calls such a drop the script gap. Abstract

The paper then trains the models on queries written both ways. On the kind of data it trained on, this nearly closes the gap for Russian and narrows it for Chinese. §4.2 On news articles from a different source, it helps much less for Chinese. §4.3

chapter 2

What you need first

1. Scripts and transliteration

A script is the set of written signs a language uses. Russian uses Cyrillic letters. Chinese uses characters. §1 Transliterated text is written with another script's letters, to keep its sound. The paper calls this transliteration. §1 The paper's example is Greek: Ελληνική Δημοκρατία becomes Helleniki Dimokratia. §1

The two languages in the paper sit at two ends. Chinese characters share nothing with Latin letters. Cyrillic letters overlap with Latin letters a great deal. §4 The usual script of a language is its native script. §1

2. Search in two steps

Information Retrieval is the field of search: finding the documents that answer what a person typed. That typed text is the query. §1 The paper's search runs in two steps. §4

The retriever is BGE-M3. It turns the query and each document into numbers and keeps the closest documents. The reranker is mT5. It reads the query with each kept document and orders them again. §2 §4

3. Two scores

The paper reports standard scores without defining them. §4 MRR@10 looks at where the first right document lands. It counts 1 if that document is first and a half if it is second. It counts 0 if none is in the top 10. The paper averages it over many queries.

R@1000 is the share of right documents found anywhere in the top 1000. For both scores, higher is better. The paper marks a result with † when its statistical test says the difference from the untrained model is unlikely to be chance. §4

chapter 3

What the paper does

The paper asks three questions. How much do the models lose? Can training close the gap? Does the fix still work on a different collection? §1

Training data in two scripts

Training data for search in a new language is scarce. A common fix is translate-train: machine-translate a large English data set into the new language. §3 MSMARCO is that English data set. Its translation into other languages is mMARCO. §2 §3

The paper adds one step. A program called Uroman rewrites each training query in Latin letters. The documents stay in their native script. The paper calls this transliterate-train. §3 §4

The paper uses fine-tuning: it trains each model a little more, in three ways. In the native configuration (N), all queries are in the native script. The mixed configuration (50) uses half of each. Last, the transliterated configuration (T) uses only queries in Latin letters. §4

Two test sets

The first test uses held-out queries from mMARCO, rewritten by Uroman. §4 The second test uses NeuCLIR, a collection of news articles with queries written for search tests in 2022 and 2023. §2 §4 The models never trained on it. This is the out-of-domain test. §1

chapter 4

What they found

Without training, the models lose much of their score

Table 1 from the paper Table 1 open full size

Top row: BGE-M3 before any extra training, scored with MRR@10. Chinese: 0.2342 in characters, 0.0078 in Latin letters.

Russian: 0.2444 in Cyrillic, 0.1244 in Latin letters.

Training on native script queries only barely moves the Chinese Latin-letter score: 0.0071, against 0.0078 before.

The half-and-half row: Latin-letter scores rise in both languages, and the native script scores stay above the top row.

Train on Latin letters only, and the score on Chinese characters drops. It ends up below the untrained model.

In the paper's tables, columns headed N hold queries in the native script. Columns headed T hold the same queries in Latin letters. Row labels reuse letters for the training mixes: N, 50 and T. §4

Look at the top row. For Chinese, the score fell from 0.2342 in characters to 0.0078 in Latin letters. That is the 97% drop. Table 1 §4.1

For Russian, the score fell from 0.2444 to 0.1244. That is a drop of 49%. Table 1 Even a script close to Latin lost about half the score. §4.1 The reranker showed the same pattern. §4.1

The half-and-half mix closes most of the gap

Training on native script queries alone raised the native script scores. It left the Chinese Latin-letter score at 0.0071, no better than before. Table 1 §4.2

Now look at the mixed configuration (50). The Chinese Latin-letter score rose from 0.0078 to 0.1382. Table 1 For Russian, the two scripts now scored about the same: 0.2770 and 0.2633. Table 1 §4.2 It was the only mix that kept native script scores and raised Latin-letter scores in both models. §4.2

The transliterated configuration (T) gave the best Latin-letter scores for BGE-M3, such as 0.1608 for Chinese. But its Chinese-character score fell to 0.2112, below the untrained model's 0.2342. Table 1 §4.2

Chinese stayed harder. Its Latin-letter score is still about half its character score. The authors suggest a reason: Cyrillic and Latin letters are alike, so Russian is easier to match. §4.2

On a different collection, the fix helps Russian more

Table 2 from the paper Table 2 open full size

Scores are nDCG@20. Chinese in Latin letters: 0.0028 before training, 0.0698 after. A small gain from a low start.

Russian in Latin letters: 0.2031 before, 0.3293 after.

The native script queries lose score in both languages after training.

The paper tests only the mixed configuration (50) on NeuCLIR, because only it kept the native script scores. §4.3 For Russian in Latin letters, the score rose from 0.2031 to 0.3293. Table 2 For Chinese, it rose only from 0.0028 to 0.0698. Table 2 Native script queries lost score in both languages. §4.3

The authors looked at ten Chinese queries by hand. §4.3 The peanut allergy query from the first picture is one of them. Another asked about Zelensky as a presidential candidate. Its Latin-letter form found general documents about Zelensky instead. §4.3

For most of the ten queries, the two forms shared three or fewer of their top ten documents. §4.3 The authors conclude that transliteration creates ambiguities, so queries lose their nuance. They also note that the news documents are more technical than the mMARCO documents the models trained on. §4.3

chapter 5

What this does not show

These limits come from the paper's own text. §4 §5

• Word breaks are untested. Chinese has no spaces between words. Splitting queries into words before rewriting helped BGE-M3 in one test and not mT5. The paper leaves this for future work. §4.1 §4.2
• Chinese on new data. The fix did not carry over for Chinese on the news collection. §4.3
• How to rewrite. The authors say more research is needed on how queries should be rewritten for search. §5

The design also limits what the results cover. The tests use two languages, one retriever and one reranker. §4 Every Latin-letter query was written by Uroman, not typed by a person. §4

chapter 6

Words to know

Hover or tap any underlined word on the page to see its entry. These are the terms this page uses, in the order it introduces them.

query

What a person types into a search box.

"the purpose of a search system is to satisfy a user’s information need expressed in a query" (§1)

native script

The script a language is normally written in, such as Cyrillic for Russian.

"Documents are commonly written in the native script of a language, whereas it can be easier for humans to express their information needs in a transliterated query." (§1)

script gap

How much worse search works when the same query is typed in Latin letters instead of its usual script.

"This creates a “script gap” between the performance of the same queries when written in their native or transliterated form." (Abstract)

script

The set of written signs a language uses: Latin letters, Cyrillic letters, Chinese characters.

"Human languages use a multitude of different scripts." (§1)

transliteration

Writing the sounds of a word with the letters of another script, such as Greek written in Latin letters.

"Transliteration is the process of converting text from one script to another. This is done to express the pronunciation of the original script in a different one whilst approximating the original pronunciation." (§1)

also called in the paper: romanization, Latinized, Romanized

Information Retrieval

Search: finding the documents in a large collection that answer what a person typed.

"In Information Retrieval (IR), the purpose of a search system is to satisfy a user’s information need expressed in a query by typically retrieving relevant documents from a collection." (§1)

retriever

The first search step: it picks candidate documents from the whole collection.

"we focus on the single-vector representation component of BGE-M3 [5] as the retriever and mT5 [29] as the reranker." (§4)

reranker

The second search step: it reads the query with each candidate and puts the candidates in a new order.

"we focus on the single-vector representation component of BGE-M3 [5] as the retriever and mT5 [29] as the reranker." (§4)

MRR@10

A standard score: 1 if the first relevant document is ranked first, 1/2 if second, and so on; 0 if none is in the top 10. Averaged over queries.

"We report MRR@10 and R@1000 for all experiments with mMARCO and nDCG@20 and R@1000 for neuCLIR." (§4 (used, not defined in the paper))

R@1000

A standard score: the share of the relevant documents that appear anywhere in the top 1000.

"We report MRR@10 and R@1000 for all experiments with mMARCO and nDCG@20 and R@1000 for neuCLIR." (§4 (used, not defined in the paper))

nDCG@20

A standard score for the top 20: relevant documents count more the higher they are ranked. 1 is a perfect order.

"We report MRR@10 and R@1000 for all experiments with mMARCO and nDCG@20 and R@1000 for neuCLIR." (§4 (used, not defined in the paper))

significance testing

A statistical check of whether a difference is larger than chance would explain.

"We perform significance testing with Bonferroni correction between each model configuration and their respective baselines, “BGE-M3” and “mT5” for the same query set." (§4)

translate-train

Making training data for a new language by machine-translating a large English data set.

"In “translate-train”, a large dataset of good-quality relevance judgements, such as MSMARCO [21], is modified by applying automatic machine translation, resulting in a large dataset for other languages, e.g., the mMARCO collection [3]." (§3)

MSMARCO

A large English search data set: real queries, passages, and which passages answer each query.

"The mMARCO [3] collection is a popular resource containing automatically translated versions of MSMARCO [21] in more than ten languages." (§2, §3)

mMARCO

MSMARCO machine-translated into other languages. The paper uses the Chinese and Russian versions.

"The mMARCO [3] collection is a popular resource containing automatically translated versions of MSMARCO [21] in more than ten languages." (§2, §4)

Uroman

A program that rewrites text from many scripts in Latin letters.

"Using Uroman [9], a tool commonly used to romanize non-Latin scripts, we produce romanized versions of both the ‘dev/small‘ and the ‘train‘ query sets for both languages." (§4)

transliterate-train

Translate-train plus one step: the training queries are also rewritten in Latin letters, while the documents stay in their usual script.

"For our task, we add a further step and produce automatic transliterations (Romanizations) of the training queries to leverage the strengths of these datasets and close the script gap between the native and transliterated queries. We denote this approach as “transliterate-train”." (§3)

fine-tuning

Training a model that is already trained a little more, on examples for one job.

"We show that by further fine-tuning IR models on an even mixture of native and Latinized text, they can perform this cross-script matching at nearly the same performance as when the query was formulated in the native script." (Abstract)

native configuration (N)

Training with queries in the usual script only.

"For our first configuration (N), we fine-tune the neural rankers only on the native-text queries." (§4)

mixed configuration (50)

Training with half the queries in the usual script and half in Latin letters.

"For our mixed configuration (50), we fine-tuned on an even split of transliterated and native text queries." (§4)

transliterated configuration (T)

Training with queries in Latin letters only.

"Finally, for our third configuration (T), we fine-tune the neural rankers only on the transliterated queries." (§4)

NeuCLIR

A collection of news articles in Chinese, Persian and Russian, with search queries from the TREC 2022 and 2023 tests.

"The TREC NeuCLIR track [11, 12] released NeuCLIR/1, a collection of Persian, Chinese and Russian Common Crawl News documents as well as neuMARCO, a cross-language automatically translated version of MSMARCO." (§2, §4)

out-of-domain

Tested on a different kind of collection from the one the model was trained on.

"Finally, we evaluate our models in an out-of-domain setting using the respective TREC NeuCLIR datasets for Russian and Chinese (RQ3)." (§1, §4.3)

BGE-M3

A search model that works in many languages. It turns a query and each document into numbers and compares them.

"Recent work, such as Chen et al. [5], leveraged XLM-R to produce a multilingual embedding model called BGE-M3." (§2, §4)

mT5

A language model for many languages. Here it reads a query and a document together and scores the match.

"mT5 [29], a multilingual variation of T5 [24], pre-trained on a dataset covering 101 languages has been later used by Bonifacio et al. [3] for retrieval by fine-tuning mT5 on mMARCO." (§2, §4)