Does MySql full text search works reasonably with non-Latin languages (Hebrew, Arabic, Japanese...)

Question

Does MySql full text search works reasonably with non-Latin languages? (Hebrew, Arabic, Japanese...)

Addition: Did some tests... It has some problems with Hebrew. Example: The name מוסינזון is pronounced the same as מושינזון but searching one won't find the other, as this is a common spelling error in Hebrew, it seems I will have to do some data manipulation for it to work perfectly.

Yeah, that's not going to catch Hebrew semantics. If it were set to a Hebrew collation, my guess is that it would. — Eric, Aug 31 '09 at 11:55
There is no Hebrew collation for utf8. I guess we do it the hard way. — Itay Moav -Malimovka, Aug 31 '09 at 14:35

score 4 · Answer 1 · edited Nov 21 '11 at 15:14

Though Hebrew support in MySQL is limited, your problem is more a problem of people using incorrect spelling, then a dysfunction of the MySQL server in this perspective. When you misspell a word in Google, it will show you a suggestion and you can click on that suggestion to search for that term.

Perhaps you could build some program that has the same behaviour, e.g. you could create a table that has 2 fields, one containing the commonly misspelled word and the other containing the correct spelling. You could then build a program that finds the misspelled word and displays the suggestion.

The usual way to implement this is using n-grams – Ami Malimovka Aug 27 '13 at 17:00 — Ami Malimovka, Aug 27 '13 at 17:00

score 2 · Answer 2 · answered Aug 30 '09 at 16:11

2

So long as your collation is set properly, it works splendidly.

Unicode will work for most of this, of course. But that doesn't really translate Latin characters to them very well (for example, in a Dutch collation aa will be recognized as å).

answered Aug 30 '09 at 16:11

Eric

92,005
12
114
115

score 2 · Answer 3 · answered Sep 02 '13 at 10:09

Yes MySQL fulltext search works well for Arabic. Just make sure of the following where needed:

COLLATION = utf8_unicode_ci & CHARACTER SET = utf8. (Databases, Tables, and Columns).
Index words of 3 letters and more. This is Very Important for Arabic, ft_min_word_len = 3 (see show variables like "ft_%";)
Check the version of MySQL (5.5 or 5.6), and Engine (InnoDb or MyIsam)

score 1 · Answer 4 · answered Aug 30 '09 at 16:27

1

Yes, however, check out what stopwords are.

answered Aug 30 '09 at 16:27

Quassnoi · Answer 5 · 2009-08-31T17:21:55.020

Japanese and Chinese use their own whitespace symbols that MySQL does not understand.

Make sure that the words in the texts you are going to index are separated with ASCII separators (spaces, commas etc). Anything outside the ASCII range will probably not work.

Besides, you'll probably need to fix ft_min_word_len: by default, MySQL won't index words shorter than 4 characters, and most Japanese and Chinese words.

In Cyrillic languages transilteration errors are quite common.

All letters from this sequence: АВЕКМНОРСТуХ / ABEKMHOPCTyX are indistinguishable in most fonts.

The worst of them is Cyrillic С/ Latin C: both these symbols are located on one key on the keyboard and do not differ at all in most fonts, but they have different codes.

MySQL will not catch it either.

Does MySql full text search works reasonably with non-Latin languages (Hebrew, Arabic, Japanese...)

5 Answers5

Linked