Whitespace/punctuation inconsistency

[tslater]

I noticed that there are some inconsistencies with how whitespace and punctuation are treated, and it causes some precision issues when trying to correlate with the original sentence. For example, a Japanese comma:、 is converted to a standard comma + space , or this combination: 。」 is converted to: . " (period space quote). I'm wondering fi there is a reason why punctuation is converted, and why spaces are added...and also if there is a way to preserve the information so that I could correlate perfectly each token index with the original sentence.

My use case is pretty common, generating the furigana for a sentence, but I want to know precisely the index in the original sentence. Another thing that might help this case is to include index locations for everything.

[tshatrov]

The punctuation substitutions are listed here: https://github.com/tshatrov/ichiran/blob/master/characters.lisp#L75

Because Japanese texts don't generally use spaces, I just automatically add spaces after relevant punctuations. It's a bit lazy, I guess.

Before each sentence is segmented, it is first split using this regex. The punctuation marks and other non-word characters are not segmented and treated separately. So suppose you have a mix of words and punctuation

>ichiran-cli.exe -f "一つ、２つ、３つ。"
[[[[["hitotsu",{"reading":"\u4E00\u3064 \u3010\u3072\u3068\u3064\u3011","text":"\u4E00\u3064","kana":"\u3072\u3068\u3064","score":200,"seq":1160820,"gloss":[{"pos":"[num]","gloss":"one"},{"pos":"[n]","gloss":"for one thing","info":"often used in itemized lists"},{"pos":"[n]","gloss":"only","info":"after a noun"},{"pos":"[n]","gloss":"(not) even","info":"with a verb in negative form"},{"pos":"[n-adv]","gloss":"just (e.g. \"just try it\")"},{"pos":"[adj-no]","gloss":"some kind of; one type of"}],"conj":[]},[]]],200]],", ",[[[["futatsu",{"reading":"2\u3064 \u3010\u3075\u305F\u3064\u3011","text":"2\u3064","kana":"\u3075\u305F\u3064","score":20,"counter":{"value":"Value: 2","ordinal":[]},"seq":2220330,"gloss":[{"pos":"[ctr]","gloss":"counter for the hito-futa-mi counting system (forming hitotsu, futatsu, mitsu, and misoji, yasoji, etc.)"}]},[]]],20]],", ",[[[["mittsu",{"reading":"3\u3064 \u3010\u307F\u3063\u3064\u3011","text":"3\u3064","kana":"\u307F\u3063\u3064","score":20,"counter":{"value":"Value: 3","ordinal":[]},"seq":2220330,"gloss":[{"pos":"[ctr]","gloss":"counter for the hito-futa-mi counting system (forming hitotsu, futatsu, mitsu, and misoji, yasoji, etc.)"}]},[]]],20]],". "]

This returns a list of 6 elements: a segmentation of 一つ, a string ", ", a segmentation of ２つ, a string ", " a segmentation of ３つ and a string ". ". This corresponds to the split of the input string using the abovementioned regex, which would also consist of 6 elements: "一つ", "、", "２つ", "、", "３つ", "。". So you would only apply furigana to the first, third and fifth element of the list, and the parts that had punctuation and got transformed don't need furigana anyway.

[tslater]

Very helpful to understand. Thanks for your response!

[molesquirrel]

Seeking clarification on something in the OP, is there a flag or plans to add one that would allow any non-word character(s) to be left intact including white space / new lines / etc?

[tshatrov]

I'm unable to actively work on the project right now, but I think this can be achieved by setting ichiran/characters::*punctuation-marks* to nil and removing non-kana characters from ichiran/characters::*abnormal-chars* and ichiran/characters::*normal-chars*

[molesquirrel]

I completely understand, and I appreciate the reply.

Edit: Doing the above worked like a charm!

[tshatrov]

Yes you would have to rebuild ichiran-cli after changing the source. вт, 5 апр. 2022 г., 23:32 molesquirrel ***@***.***>:

…

I completely understand, and I appreciate the reply. If we are using the cli, would we simply need to rebuild the cli after making any changes to the lisp files? — Reply to this email directly, view it on GitHub <#19 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA7DRHNLC7QLDPQFGYNTERTVDSWQFANCNFSM5LLQPHLA> . You are receiving this because you commented.Message ID: ***@***.***>

[fasiha]

Just ran into this as well. I had assumed that any string elements of the array Ichiran returned would be present in the text being parsed (that was the case for newlines or regular ASCII) but お姉ちゃん、かぜひいた？ (ran via docker exec ichiran-main-1 ichiran-cli -f "お姉ちゃん、かぜひいた？") surprised me because

1. 、 is mapped to two characters, , (regular-width comma, in contrast to the full-width original), and
2. ？ to ? (also regular-width)

Since the objects returned in the output array (the actual Ichiran analysis results) don't indicate which part of the input string they correspond to, and since the string elements of the output array can expand per the above thread, I have to be really careful in lining up which parts of the input correspond to each object: I search the input to find the text field, which works but feels fragile. So I guess two alternative solutions might be:

• return punctuation/irregular characters as-is, don't make them regular-width or add spaces, etc. (this is backwards incompatible 😢)
• Or include a number in each object that tells you which occurrence of text in the input this object corresponds to.

Hope this makes sense! As always, thanks for your hard work on Ichiran 🙇

[tshatrov]

you can achieve the same split by segments by calling the function basic-split. Then each part of romanize* output (-f option in ichiran-cli) corresponds to the same segment in basic-split:

ichiran-cli.exe -e "(ichiran/characters:basic-split \"お姉ちゃん、かぜひいた？\")"

((:WORD . "お姉ちゃん") (:MISC . "、") (:WORD . "かぜひいた") (:MISC . "？"))

You can also use :method :kana which does not transform the characters in the original string. Setting the romanization method is not yet supported with -f (I should probably add it), but you can do the same thing with -e

ichiran-cli.exe -e "(jsown:to-json (romanize* \"お姉ちゃん、かぜひいた？\" :limit 1 :method :kana))"

"[[[[[\"\\u304A\\u306D\\u3048\\u3061\\u3083\\u3093\",{\"reading\":\"\\u304A\\u59C9\\u3061\\u3083\\u3093 \\u3010\\u304A\\u306D\\u3048\\u3061\\u3083\\u3093\\u3011\",\"text\":\"\\u304A\\u59C9\\u3061\\u3083\\u3093\",\"kana\":\"\\u304A\\u306D\\u3048\\u3061\\u3083\\u3093\",\"score\":440,\"seq\":2222580,\"gloss\":[{\"pos\":\"[n]\",\"gloss\":\"older sister\"},{\"pos\":\"[n]\",\"gloss\":\"lass; young lady\"}],\"conj\":[]},[]]],440]],\"\\u3001\",[[[[\"\\u304B\\u305C\\u3072\\u3044\\u305F\",{\"reading\":\"\\u304B\\u305C\\u3072\\u3044\\u305F\",\"text\":\"\\u304B\\u305C\\u3072\\u3044\\u305F\",\"kana\":\"\\u304B\\u305C\\u3072\\u3044\\u305F\",\"score\":125,\"seq\":10069663,\"conj\":[{\"prop\":[{\"pos\":\"v5k\",\"type\":\"Past (~ta)\"}],\"reading\":\"\\u98A8\\u90AA\\u3072\\u304F \\u3010\\u304B\\u305C\\u3072\\u304F\\u3011\",\"gloss\":[{\"pos\":\"[v5k,exp]\",\"gloss\":\"to catch a cold\"}],\"readok\":true}]},[]]],125]],\"\\uFF1F\"]"