Japanese ChatGPT Limitations

Open Forum
1

I’ve experimented with asking for Japanese stories and essays and have come to the conclusion that ChatGPT does not always give you what you ask for.

In another thread, someone was elated that they asked ChatGPT to provide a story only using only the most common 1000 words in Japanese, and they got their story.

I would impress upon this story that just because ChatGPT gives an answer, it isn’t always per your instructions.

I asked for a story about going for a walk and finding a lost dog and I asked for it using the most common 2000 words, 1000 words, 800 words, and 500 words and got nearly EXACTLY the same results for each query. I also dabbled with asking for unique Kanji rather than vocabulary and got the same results.

Here is what I’ve discovered. ChatGPT will give you an easier story if you ask for it in Easy Japanese or Simple Japanese. If you just say “Japanese” the vocab will be a bit harder. If you ask for a story in “Difficult Japanese” it will be nearly identical to “Japanese”. It doesn’t raise the bar. Also, when I ask for “Very Easy Japanese” or “Very Simple Japanese”, the “very” doesn’t do anything. I also found no difference asking for the story in N5, N4, or N3 Japanese.

If I ask for “Simplish Japanese” which is a registered trademark for a language using the top 1000 vocab, it gives me an output but uses more advanced vocab than top 1000.

Bottom line, saying top “1000 Kanji” or “1000 Vocab” doesn’t work. “Easy Japanese” & “Simple Japanese” work. “Difficult Japanese” just spits out regular Japanese.

“Easy” and “Simple” Japanese gives an output that includes vocab and kanji more obscure than the top 1000. Trust me, I’m familar with more than 2000 Kanji but fuzzy on a lot of them but not the top 1000. If I see an output with Kanji I don’t immediately recognize, it’s not giving me top 1000 even though it might be easier than full-blown regular when just asking for a story in “Japanese”.

The only other parameter I’ve found helpful is asking for a 200 word story, 300 word story, 500 word story, etc. and it does that pretty well, although it doesn’t match exactly with LingQ’s counting which is no big deal.

I like choosing 300 or 500 words because it gives me a bit of vocabulary repetition and I ALWAYS read the story in ChatGPT without LingQ training wheels first. There’s no reason to mindlessly import everything. I like to give my brain an exercise with no help, no dictionaries, no LingQ. Then, if the story was interesting enough, and I had several unknown vocabs then I will import it into LingQ. If I pretty much nailed the whole story comprehension-wise, I don’t import it and I just ask for another story on a different subject.

2 Likes
2

I asked for a story about going for a walk and finding a lost dog and I asked for it using the most common 2000 words, 1000 words, 800 words, and 500 words and got nearly EXACTLY the same results for each query. I also dabbled with asking for unique Kanji rather than vocabulary and got the same results.

Did you do it in the same chat or you started a new one each time?

Could you make some snapshots or copy the prompts and answers of your chat sessions?

3

If you just say “Japanese” the vocab will be a bit harder. If you ask for a story in “Difficult Japanese” it will be nearly identical to “Japanese”. It doesn’t raise the bar.

This is what I would expect. For native speakers what is the difference between Difficult English and English, for example? I expect they are the same thing unless you get into something really literary or technical, which is not what you are likely going for. C1-C2 English is advanced (maybe difficult) for a learner, but for a native is just English.

I imagine it is also difficult for an AI to assess relative difficulty in general, as that depends on what your level is. If you give it more absolute terms, it does adjust. For example and for Japanese in particular, you could potentially ask for it to write an article in newspaper style, magazine style or simple style for elementary students, etc. Each of those can be evaluated independent of your level. I just tried it and it generates substantially different results.

1 Like
4

Newspaper Style:

※以下はフィクションであり、実際の出来事や人物とは関係ありません。
ロンドン、イギリス - 2023年3月17日
ブレグジット交渉が継続中、イギリスとEUに影響
イギリスが欧州連合を正式に離脱してから2年以上が経過したが、ブレグジットの影響は両者にとってまだ続いている。イギリスとEUの交渉は継続中で、貿易、移民、安全保障などに関する問題に焦点が当てられている。
最も論争の的となっている問題の1つは、アイルランド国境の問題である。イギリスとEUは、イギリス領北アイルランドとEU加盟国のアイルランド共和国の間にハードボーダーが設けられることを避けるために取り組んでいる。関税検査や規制の実施は、数十年にわたる地域の暴力を終わらせたグッドフライデー協定を脅かす可能性がある。
もう1つの重要な問題は、イギリスとEUの経済関係である。イギリスは、EUの単一市場や関税同盟のメンバーではなくなったため、企業にとって貿易の障壁や混乱が生じている。イギリスとEUは、今後も交渉を続ける予定である。

Simple Style for Elementary School Students:
イギリスとEUの間で、ブレグジットという問題が起こっています。2年以上が経ちましたが、まだ解決されていません。イギリスとEUは、貿易や移民、安全保障などについて交渉を続けています。
問題の一つは、アイルランド国境の問題です。イギリス領北アイルランドとアイルランド共和国の間にハードボーダーができることを避けるために、交渉をしています。これは、暴力があった過去を終わらせる大切な協定を脅かすことになります。
もう一つの問題は、イギリスとEUの経済関係です。イギリスがEUから離れたことで、企業にとって貿易の障壁や混乱が起こっています。今後も交渉を続ける予定です。

Most common 500 words:
イギリスとEUが話し合いを続けています。イギリスがEUを離れたことで、貿易や移民、安全保障などについて交渉しています。問題の一つは、アイルランド国境です。ハードボーダーができることを避けるために、交渉をしています。もう一つの問題は、イギリスとEUの経済関係です。企業にとって貿易の障壁や混乱が起こっています。

1 Like
5

Thanks. I’ve tried ChatGPT for writing corrections (in Japanese). I have yet to confirm how accurate it is though.

Side-note: The reason I prefer listening and reading podcasts is because transcripts reflect human speech. I’m more focused on output and reading this kind of material is helpful. The type of content ChatGPT spits leans towards ‘polite’ Japanese, which is fine, but not what I’m looking for.

2 Likes
6

It has provided useful but inconsistent feedback for me in Spanish. In one passage I asked it to correct, it said it found no errors. Then I pointed out an error and it said I was correct that I should have used the subjunctive. So it did not notice the error the first time.

I have asked at times to correct and point out phrases that could sound more natural stated another way. That has given me some useful pointers.

All in all, I’d say it is useful but doesn’t beat native content, and shouldn’t be overly relied upon.

1 Like
7

for @ericfromlingq

I tried this method to generate a casual Japanese version. This is a stiff topic for a casual dialogue, but it came out pretty well. I imagine you might be able to use this method to generate more casual speech.

Please generate a version of magazine level article in the form of a casual dialogue in Japanese between two people.
A: ねえ、最近ブレグジットについて何か聞いたことある？
B: ああ、イギリスがEUから離脱する問題だよね。
A: そうそう、もう2年以上経つんだけど、まだ話し合いが続いてるみたいなんだよね。
B: そうなんだ。最近どうなってるのかな。
A: アイルランド国境の問題が大きいんだって。イギリス領北アイルランドとアイルランド共和国の間にハードボーダーができることを避けるために、交渉をしてるんだって。
B: それは大変だね。グッドフライデー協定を脅かすことになりかねないから、慎重に考えないといけないよね。
A: そうそう、それにイギリスとEUの経済関係も影響を受けてるみたいなんだよ。企業にとって貿易の障壁や混乱が起こってるっていう話もあるんだよ。
B: そうなんだ。でも、今後も話し合いを続けていくんだろうね。
A: そうだね、もちろん。交渉が続いているみたいだから、これからも注目していこうよ。
1 Like
8

There is a trick. If you can find a list of 1000 most common words in Japanese online. You can create many short stories by inputting different 200 common words from the list. You can prompt" Write a story in Japanese by using these words separated by a comma". I am more than sure you will get a different story if you change the list of words every time. Sometimes it cuts off and does not finish the whole story. I prompt it to keep writing the story where you left off in the above story. It finishes the story to the end. There is also a problem with maximum tokens it can handle. I just finished creating many short stories of 100 most common Verbs in German. I inputted 10 different Verbs per story. I got 10 different stories in the end. Imported them on LingQ for creating LingQs and for intensive study where I can focus on the grammar structures.

2 Likes
9

OK this is just really fun:

Can you regenerate that same conversation in kansai dialect?
A: ねぇ、最近ブレグジットについて何か聞いたかい？
B: ああ、イギリスがEUから離脱する問題やろ？
A: そうそう、もう2年以上経っとるねんけど、まだ話し合いが続いとるみたいやねん。
B: ほんまや。最近どないなっとるんやろか？
A: アイルランド国境の問題が大きいみたいや。イギリス領北アイルランドとアイルランド共和国の間にハードボーダーができることを避けるために、交渉をしてるんやって。
B: それはやばいな。グッドフライデー協定を脅かすことになっちゃうから、慎重に考えななあかんよな。
A: そやで、それにイギリスとEUの経済関係も影響うけとるみたいやねん。企業にとって貿易の障壁や混乱が起こっとるっていう話もあるで。
B: そやなあ。でも、今後も話し合い続けていくんやろうねん。
A: そやで、もちろんや。交渉続いとるみたいやから、これからも注目していこうや。
10

It’s interesting to hear about your experiments with different language settings and parameters, and the results you’ve found.

It’s definitely important to keep in mind that ChatGPT is not perfect and may not always give you exactly what you ask for. However, it seems that asking for a story in “Easy Japanese” or “Simple Japanese” may be more effective in getting a story with simpler vocabulary and grammar. Asking for a story of a certain length also seems to be a useful parameter to consider.

I think your approach of reading the story in ChatGPT without any help first, and then using LingQ to reinforce vocabulary and comprehension, is a good one. It allows for a more active engagement with the language and helps to build reading skills.

Overall, it’s great to see that you are actively experimenting and finding ways to improve your language learning with ChatGPT. Thanks for sharing your insights with the community!

11

Some errors in there, but amusing. :slight_smile: