Feature/use huggingface compatible pretokenizer

[t-yamamura]

No description provided.

[eiennohito]

Also please add tests for correct token mapping logic (if I am wording that correctly).

[eiennohito]

zip is incorrect when `len(morphemes) != len(normalized_strings)

[eiennohito]

This must be a single loop, not two loops.

[eiennohito]

m.surface() != '' can also be written as len(m) != 0, but in this case there are already m.begin() and m.end() calls.

So it can be done something like this:

result = []
for m in morphemes:
  b = m.begin()
  e = m.end()
  if b == e: # empty token
    continue
  token = original[b:e]
  if _word_formatter is not None:
    token.replace(token.normalized, _word_formatter(m))
  result.append(token)
return result

Note that this code tries to avoid unnecessary computations. PreTokenizer handler is a performance-critical component and unnecessary calls should be avoided here

[t-yamamura]

I agree with your proposal from a performance of view. 64df1cd

My only concern is that there is the if statement in each iteration.
To avoid this, we could use an if statement before the loop, but I think that would be redundant (even though the performance is almost the same).
What do you think?

[eiennohito]

The best way would be probably to emit different handler functions depending on the condition. That probably can be overkill (or you can try and measure, that can be a nice learning experience as well).
My comments on the performance were more related to calling m.surface(), which creates a string object, allocating memory, for each call.
Comparison to None should be pretty cheap, but you can measure it.

[eiennohito]

Also, your previous version was incorrect in case of mismatching lengths of MorphemeList and tokens so creating a token and calling replace should share a single condition anyway and can't be easily written in two loops.

[t-yamamura]

Also please add tests for correct token mapping logic (if I am wording that correctly).

I added these tests (66afb37), except for the ones that can't be tested now due to (#42).