Support comparing line GT directories with line OCR directories

[mikegerber]

In #62, @stweil's original problem was - as I understand it - to compare a directory with line GT text files with a directory of line OCR text files. For now I've created fake test data to implement this fake-line-gt.zip. It looks like this:

% ls *
gt:
line001.gt.txt  line003.gt.txt  line005.gt.txt  line007.gt.txt  line009.gt.txt  line011.gt.txt
line002.gt.txt  line004.gt.txt  line006.gt.txt  line008.gt.txt  line010.gt.txt

some-ocr:
line001.some-ocr.txt  line003.some-ocr.txt  line005.some-ocr.txt  line007.some-ocr.txt  line009.some-ocr.txt  line011.some-ocr.txt
line002.some-ocr.txt  line004.some-ocr.txt  line006.some-ocr.txt  line008.some-ocr.txt  line010.some-ocr.txt

A first implementation should compare the text of pairs files (matching by filename) and produce a report of metrics & differences over all of the lines. First idea of the CLI interface:

dinglehopper-lines gt/ --gt-suffix .gt.txt some-ocr/ --ocr-suffix .some-ocr.txt

I'm not sure if this will be the final CLI interface but it's what seems necessary on first glance.

[stweil]

What about an even simpler interface:

dinglehopper [OPTIONS] GTDIR OCRDIR [REPORT_PREFIX]

The existing dinglehopper could be extended to accept directory names for its GT and OCR argument and then either strip all extensions when matching ground truth and ocr lines by default or use new optional --gt-suffix and --ocr-suffix options.

[mikegerber]

What about an even simpler interface:

dinglehopper [OPTIONS] GTDIR OCRDIR [REPORT_PREFIX]

The existing dinglehopper could be extended to accept directory names for its GT and OCR argument

For now and until the interface is finalized I'd like to keep the CLI interface separate, it will share the code anyway.

and then either strip all extensions when matching ground truth and ocr lines by default or use new optional --gt-suffix and --ocr-suffix options.

For the stripping of all extensions to work we would need to assume that the common prefix for a pair does not contain a dot, and the explicit suffix options seemed saner.

But I think I'll start implementing this, CLI details can still be refined later.

[mikegerber]

For the stripping of all extensions to work we would need to assume that the common prefix for a pair does not contain a dot, and the explicit suffix options seemed saner.

They will default to something useful: the longest common suffix, i.e.

import itertools


def all_equal(iterable):
    g = itertools.groupby(iterable)
    return next(g, True) and not next(g, False)


def common_prefix(its):
    return [p[0] for p in itertools.takewhile(all_equal, zip(*its))]


def common_suffix(its):
    return reversed(common_prefix(reversed(it) for it in its))


#print("".join(common_prefix(["line001.gt.txt", "line02.gt.txt", "line3.gt.txt"])))
print("".join(common_suffix(["line001.gt.txt", "line02.gt.txt", "line3.gt.txt"])))

(gives .gt.txt)

[mikegerber]

dinglehopper-line-dirs gt some-ocr from the feat/compare-line-texts branch now compares the line texts from the gt and some-ocr. It auto-detects the file suffixes. It's WIP - but only WER and word differences are missing.

@stweil Could you test if this works for you?

[mikegerber]

The lines also line up perfectly, because each pair is put into its own <div class="row">!

[stweil]

My first test fails:

dinglehopper-line-dirs gt frak2021_1.069 frak2021_1.069
free(): invalid next size (fast)
Aborted

[stweil]

The crash happens in rapidfuzz-1.9.0-py3.9-linux-x86_64.egg/rapidfuzz/cpp_string_metric.cpython-39-x86_64-linux-gnu.so.

[stweil]

@maxbachmann, I now tried to debug the RapidFuzz code, but pip install . fails:

 src/cpp_common.hpp:4:10: fatal error: rapidfuzz/fuzz.hpp: No such file or directory

[mikegerber]

I can't reproduce with Python 3.9 and rapidfuzz-1.9.0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl. Hmm. Could we have a look at the data (or a portion of it) that triggers this?

[maxbachmann]

@stweil did you clone the repository including submodules?

git clone --recursive git@github.com:maxbachmann/RapidFuzz.git

As @mikegerber mentioned it would help if you could provide me with some data to reproduce this.

[stweil]

Minimal single line test case (found by bisecting the original large test set):

mkdir a b
echo "Vorjahres.“ (24 % gegenüber 42 %. Daneben auch Anſtiege um 11 %, 22 %, 34 %," >a/demo.txt
echo "PVorſahres.“ (24 0% gegenüber 42 95, Daneben auch Anſtiege um 11 % 22 % 34" >b/demo.txt
dinglehopper-line-dirs a b c

[stweil]

did you clone the repository including submodules?

No, I did not. The installation works after git submodule update --init. I suggest to add that information to the instructions in the README.

[maxbachmann]

Minimal single line test case (found by bisecting the original large test set):

thanks I could reproduce the crash. I will look into it