Skip to content

Revert "Fix token/label offset issue" #23

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Mar 24, 2020
Merged

Revert "Fix token/label offset issue" #23

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 10 additions & 44 deletions deep_reference_parser/io/io.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,49 +13,18 @@

from ..logger import logger

def _unpack(tuples):
"""Convert list of tuples into the correct format:

From:

[
(
(token0, token1, token2, token3),
(label0, label1, label2, label3),
),
(
(token0, token1, token2),
(label0, label1, label2),
),
)

to:
]
(
(token0, token1, token2, token3),
(token0, token1, token2),
),
(
(label0, label1, label2, label3),
(label0, label1, label2),
),
]
"""
return list(zip(*list(tuples)))

def _split_list_by_linebreaks(rows):
def _split_list_by_linebreaks(tokens):
"""Cycle through a list of tokens (or labels) and split them into lists
based on the presence of Nones or more likely math.nan caused by converting
pd.DataFrame columns to lists.
"""
out = []
rows_gen = iter(rows)
tokens_gen = iter(tokens)
while True:
try:
row = next(rows_gen)
token = row[0]
token = next(tokens_gen)
if isinstance(token, str) and token:
out.append(row)
out.append(token)
else:
yield out
out = []
Expand All @@ -71,8 +40,10 @@ def load_tsv(filepath, split_char="\t"):
Expects data in the following format (tab separations).

References o o
o o
1 o o
. o o
o o
WHO title b-r
treatment title i-r
guidelines title i-r
Expand All @@ -84,6 +55,8 @@ def load_tsv(filepath, split_char="\t"):
, title i-r
2016 title i-r



Args:
filepath (str): Path to the data.
split_char(str): Character to be used to split each line of the
Expand All @@ -94,16 +67,9 @@ def load_tsv(filepath, split_char="\t"):
filepath.

"""
df = pd.read_csv(filepath, delimiter=split_char, header=None, skip_blank_lines=False)
tuples = _split_list_by_linebreaks(df.to_records(index=False))

# Remove leading empty lists if found

tuples = list(filter(None, tuples))

unpacked_tuples = list(map(_unpack, tuples))

out = _unpack(unpacked_tuples)
df = pd.read_csv(filepath, delimiter=split_char, header=None, skip_blank_lines=False)
out = [list(_split_list_by_linebreaks(column)) for _, column in df.iteritems()]

logger.info("Loaded %s training examples", len(out[0]))

Expand Down
Loading