Skip to content

Commit 2fbce3e

Browse files
Fix the loading error of jsonl file (#644)
* fixjsonl Signed-off-by: XuhuiRen <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: XuhuiRen <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
1 parent 48d4e53 commit 2fbce3e

File tree

1 file changed

+13
-1
lines changed

1 file changed

+13
-1
lines changed

comps/dataprep/utils.py

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -285,6 +285,16 @@ def load_json(json_path):
285285
return content_list
286286

287287

288+
def load_jsonl(jsonl_path):
289+
"""Load and process jsonl file."""
290+
content_list = []
291+
with open(jsonl_path, "r") as file:
292+
for line in file:
293+
json_obj = json.loads(line)
294+
content_list.append(json_obj)
295+
return content_list
296+
297+
288298
def load_yaml(yaml_path):
289299
"""Load and process yaml file."""
290300
with open(yaml_path, "r") as file:
@@ -351,8 +361,10 @@ def document_loader(doc_path):
351361
return load_md(doc_path)
352362
elif doc_path.endswith(".xml"):
353363
return load_xml(doc_path)
354-
elif doc_path.endswith(".json") or doc_path.endswith(".jsonl"):
364+
elif doc_path.endswith(".json"):
355365
return load_json(doc_path)
366+
elif doc_path.endswith(".jsonl"):
367+
return load_jsonl(doc_path)
356368
elif doc_path.endswith(".yaml"):
357369
return load_yaml(doc_path)
358370
elif doc_path.endswith(".xlsx") or doc_path.endswith(".xls"):

0 commit comments

Comments
 (0)