feat(pipeline): Data Pipeline for Safaa main PR#22

Open

smilingprogrammer wants to merge 21 commits intofossology:mainfrom

smilingprogrammer:copyright-pipeline

Commits on Jul 9, 2025

feat(script): Integrated script to preprocess fetched copyright content, and added it into the pipeline
smilingprogrammer
committed
feat(script): add script to fetch copyright from server
smilingprogrammer
committed
feat(script): Integrated script to declutter fetched copyright content, and added it into the pipeline, I also included an upgraded declutter script with regex
smilingprogrammer
committed

Commits on Jul 12, 2025

feat(script): Resolve corrections in script for fetching of content from the fossology serv
smilingprogrammer
committed

Commits on Jul 13, 2025

feat(script): added different files into one single script with their various functionalities
smilingprogrammer
committed

Commits on Jul 14, 2025

feat(script): Resolve corrections in pipeline.yml scrit
smilingprogrammer
committed

Commits on Jul 21, 2025

feat(script): changed pipeline scripts location, and renamed folder to 'retraining'
smilingprogrammer
committed

Commits on Jul 22, 2025

Commits on Jul 28, 2025

feat(script): declared global paths, implements argument, and direct dependency installation
smilingprogrammer
committed

Commits on Jul 29, 2025

Commits on Aug 4, 2025

feat(script): updated sql script to leave out ignored contents when fetching
smilingprogrammer
committed

Commits on Aug 6, 2025

Commits on Aug 8, 2025

feat(script): fixed preprocessing and decluttering to maintain dataframe alignment
smilingprogrammer
committed

Commits on Aug 10, 2025

feat(script): added data for testing pipeline
smilingprogrammer
committed

Commits on Aug 16, 2025

feat(script): updated pipeline scripts to automate creation of PR
smilingprogrammer
committed

Commits on Aug 25, 2025

feat(docs): included a README.md for the folder
smilingprogrammer
committed