[Feature][Customize] Add Support for Incremental CSV Upload in the Customize Plugin

[narrowizard]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Use case

As a DevLake user leveraging the Customize plugin to upload issues and issue_repo_commits data for further analysis, I need the ability to perform incremental CSV uploads. This would allow me to append new data to existing records without overwriting or replacing the entire dataset.

Description

Currently, the Customize plugin in DevLake only supports full data uploads, which replace all existing data with the new data from the uploaded CSV file. While this functionality works for initial data loads, it poses significant challenges as the dataset grows over time:

1. Data Integrity Risks: Full uploads may inadvertently overwrite or lose historical data, compromising the dataset's accuracy and completeness.
2. File Maintenance Overhead: CSV files become increasingly large as time progresses, making them cumbersome to maintain and manage.
   To address these challenges, I propose adding incremental upload support to the Customize plugin. This feature would enable users to append new records from CSV files to the existing dataset without requiring a complete overwrite.

Benefits:

• Enhanced Data Integrity: Ensures existing data remains untouched while appending new entries.
• Improved Scalability: Reduces the need to maintain and manage increasingly large CSV files.
• Better User Experience: Simplifies data upload workflows for users.
  I envision this feature functioning as follows:

1. Users upload a new CSV file containing only new data entries.
2. The Customize plugin compares the uploaded data with existing records.
3. New records are appended to the domain layer, while existing records remain unchanged.
   This functionality would greatly improve the usability of the Customize plugin and make it more suitable for long-term data collection and analysis workflows.

Let me know if additional details or clarifications are needed!

Related issues

No

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[narrowizard]

Plan

The implementation involves two APIs for uploading issues.csv and issue_repo_commits.csv data, respectively.

For issues.csv:

• Parameters: boardId, boardName, file
• Existing Logic:
  1. Update data in the boards table based on boardId.
  2. Clear data in the issues and issue_labels tables (clearing logic: remove records associated only with the given boardId).
  3. Clear data in the board_issues table.
  4. Create or update records in the issue_labels table based on the uploaded file (primary key: issue_id, label_name).
  5. Create or update records in the board_issues table based on the uploaded file (primary key: board_id, issue_id).
  6. Create new records in the issues table based on the uploaded file.
• Incremental Update Logic:
  1. Add a new parameter: incremental = false.
  2. When incremental = true:
  • Skip the data-clearing steps from the existing logic.
  • Modify the issues table operation to CreateOrUpdate (primary key: id).
  • Retain all other existing logic.

For issue_repo_commits.csv:

• Parameters: boardId, file
• Existing Logic:
  1. Clear data in the issue_repo_commits and issue_commits tables (clearing logic: remove records associated only with the given boardId).
  2. Create new records in the issue_repo_commits and issue_commits tables based on the uploaded file.
• Incremental Update Logic:
  1. Add a new parameter: incremental = false.
  2. When incremental = true:
  • Skip the data-clearing steps from the existing logic.
  • Modify the issue_repo_commits table operation to CreateOrUpdate (primary key: issue_id, repo_url, commit_sha).

This plan ensures support for incremental updates while retaining the full upload functionality when needed.

@dosubot Pls review the plan above.

[klesh]

Excellent, It is very descriptive. Sounds great to me
Looking forward to your PR.