Skip to content

Make batch importer work in local development environments #7236

@jimchamp

Description

@jimchamp

Our batch importer currently does not run in local development environments. This makes it difficult to test changes to the importer, and impossible for volunteer contributors to make meaningful improvements to the importer.

Describe the problem that you'd like solved

Proposal & Constraints

Modify local environments such that the Open Library importer can be run. The following items must be completed in order for this to happen:

  • Create import_item and import_batch tables in local DB
  • Replace /scripts/dev-instance/dev_db.pg_dump with new pg_dump file
  • Generate dummy data for the importer and populate tables
  • The importer script can be started and stopped as needed by a developer
  • Any new, valid data in the import_batch table is added to the catalog by the importer

Additional context

DDL for new tables: https://github.com/internetarchive/openlibrary/blob/master/openlibrary/core/schema.py#L73-L100

Stakeholders

Metadata

Metadata

Assignees

Labels

Lead: @jimchampIssues overseen by Jim (Front-end Lead, BookNotes) [managed]Module: ImportIssues related to the configuration or use of importbot and other bulk import systems. [managed]Priority: 4An issue, but should be worked on when no other pressing work can be done. [managed]Type: Feature RequestIssue describes a feature or enhancement we'd like to implement. [managed]

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions