Technical Exercise Product Engineer - Adhitya Bagus Putra Erlangga

[adhityabpe]

Create Pull Req

[adhityabpe]

Benchmark Comparison

Before Optimization:
• Memory Usage: 626.00 MiB
• Execution Time: 674666 ms

After Optimization:
• Memory Usage: 24.00 MiB
• Execution Time: 881116 ms

After Additional Optimization (Caching):
• Memory Usage: 24.00 MiB
• Execution Time: 2130294 ms

After Adjusting Chunk Size:
• Memory Usage: 26.00 MiB
• Execution Time: 3442164 ms

Thinking Process and Optimization Steps

Step 1: Analyzing the Current Implementation
• Initially, the DropOutEnrollments command was loading all fields for enrollments that met the criteria, leading to high memory usage.
• Sequential processing of enrollments resulted in high memory consumption and longer processing time.
• The transaction was started but not committed unless an error occurred, leading to lack of atomicity and data consistency.

Step 2: Implementing Selective Data Loading
• Modified the code to load only the necessary fields (id, course_id, student_id) to reduce memory usage.

Step 3: Batch Processing with chunkById
• Changed the processing to use chunkById with a chunk size of 1000 to reduce memory usage and improve performance by processing smaller batches.

Step 4: Efficient Queries
• Combined multiple conditions into a single query to minimize database operations and improve query performance.

Step 5: Transactional Operations
• Wrapped updates in a database transaction to ensure atomicity and consistency, committing or rolling back as needed.

Step 6: Adjusting Chunk Size
• Adjusted the chunk size to 2500 to allow processing larger batches of enrollments at once, potentially reducing execution time.

Why These Optimizations Are Important:

• Selective Data Loading: Reduces memory usage by loading only necessary fields, minimizing memory overhead.
• Batch Processing with chunkById: Prevents memory exhaustion and allows for faster processing by managing memory usage effectively.
• Efficient Queries: Reduces the number of database calls, speeding up query execution.
• Transactional Operations: Ensures that all updates are applied together or not at all, maintaining data integrity and consistency.
• Adjusting Memory Limit and Chunk Size: Allows for processing larger batches of enrollments at once, reducing the number of chunks and iterations, leading to potentially shorter execution time.
How These Optimizations Were Achieved:
• Analyzed the current implementation to identify bottlenecks and inefficiencies.
• Implemented selective data loading, batch processing, and efficient queries to reduce memory usage and improve performance.
• Used database transactions to ensure atomicity and data consistency.
• Adjusted the memory limit and chunk size to balance memory usage and execution time.

I have chosen to use the first optimization without adding caching and adjusted the chunk size because it proved to be faster and used less memory.