Skip to content

Conversation

@shlomi-noach
Copy link
Contributor

This PR applies DML events (events read from the binary logs) in batches, i.e. it aggregates events from the binary log and writes multiple DMLs (insert, delete, update) in a single transaction.

This has the potential to increase gh-ost's write capacity. This would potentially help #348, though we have not yet diagnosed #348 enough.

The PR introduces a --dml-batch-size flag, default 10.

Noteworthy about the implementation: gh-ost does not wait for events to arrive in order to batch. The logic is as follows:

  • If there's nothing in the event queue (coming from the binary log), copy some rows
  • If there's an event to handle in the events queue, read it. Further determine how many more available events are in the queue. Of these, read at most dml-batch-size. Of these, only handle the prefix that is true DML events (distinguished from gh-ost's internal events).

So the logic doesn't block on waiting for "more events to arrive", but is greedy, and takes as many available events as there are, up to some limit, and only in suh case that there are events to apply.

@shlomi-noach shlomi-noach deployed to production/ghost-mysql001 January 4, 2017 06:37 Active
@shlomi-noach shlomi-noach deployed to production/ghost-mysql001 January 4, 2017 06:44 Active
@shlomi-noach shlomi-noach deployed to production/ghost-mysql001 January 5, 2017 06:14 Active
@shlomi-noach
Copy link
Contributor Author

This was subjected to prod testing for the past days and behaves as expected; merging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants