feat: Implement LocalManager::recover

[evenyag]

I hereby agree to the terms of the GreptimeDB CLA

What's changed and what's your intention?

This PR implements LocalManager::recover that recover uncommitted procedures from ProcedureStore and re-submits them to execute.

Recovery

• LocalManager loads messages of all uncommitted procedures from ProcedureStore and caches in memory.
• Instead of recovering each subprocedures, LocalManager just submit the root procedure to execute.
• Because each step of the procedure is idempotent, the root procedure should submits its subprocedures again.
• Then the manager can load the subprocedure from the cached message by the procedure id.
• Once the root procedure is finished, we can remove the cached message from the manager.

This simplify the recover process a lot as we don't need to reconstruct the whole DAG.

Checklist

• I have written the necessary rustdoc comments.
• I have added the necessary unit tests and integration tests.

Refer to a related PR or issue link (optional)

• Tracking issue for the procedure framework #286

[codecov]

Codecov Report

Merging #981 (29952d4) into develop (c1a9f84) will decrease coverage by 0.09%.
The diff coverage is 90.65%.

@@             Coverage Diff             @@
##           develop     #981      +/-   ##
===========================================
- Coverage    85.74%   85.65%   -0.09%     
===========================================
  Files          461      461              
  Lines        62932    62982      +50     
===========================================
- Hits         53962    53949      -13     
- Misses        8970     9033      +63     

Flag	Coverage Δ
rust	85.65% <90.65%> (-0.09%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
src/common/procedure/src/local.rs	91.68% <89.36%> (+5.65%)	⬆️
src/common/procedure/src/local/runner.rs	82.30% <100.00%> (+0.02%)	⬆️
src/common/procedure/src/store.rs	96.35% <100.00%> (+0.04%)	⬆️
src/object-store/src/test_util.rs	0.00% <0.00%> (-100.00%)	⬇️
src/datanode/src/instance.rs	41.56% <0.00%> (-14.91%)	⬇️
tests-integration/src/test_util.rs	79.47% <0.00%> (-9.33%)	⬇️
src/servers/src/error.rs	37.93% <0.00%> (-6.90%)	⬇️
src/servers/src/grpc.rs	86.95% <0.00%> (-4.35%)	⬇️
src/servers/src/opentsdb.rs	96.82% <0.00%> (-1.59%)	⬇️
src/meta-srv/src/service/heartbeat.rs	86.17% <0.00%> (-1.07%)	⬇️
... and 4 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[fengjiachun]

lgtm