feat: create Jupyter notebook pipeline file based on main.py file

[pwang347]

Description

Add support for a NotebookConverter class associated with the Pipeline coder, which is enabled with the DS_ENABLE_NOTEBOOK_CONVERSION config. When the flag is enabled, additional prompt is passed to the Pipeline coder to add print statements in the main() function section which is used to properly split the .py file into .ipynb and re-attach outputs.

Motivation and Context

.ipynb files can be useful for debugging after the agent has completed its execution loops. With this change we can generate both useful .py files and a reasonably formatted .ipynb file at the same time.

Known Limitations

• This currently relies on the fact that output stderr and stdout are approximately ordered using

  RD-Agent/rdagent/utils/env.py

  Line 550 in 512d08f

  if self.conf.live_output:

  to reattach the outputs in sequence
• All output is treated as stdout in the notebook, though this will result in different appearance to the actual run in notebook
• We assume for now it is fine that the execution environment is not Jupyter, but this may be a follow-up item in the future

How Has This Been Tested?

• If you are adding a new feature, test on your own test scripts.

Added unit tests in test/notebook/test_util.py

Screenshots of Test Results (if appropriate):

1. Your own tests:

Test Outputs

(base) Pauls-MacBook-Pro:RD-Agent paulwang$ pytest test/notebook/test_util.py
=============================================================================================== test session starts ================================================================================================
platform darwin -- Python 3.11.0, pytest-8.4.1, pluggy-1.6.0
rootdir: /Users/paulwang/Desktop/tmp/RD-Agent
configfile: pyproject.toml
plugins: anyio-4.9.0, time-machine-2.16.0, typeguard-4.3.0
collected 47 items

test/notebook/test_util.py::TestExtractFunctionBody::test_empty PASSED
test/notebook/test_util.py::TestExtractFunctionBody::test_happy_path PASSED
test/notebook/test_util.py::TestExtractFunctionBody::test_happy_path_complex PASSED
test/notebook/test_util.py::TestExtractFunctionBody::test_missing_func PASSED
test/notebook/test_util.py::TestSplitCodeSections::test_empty PASSED
test/notebook/test_util.py::TestSplitCodeSections::test_happy_path PASSED
test/notebook/test_util.py::TestSplitCodeSections::test_happy_path_no_header PASSED
test/notebook/test_util.py::TestSplitCodeSections::test_no_sections PASSED
test/notebook/test_util.py::TestSplitCodeSections::test_single_no_sections PASSED
test/notebook/test_util.py::TestSplitCodeSections::test_single_with_section PASSED
test/notebook/test_util.py::TestSplitCodeSections::test_wrong_format PASSED
test/notebook/test_util.py::TestSplitOutputSections::test_empty PASSED
test/notebook/test_util.py::TestSplitOutputSections::test_happy_path PASSED
test/notebook/test_util.py::TestSplitOutputSections::test_happy_path_no_header PASSED
test/notebook/test_util.py::TestSplitOutputSections::test_no_sections PASSED
test/notebook/test_util.py::TestSplitOutputSections::test_single_no_sections PASSED
test/notebook/test_util.py::TestSplitOutputSections::test_single_with_section PASSED
test/notebook/test_util.py::TestSplitOutputSections::test_wrong_format PASSED
test/notebook/test_util.py::TestExtractSectionComments::test_arbitrary_print_happy_path PASSED
test/notebook/test_util.py::TestExtractSectionComments::test_empty_string PASSED
test/notebook/test_util.py::TestExtractSectionComments::test_happy_path PASSED
test/notebook/test_util.py::TestExtractSectionComments::test_happy_path_multiline PASSED
test/notebook/test_util.py::TestExtractSectionComments::test_no_comment PASSED
test/notebook/test_util.py::TestExtractFirstSectionNameFromCode::test_empty_string PASSED
test/notebook/test_util.py::TestExtractFirstSectionNameFromCode::test_happy_path PASSED
test/notebook/test_util.py::TestExtractFirstSectionNameFromCode::test_no_section PASSED
test/notebook/test_util.py::TestExtractFirstSectionNameFromOutput::test_empty_string PASSED
test/notebook/test_util.py::TestExtractFirstSectionNameFromOutput::test_happy_path PASSED
test/notebook/test_util.py::TestExtractFirstSectionNameFromOutput::test_no_section PASSED
test/notebook/test_util.py::TestIsFunctionCalled::test_empty_string PASSED
test/notebook/test_util.py::TestIsFunctionCalled::test_happy_path PASSED
test/notebook/test_util.py::TestIsFunctionCalled::test_happy_path_with_args PASSED
test/notebook/test_util.py::TestIsFunctionCalled::test_happy_path_with_args_multiline PASSED
test/notebook/test_util.py::TestIsFunctionCalled::test_not_called PASSED
test/notebook/test_util.py::TestIsFunctionCalled::test_wrong_format PASSED
test/notebook/test_util.py::TestRemoveFunction::test_empty PASSED
test/notebook/test_util.py::TestRemoveFunction::test_function_does_not_exist PASSED
test/notebook/test_util.py::TestRemoveFunction::test_happy_path PASSED
test/notebook/test_util.py::TestRemoveFunction::test_preserves_comments PASSED
test/notebook/test_util.py::TestRemoveMainBlock::test_block_does_not_exist PASSED
test/notebook/test_util.py::TestRemoveMainBlock::test_empty PASSED
test/notebook/test_util.py::TestRemoveMainBlock::test_happy_path PASSED
test/notebook/test_util.py::TestRemoveMainBlock::test_happy_path_arbitrary_content PASSED
test/notebook/test_util.py::TestRemoveMainBlock::test_one_liner PASSED
test/notebook/test_util.py::TestSplitCodeAndOutputIntoSections::test_empty_code PASSED
test/notebook/test_util.py::TestSplitCodeAndOutputIntoSections::test_empty_outputs PASSED
test/notebook/test_util.py::TestSplitCodeAndOutputIntoSections::test_happy_path PASSED

================================================================================================ slowest durations =================================================================================================

(141 durations < 0.005s hidden.  Use -vv to show these durations.)
================================================================================================ 47 passed in 0.09s ================================================================================================

Types of changes

• Fix bugs
• Add new feature
• Update documentation

---

📚 Documentation preview 📚: https://RDAgent--1134.org.readthedocs.build/en/1134/