Skip to content

Make preparser can handle large(4301+ digits) integers #40178

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Jun 25, 2025
Merged

Make preparser can handle large(4301+ digits) integers #40178

Changes from 5 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
45aaaa9
update preparser
soon-haari May 28, 2025
49ff108
unify quote usage
soon-haari May 28, 2025
aaf84e1
add length condition
soon-haari May 28, 2025
f43e339
update docs
soon-haari May 29, 2025
50c9eb1
add issue link
soon-haari May 30, 2025
de3a3da
Update src/sage/repl/preparse.py
soon-haari May 31, 2025
ee21b0c
fix typo
soon-haari May 31, 2025
c6d9e2d
update docs - quotes
soon-haari Jun 5, 2025
94b8909
move test
soon-haari Jun 6, 2025
266a113
minor fixes
soon-haari Jun 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 22 additions & 3 deletions src/sage/repl/preparse.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1124,8 +1124,9 @@
name-construction pairs

- ``quotes`` -- string (default: ``"'"``); used to surround string
arguments to RealNumber and ComplexNumber. If ``None``, will rebuild
the string using a list of its Unicode code-points.
arguments to RealNumber and ComplexNumber, and Integer when the
number is longer than 4300 digits. If ``None``, will rebuild the
string using a list of its Unicode code-points.

OUTPUT:

Expand Down Expand Up @@ -1189,6 +1190,14 @@
sage: preparse_numeric_literals('000042')
'Integer(42)'

Cheeck that :issue:`40179` is fixed::
sage: preparse_numeric_literals("1" * 4300) == f"Integer({"1" * 4300})"

Check failure on line 1194 in src/sage/repl/preparse.py

View workflow job for this annotation

GitHub Actions / Conda (macos, Python 3.11, all)

Failed example: 

Failed example:: Exception raised:
Traceback (most recent call last):
  File "/Users/runner/miniconda3/envs/sage-dev/lib/python3.11/site-packages/sage/doctest/forker.py", line 730, in _run
    self.compile_and_execute(example, compiler, test.globs)
  File "/Users/runner/miniconda3/envs/sage-dev/lib/python3.11/site-packages/sage/doctest/forker.py", line 1152, in compile_and_execute
    compiled = compiler(example)
               ^^^^^^^^^^^^^^^^^
  File "/Users/runner/miniconda3/envs/sage-dev/lib/python3.11/site-packages/sage/doctest/forker.py", line 696, in compiler
    code = compile(example.source, filename, "single",
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<doctest sage.repl.preparse.preparse_numeric_literals[27]>", line 1
    preparse_numeric_literals("1" * Integer(4300)) == f"Integer({"1" * Integer(4300)})"
                                                                  ^
SyntaxError: f-string: expecting '}'

Check failure on line 1194 in src/sage/repl/preparse.py

View workflow job for this annotation

GitHub Actions / Conda (ubuntu, Python 3.11, all)

Failed example: 

Failed example:: Exception raised:
Traceback (most recent call last):
  File "/usr/share/miniconda/envs/sage-dev/lib/python3.11/site-packages/sage/doctest/forker.py", line 730, in _run
    self.compile_and_execute(example, compiler, test.globs)
  File "/usr/share/miniconda/envs/sage-dev/lib/python3.11/site-packages/sage/doctest/forker.py", line 1152, in compile_and_execute
    compiled = compiler(example)
               ^^^^^^^^^^^^^^^^^
  File "/usr/share/miniconda/envs/sage-dev/lib/python3.11/site-packages/sage/doctest/forker.py", line 696, in compiler
    code = compile(example.source, filename, "single",
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<doctest sage.repl.preparse.preparse_numeric_literals[27]>", line 1
    preparse_numeric_literals("1" * Integer(4300)) == f"Integer({"1" * Integer(4300)})"
                                                                  ^
SyntaxError: f-string: expecting '}'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
sage: preparse_numeric_literals("1" * 4300) == f"Integer({"1" * 4300})"
sage: preparse_numeric_literals("1" * 4300) == f"Integer({'1' * 4300})"

This test is broken because the inner quote on the RHS needs to be different (or escaped \").

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

interestingly it's new feature in 3.12 https://stackoverflow.com/a/77858818/5267751 . Lowest supported version is (still) 3.11 so yes.

True
sage: preparse_numeric_literals("1" * 4301) == f"Integer({"1" * 4301})"

Check failure on line 1196 in src/sage/repl/preparse.py

View workflow job for this annotation

GitHub Actions / Conda (macos, Python 3.11, all)

Failed example: 

Failed example:: Exception raised:
Traceback (most recent call last):
  File "/Users/runner/miniconda3/envs/sage-dev/lib/python3.11/site-packages/sage/doctest/forker.py", line 730, in _run
    self.compile_and_execute(example, compiler, test.globs)
  File "/Users/runner/miniconda3/envs/sage-dev/lib/python3.11/site-packages/sage/doctest/forker.py", line 1152, in compile_and_execute
    compiled = compiler(example)
               ^^^^^^^^^^^^^^^^^
  File "/Users/runner/miniconda3/envs/sage-dev/lib/python3.11/site-packages/sage/doctest/forker.py", line 696, in compiler
    code = compile(example.source, filename, "single",
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<doctest sage.repl.preparse.preparse_numeric_literals[28]>", line 1
    preparse_numeric_literals("1" * Integer(4301)) == f"Integer({"1" * Integer(4301)})"
                                                                  ^
SyntaxError: f-string: expecting '}'

Check failure on line 1196 in src/sage/repl/preparse.py

View workflow job for this annotation

GitHub Actions / Conda (ubuntu, Python 3.11, all)

Failed example: 

Failed example:: Exception raised:
Traceback (most recent call last):
  File "/usr/share/miniconda/envs/sage-dev/lib/python3.11/site-packages/sage/doctest/forker.py", line 730, in _run
    self.compile_and_execute(example, compiler, test.globs)
  File "/usr/share/miniconda/envs/sage-dev/lib/python3.11/site-packages/sage/doctest/forker.py", line 1152, in compile_and_execute
    compiled = compiler(example)
               ^^^^^^^^^^^^^^^^^
  File "/usr/share/miniconda/envs/sage-dev/lib/python3.11/site-packages/sage/doctest/forker.py", line 696, in compiler
    code = compile(example.source, filename, "single",
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<doctest sage.repl.preparse.preparse_numeric_literals[28]>", line 1
    preparse_numeric_literals("1" * Integer(4301)) == f"Integer({"1" * Integer(4301)})"
                                                                  ^
SyntaxError: f-string: expecting '}'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
sage: preparse_numeric_literals("1" * 4301) == f"Integer({"1" * 4301})"
sage: preparse_numeric_literals("1" * 4301) == f"Integer({'1' * 4301})"

Same as above.

False
sage: preparse_numeric_literals("1" * 4301) == f"Integer('{"1" * 4301}')"

Check failure on line 1198 in src/sage/repl/preparse.py

View workflow job for this annotation

GitHub Actions / Conda (macos, Python 3.11, all)

Failed example: 

Failed example:: Exception raised:
Traceback (most recent call last):
  File "/Users/runner/miniconda3/envs/sage-dev/lib/python3.11/site-packages/sage/doctest/forker.py", line 730, in _run
    self.compile_and_execute(example, compiler, test.globs)
  File "/Users/runner/miniconda3/envs/sage-dev/lib/python3.11/site-packages/sage/doctest/forker.py", line 1152, in compile_and_execute
    compiled = compiler(example)
               ^^^^^^^^^^^^^^^^^
  File "/Users/runner/miniconda3/envs/sage-dev/lib/python3.11/site-packages/sage/doctest/forker.py", line 696, in compiler
    code = compile(example.source, filename, "single",
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<doctest sage.repl.preparse.preparse_numeric_literals[29]>", line 1
    preparse_numeric_literals("1" * Integer(4301)) == f"Integer('{"1" * Integer(4301)}')"
                                                                   ^
SyntaxError: f-string: expecting '}'

Check failure on line 1198 in src/sage/repl/preparse.py

View workflow job for this annotation

GitHub Actions / Conda (ubuntu, Python 3.11, all)

Failed example: 

Failed example:: Exception raised:
Traceback (most recent call last):
  File "/usr/share/miniconda/envs/sage-dev/lib/python3.11/site-packages/sage/doctest/forker.py", line 730, in _run
    self.compile_and_execute(example, compiler, test.globs)
  File "/usr/share/miniconda/envs/sage-dev/lib/python3.11/site-packages/sage/doctest/forker.py", line 1152, in compile_and_execute
    compiled = compiler(example)
               ^^^^^^^^^^^^^^^^^
  File "/usr/share/miniconda/envs/sage-dev/lib/python3.11/site-packages/sage/doctest/forker.py", line 696, in compiler
    code = compile(example.source, filename, "single",
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<doctest sage.repl.preparse.preparse_numeric_literals[29]>", line 1
    preparse_numeric_literals("1" * Integer(4301)) == f"Integer('{"1" * Integer(4301)}')"
                                                                   ^
SyntaxError: f-string: expecting '}'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
sage: preparse_numeric_literals("1" * 4301) == f"Integer('{"1" * 4301}')"
sage: preparse_numeric_literals("1" * 4301) == f"Integer('{'1' * 4301}')"

I think this will work (the test is a bit different). Please check locally before pushing.

True

Test underscores as digit separators (PEP 515,
https://www.python.org/dev/peps/pep-0515/)::

Expand Down Expand Up @@ -1253,6 +1262,10 @@
'RealNumber(str().join(map(chr, [51, 46, 49, 52])))'
sage: preparse_numeric_literals('5j', quotes=None)
'ComplexNumber(0, str().join(map(chr, [53])))'
sage: preparse_numeric_literals("1" * 4301, quotes=None) == f'Integer(str().join(map(chr, {[49] * 4301})))'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests that specifically address a bug should provide a link to the bug they are testing a fix for. Add a brief explanation of what this test is doing before hand and use :issue:`40179` to add a link to it.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree; in particular, this should be moved up to the place with all of the other tests related to this issue.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@soon-haari My suggestion was to move the test, not add another docstring. I think it is better if they are all grouped together.

True
sage: preparse_numeric_literals("1" * 4301, quotes="'''") == f"Integer('''{"1" * 4301}''')"

Check failure on line 1267 in src/sage/repl/preparse.py

View workflow job for this annotation

GitHub Actions / Conda (macos, Python 3.11, all)

Failed example: 

Failed example:: Exception raised:
Traceback (most recent call last):
  File "/Users/runner/miniconda3/envs/sage-dev/lib/python3.11/site-packages/sage/doctest/forker.py", line 730, in _run
    self.compile_and_execute(example, compiler, test.globs)
  File "/Users/runner/miniconda3/envs/sage-dev/lib/python3.11/site-packages/sage/doctest/forker.py", line 1152, in compile_and_execute
    compiled = compiler(example)
               ^^^^^^^^^^^^^^^^^
  File "/Users/runner/miniconda3/envs/sage-dev/lib/python3.11/site-packages/sage/doctest/forker.py", line 696, in compiler
    code = compile(example.source, filename, "single",
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<doctest sage.repl.preparse.preparse_numeric_literals[46]>", line 1
    preparse_numeric_literals("1" * Integer(4301), quotes="'''") == f"Integer('''{"1" * Integer(4301)}''')"
                                                                                   ^
SyntaxError: f-string: expecting '}'

Check failure on line 1267 in src/sage/repl/preparse.py

View workflow job for this annotation

GitHub Actions / Conda (ubuntu, Python 3.11, all)

Failed example: 

Failed example:: Exception raised:
Traceback (most recent call last):
  File "/usr/share/miniconda/envs/sage-dev/lib/python3.11/site-packages/sage/doctest/forker.py", line 730, in _run
    self.compile_and_execute(example, compiler, test.globs)
  File "/usr/share/miniconda/envs/sage-dev/lib/python3.11/site-packages/sage/doctest/forker.py", line 1152, in compile_and_execute
    compiled = compiler(example)
               ^^^^^^^^^^^^^^^^^
  File "/usr/share/miniconda/envs/sage-dev/lib/python3.11/site-packages/sage/doctest/forker.py", line 696, in compiler
    code = compile(example.source, filename, "single",
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<doctest sage.repl.preparse.preparse_numeric_literals[46]>", line 1
    preparse_numeric_literals("1" * Integer(4301), quotes="'''") == f"Integer('''{"1" * Integer(4301)}''')"
                                                                                   ^
SyntaxError: f-string: expecting '}'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
sage: preparse_numeric_literals("1" * 4301, quotes="'''") == f"Integer('''{"1" * 4301}''')"
sage: preparse_numeric_literals("1" * 4301, quotes="'''") == f"Integer('''{'1' * 4301}''')"

I think this test has become a bit overkill now. I would consider removing it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed! Althought my local test using sage -t .../preparse.py succeeded with double quotes inside double quotes 🤔, but still good to know, thanks!

True
"""
literals = {}
last = 0
Expand Down Expand Up @@ -1324,7 +1337,13 @@
# Python 3 does not allow leading zeroes. Sage does, so just strip them out.
# The number is still interpreted as decimal, not octal!
num = re.sub(r'^0+', '', num)
num_make = "Integer(%s)" % num
if len(num) <= 4300:
num_make = "Integer(%s)" % num
elif quotes:
num_make = "Integer(%s%s%s)" % (quotes, num, quotes)
else:
code_points = list(map(ord, list(num)))
num_make = "Integer(str().join(map(chr, %s)))" % code_points

literals[num_name] = num_make

Expand Down
Loading