ReST formatting fails, and there is no way to see why

[ewdurbin]

Originally reported by: Wichert Akkerman (Bitbucket: wichert, GitHub: wichert)

---

I just uploaded a new pyramid_sqlalchemy package to PyPI. Looking at the distribution page PyPI is not rendering ReST. I can’t figure out why. To debug this I all known (as far as I am aware) ways to debug this:

• I ran python setup.py check, which showed no errors.
• I ran python setup.py --long-description | rst2html-2.7.py > /dev/null. This also did not show any errors.
• I copied the description manually from the PKG-INFO file in the sdist and ran rst2html on that. That also did not show any errors.

As an uploader to PyPI this is very frustrating. It would be very useful if PyPI can report any errors it finds in descriptions. I don't quite care how this is done. Possible options are:

• when registering a distribution release with python setup.py register return all ReST errors so they are shown immediately when uploading.
• email any ReST rendering errors after an upload to the uploader. Perhaps email it to all uploaders in case uploading happens using a role account for which email is never read.
• show it in the web UI when updating the distribution information manually. This is slightly awkward since it requires an actual manual step to re-save the data without changing anything.

I don't think coming up with a way to run the syntax checks on a developer's machine, for example by creating a package which mimics PyPI's behaviour, as a workable approach: there is too much opportunity for environment changes (things like locale settings) and different package versions to lead to the wrong results.

---

• Bitbucket: https://bitbucket.org/pypa/pypi/issue/161

[ewdurbin]

Original comment by Sorin Sbarnea (Bitbucket: sorin, GitHub: sorin):

---

I have the same problem with https://pypi.python.org/pypi/jira and github is rendering it perfectly https://github.com/pycontribs/jira

[ewdurbin]

Original comment by Richard Jones (Bitbucket: r1chardj0n3s, GitHub: r1chardj0n3s):

---

Sorin: rst2html says:

x.rst:1: (WARNING/2) Title overline too short.

=========
JIRA Python Library
=========

[ewdurbin]

Original comment by Jens Diemer (Bitbucket: jedie, GitHub: jedie):

---

Me, too: https://pypi.python.org/pypi/DragonPyEmulator/0.2.0 ;(

EDIT: I fixed my README, thanks to: jedie/python-creole@23e519e
It was "link scheme not allowed" because of a link with: "xmpp://"

But definitely: PyPi should be less strict and display errors.

[ewdurbin]

Original comment by Daniel Greenfeld (Bitbucket: pydanny, GitHub: pydanny):

---

I would love to see parsing errors displayed. Why?

• Embarrassing for Python to have so many ugly pages on it's main package index.
• Makes beginners feel like they are screwing up, when in reality it's just PyPI and it's super-strict RestructuredText settings.
• Arguably obfuscates documentation!

Perhaps rst2html could be used, then have it's output captured and displayed?

[ewdurbin]

Original comment by Donald Stufft (Bitbucket: dstufft, GitHub: dstufft):

---

We can't return the RST errors directly because there's no way to prevent it from potentionally leaking data from the machines that run it. At least we'd need to seriously audit it to make sure it's not going to do that. In the works I have a thing that will run rst2html (basically) using the exact same settings as PyPI uses, I just haven't switched PyPI itself over to it yet.

[ewdurbin]

Original comment by Daniel Greenfeld (Bitbucket: pydanny, GitHub: pydanny):

---

What about switching PyPI to behave like the rest of the RestructuredText world? Like Sphinx, Pelican, et al?

[ewdurbin]

Original comment by Richard Jones (Bitbucket: r1chardj0n3s, GitHub: r1chardj0n3s):

---

Sorry, what does that request mean?

I ask that because there's no "just like the rest of the world" for embedding docutils output in a page.

[ewdurbin]

Original comment by Daniel Greenfeld (Bitbucket: pydanny, GitHub: pydanny):

---

The same RestructuredText that works in Pelican and Sphinx will often fail in PyPI.

[ewdurbin]

Original comment by Donald Stufft (Bitbucket: dstufft, GitHub: dstufft):

---

Pelican and Sphinx and PyPI have two very different threat models.

Pelican and Sphinx run on the developers local machine, so if you say... include a file, then at worst you're only going to include your own file.

PyPI runs on the PyPI servers, so if you include a file, maybe you'll include config.ini which holds the database passwords for PyPI.

Also Pelican and Sphinx tend to not be automatically ran on someone elses domain (RTD aside), so if you say inject a URL with a javacript: scheme the only site you're attacking is your own, vs on PyPI where you could use it steal PyPI cookies.

[ewdurbin]

Original comment by Chris Jerdonek (Bitbucket: cjerdonek, GitHub: cjerdonek):

---

Since the Packaging User Guide recommends using the web form on PyPI (e.g. uploading a PKG-INFO file), it makes sense to prioritize that over the command-line user experience.

If information from the parsing errors can't be provided to the user, can the web form at least say something like, "there were errors converting your long_description to reST, so PyPI is rendering it as plain-text. Go for help on resolving the errors."

[ewdurbin]

Original comment by Daniel Greenfeld (Bitbucket: pydanny, GitHub: pydanny):

---

+1 to what @cjerdonek says.

[ewdurbin]

Original comment by Chris Jerdonek (Bitbucket: cjerdonek, GitHub: cjerdonek):

---

In the works I have a thing that will run rst2html (basically) using the exact same settings as PyPI uses, I just haven't switched PyPI itself over to it yet.

@dstufft, is this in source control somewhere that people can look at?

[ewdurbin]

Original comment by Marius Gedminas (Bitbucket: mgedmin, GitHub: mgedmin):

---

@cjerdonek, the code is here.

Shameless plug: restview has a --pypi-strict mode that uses the exact same settings as PyPI, specifically for debugging issues like @wichert had.

[ewdurbin]

Original comment by Chris Jerdonek (Bitbucket: cjerdonek, GitHub: cjerdonek):

---

@mgedmin, it sounded like @dstufft had a new library he was working on. I was asking about the location of that.

[ewdurbin]

Original comment by Marius Gedminas (Bitbucket: mgedmin, GitHub: mgedmin):

---

restview --pypi-strict fails to find the error that breaks ReST rendering on https://pypi.python.org/pypi/pep257

It has been suggested on #pypa that the problem might be an older docutils version used on PyPI, since .. code:: was added in distutils 0.9 (August 2012).