Skip to content

Benchmarking #10

Closed
Closed
@gvanrossum

Description

@gvanrossum

I'd like to have a benchmark so we have something concrete to target.

There are many benchmarks in PyPerformance and it runs for a long time. Some of the benchmarks are ancient (from the FORTRAN days) and focus on numeric array operations. I'm not interested in those (the users who have numeric arrays are all using numpy or a tensor package).

I like benchmarks that represent a more OO style of coding. (Note that even the "float" benchmark, which is supposed to measure some float operations including sqrt() and sin()/cos() was sped up by an improvement to the LOAD_ATTR opcode to speed up slots. :-) In PyPerformance there is a group that represent "apps" that we could use, or we could pick one of these.

There are also some benchmarks that the Pyston v2 project created: https://github.com/pyston/python-macrobenchmarks/ -- these would be interesting to try since they've got a somewhat similar goal as we do (keep the C/API unchanged) and they're farther along (claiming to be 20% faster) but they're closed source (for now).

For me, an important requirement is that a benchmark runs fairly quickly. If I have a benchmark that runs for a minute I'd probably be running it a lot to validate various tweaks I am experimenting with, even if I knew that the results were pretty noisy. OTOH if I only had a benchmark that ran for 15 minutes I'd probably run it only once or twice a day. If it ran for an hour I'd probably only run it overnight. We should probably run all of PyPerformance occasionally since it is used by the core dev team to validate whether a proposed speedup (a) does anything good for at least some of the benchmarks, and (b) doesn't slow anything down.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions