Existing sample/recipe for integrating asv with github

Backstory aside - is anything aware of an existing use of asv integrated with GitHub?

What I envision is having a container image with eg buildbot and asv installed, to be deployed on a dedicated box, configured to monitor/report back on PRs to GitHub about impact on benchmarks.

May be there is a service which already does something like that?

I will look into setting up such one, but wanted to make sure I am not reinventing the wheel or making it square.

@yarikoptic

@yarikoptic did you manage to build such a service?

Nope, but my colleagues set it up to just run asv to get an estimate for performance impact comparing against master in PRs. See e.g. our datalad github actions workflow: https://github.com/datalad/datalad/blob/master/.github/workflows/benchmarks.yml and the actual helper https://github.com/datalad/datalad/blob/master/tools/ci/benchmark-travis-pr.sh . Eventually we just might look into establishing self-hosted runner for actions: https://help.github.com/en/actions/automating-your-workflow-with-github-actions/about-self-hosted-runners so we get more stable estimates

note - that we nohow report back overall effect, this action is merely for manual investigation if there is a suspicion that PR might affect performance. Would be nice to report back to action state if e.g. any benchmark became significantly slower. we haven't looked into that yet

In case it helps anyone, I just set up asv to run a job on each PR which compares it against the branch it is being merged into. It doesn't save the results anywhere, but anyone can look at the logs there to see if affects the performance.

My github action looked like this:

  benchmark:
    name: benchmark
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-python@v2
        with:
          python-version: "3.10"
      - uses: actions/checkout@v3
      - run: git fetch origin $GITHUB_BASE_REF:base $GITHUB_REF:pr
      - run: pip install asv virtualenv
      - run: asv machine --yes
      - run: asv continuous base pr --strict -e

in https://github.com/datalad/datalad/blob/master/.github/workflows/benchmarks.yml we also fail the workflow if any benchmarks gets significantly worse:

    - name: Fail if any benchmarks have slowed down too much
      run: |
        ! asv compare --factor 1.2 --split refs/bm/merge-target refs/bm/pr | grep -q "got worse"

@HaoZeke

Hi, I managed to integrate asv with django-components in django-components/django-components#999.

Our integration covers these use cases:

Performance report on pull requests.
Benchmarking the package across releases.
Displaying performance results on a website.

Since I have this info so fresh in my mind, and since it was so tricky to get it working (just the Github workflow integration took me a day), I'm up for writing this down as a tutorial, so that others don't have to reinvent the wheel.

I'm thinking there could be a "Guides" section in the docs, and the use cases above could be pages in the "guides" section.

@HaoZeke @mattip What do you think?

Related to this, I made some changes as I was trying to get things working:

I had to monkeypatch from outside these bugs:

These are things that were not clear or missing from the docs, so I updated them and opened PRs for them:

I'd be great if we could get these bug fixes and docs PRs resolved as soon as possible, so that the guides articles wouldn't have to include the workarounds. Because the workarounds don't look pretty (see example)

P.S.: For completeness, here are other suggestions I made that would make the interaction with asv smoother and more flexible:

Hi, I managed to integrate asv with django-components in django-components/django-components#999.

Our integration covers these use cases:

Performance report on pull requests.

Benchmarking the package across releases.

Displaying performance results on a website.

Since I have this info so fresh in my mind, and since it was so tricky to get it working (just the Github workflow integration took me a day), I'm up for writing this down as a tutorial, so that others don't have to reinvent the wheel.

I'm thinking there could be a "Guides" section in the docs, and the use cases above could be pages in the "guides" section.

@HaoZeke @mattip What do you think?

Related to this, I made some changes as I was trying to get things working:

I had to monkeypatch from outside these bugs:

BUG: "OSError: [Errno 7] Argument list too long" asv_runner#45

fix: timeraw with params doesn't use the returned script string asv_runner#44

These are things that were not clear or missing from the docs, so I updated them and opened PRs for them:

docs: document pitfalls of changing benchmark_name benchmark attribute #1470

docs: document the version field in asv.conf.json.rst #1467

I'd be great if we could get these bug fixes and docs PRs resolved as soon as possible, so that the guides articles wouldn't have to include the workarounds. Because the workarounds don't look pretty (see example)

P.S.: For completeness, here are other suggestions I made that would make the interaction with asv smoother and more flexible:

Feature request: Allow to pass env vars to timeraw #1471

Feature request: @benchmark decorator #1469

Feature request: Allow to override the files generated with asv publish #1473

A guides section would be fantastic. I've been meaning to bring the ASV documentation into the modern style (e.g. NumPy, SciPy, anything following diataxis basically) of having user stories, API details and tutorials, but haven't gotten the time to focus on it yet. This would make a great guide, and I look forward to reviewing it as soon as it is ready

Great to hear! I've made the PR for the documentation here #1475.

I also made a PR to fix the "OSError: [Errno 7] Argument list too long" error #1475

Btw that diataxis is really cool. I was implicitly kinda aware of it, even django-component's documentation follows that structure, but it's great to know that it has a name 😄 I arrived at similar setup when looking at how Pydantic or FastAPI structure their docs.

I also made a PR to fix the "OSError: [Errno 7] Argument list too long" error

Did this ever show up? I couldn't find it

Edit: yes, it was in the runner airspeed-velocity/asv_runner#46

Navigation Menu

Repository navigation

Existing sample/recipe for integrating asv with github #796

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Existing sample/recipe for integrating asv with github #796

Description

Activity

saraedum commented on Jan 22, 2020

yarikoptic commented on Jan 22, 2020

yarikoptic commented on Jan 22, 2020

saulshanabrook commented on Jul 18, 2022

yarikoptic commented on Jul 19, 2022

JuroOravec commented on Feb 23, 2025

HaoZeke commented on Feb 24, 2025

JuroOravec commented on Feb 24, 2025

mattip commented on Dec 3, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions