luau-lang/luau
1
0
Fork 0
mirror of https://github.com/luau-lang/luau.git synced 2024-12-13 05:20:38 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
luau/.github/workflows/benchmark-dev.yml

186 lines
6.4 KiB
YAML
Raw Normal View History

bench: Implement first class support for callgrind (#570) Since callgrind allows to control stats collection from the guest, this allows us to reset the collection right before the benchmark starts. This change exposes this to the benchmark runner and integrates callgrind data parsing into bench.py, so that we can run bench.py with --callgrind argument and, as long as the runner was built with callgrind support, we get instruction counts from the run. We convert instruction counts to seconds using 10G instructions/second rate; there's no correct way to do this without simulating the full CPU pipeline but it results in time units on a similar scale to real runs.
2022-07-04 19:13:07 +01:00
name: benchmark-dev
on:
push:
branches:
- master
Feat: Added retry functionality to benchmarks (#667) Resolves #668 ## The problem Benchmarks jobs run concurrently for the different operating systems. This means that when it comes time to push the benchmark results to [the assigned benchmark results repo](https://github.com/luau-lang/benchmark-data), there can be two different jobs trying to push changes at the same time. In such a case, one of the pushes will fail and we end up missing some benchmark results data from the workflow run. ## The solution Whenever a push fails, we need to retry the steps leading up to the push (checking out the benchmark results repo, storing benchmark results, pushing the results to [a specific repo](https://github.com/luau-lang/benchmark-data)). ### Note There are 3 push attempts before submitting to failure. ## TL;DR This PR retries pushing benchmark results when they fail to get pushed (often due to pushing from multiple jobs concurrently) Co-authored-by: Jamie Kuppens <reshurum@gmail.com> Co-authored-by: Ignacio Falk <flakolefluk@gmail.com>
2022-09-12 16:45:55 +01:00
bench: Implement first class support for callgrind (#570) Since callgrind allows to control stats collection from the guest, this allows us to reset the collection right before the benchmark starts. This change exposes this to the benchmark runner and integrates callgrind data parsing into bench.py, so that we can run bench.py with --callgrind argument and, as long as the runner was built with callgrind support, we get instruction counts from the run. We convert instruction counts to seconds using 10G instructions/second rate; there's no correct way to do this without simulating the full CPU pipeline but it results in time units on a similar scale to real runs.
2022-07-04 19:13:07 +01:00
paths-ignore:
- "docs/**"
- "papers/**"
- "rfcs/**"
- "*.md"
jobs:
windows:
name: windows-${{matrix.arch}}
strategy:
fail-fast: false
matrix:
os: [windows-latest]
arch: [Win32, x64]
bench:
- {
script: "run-benchmarks",
timeout: 12,
title: "Luau Benchmarks",
}
benchResultsRepo:
- { name: "luau-lang/benchmark-data", branch: "main" }
runs-on: ${{ matrix.os }}
steps:
- name: Checkout Luau repository
uses: actions/checkout@v3
- name: Build Luau
shell: bash # necessary for fail-fast
run: |
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
cmake --build . --target Luau.Repl.CLI --config Release
cmake --build . --target Luau.Analyze.CLI --config Release
- name: Move build files to root
run: |
move build/Release/* .
- uses: actions/setup-python@v3
with:
python-version: "3.9"
architecture: "x64"
- name: Install python dependencies
run: |
python -m pip install requests
python -m pip install --user numpy scipy matplotlib ipython jupyter pandas sympy nose
- name: Run benchmark
run: |
python bench/bench.py | tee ${{ matrix.bench.script }}-output.txt
Feat: Added retry functionality to benchmarks (#667) Resolves #668 ## The problem Benchmarks jobs run concurrently for the different operating systems. This means that when it comes time to push the benchmark results to [the assigned benchmark results repo](https://github.com/luau-lang/benchmark-data), there can be two different jobs trying to push changes at the same time. In such a case, one of the pushes will fail and we end up missing some benchmark results data from the workflow run. ## The solution Whenever a push fails, we need to retry the steps leading up to the push (checking out the benchmark results repo, storing benchmark results, pushing the results to [a specific repo](https://github.com/luau-lang/benchmark-data)). ### Note There are 3 push attempts before submitting to failure. ## TL;DR This PR retries pushing benchmark results when they fail to get pushed (often due to pushing from multiple jobs concurrently) Co-authored-by: Jamie Kuppens <reshurum@gmail.com> Co-authored-by: Ignacio Falk <flakolefluk@gmail.com>
2022-09-12 16:45:55 +01:00
- name: Push benchmark results
id: pushBenchmarkAttempt1
continue-on-error: true
uses: ./.github/workflows/push-results
bench: Implement first class support for callgrind (#570) Since callgrind allows to control stats collection from the guest, this allows us to reset the collection right before the benchmark starts. This change exposes this to the benchmark runner and integrates callgrind data parsing into bench.py, so that we can run bench.py with --callgrind argument and, as long as the runner was built with callgrind support, we get instruction counts from the run. We convert instruction counts to seconds using 10G instructions/second rate; there's no correct way to do this without simulating the full CPU pipeline but it results in time units on a similar scale to real runs.
2022-07-04 19:13:07 +01:00
with:
repository: ${{ matrix.benchResultsRepo.name }}
Feat: Added retry functionality to benchmarks (#667) Resolves #668 ## The problem Benchmarks jobs run concurrently for the different operating systems. This means that when it comes time to push the benchmark results to [the assigned benchmark results repo](https://github.com/luau-lang/benchmark-data), there can be two different jobs trying to push changes at the same time. In such a case, one of the pushes will fail and we end up missing some benchmark results data from the workflow run. ## The solution Whenever a push fails, we need to retry the steps leading up to the push (checking out the benchmark results repo, storing benchmark results, pushing the results to [a specific repo](https://github.com/luau-lang/benchmark-data)). ### Note There are 3 push attempts before submitting to failure. ## TL;DR This PR retries pushing benchmark results when they fail to get pushed (often due to pushing from multiple jobs concurrently) Co-authored-by: Jamie Kuppens <reshurum@gmail.com> Co-authored-by: Ignacio Falk <flakolefluk@gmail.com>
2022-09-12 16:45:55 +01:00
branch: ${{ matrix.benchResultsRepo.branch }}
bench: Implement first class support for callgrind (#570) Since callgrind allows to control stats collection from the guest, this allows us to reset the collection right before the benchmark starts. This change exposes this to the benchmark runner and integrates callgrind data parsing into bench.py, so that we can run bench.py with --callgrind argument and, as long as the runner was built with callgrind support, we get instruction counts from the run. We convert instruction counts to seconds using 10G instructions/second rate; there's no correct way to do this without simulating the full CPU pipeline but it results in time units on a similar scale to real runs.
2022-07-04 19:13:07 +01:00
token: ${{ secrets.BENCH_GITHUB_TOKEN }}
path: "./gh-pages"
Feat: Added retry functionality to benchmarks (#667) Resolves #668 ## The problem Benchmarks jobs run concurrently for the different operating systems. This means that when it comes time to push the benchmark results to [the assigned benchmark results repo](https://github.com/luau-lang/benchmark-data), there can be two different jobs trying to push changes at the same time. In such a case, one of the pushes will fail and we end up missing some benchmark results data from the workflow run. ## The solution Whenever a push fails, we need to retry the steps leading up to the push (checking out the benchmark results repo, storing benchmark results, pushing the results to [a specific repo](https://github.com/luau-lang/benchmark-data)). ### Note There are 3 push attempts before submitting to failure. ## TL;DR This PR retries pushing benchmark results when they fail to get pushed (often due to pushing from multiple jobs concurrently) Co-authored-by: Jamie Kuppens <reshurum@gmail.com> Co-authored-by: Ignacio Falk <flakolefluk@gmail.com>
2022-09-12 16:45:55 +01:00
bench_name: "${{ matrix.bench.title }} (Windows ${{matrix.arch}})"
bench_tool: "benchmarkluau"
bench_output_file_path: "./${{ matrix.bench.script }}-output.txt"
bench_external_data_json_path: "./gh-pages/dev/bench/data-${{ matrix.os }}.json"
- name: Push benchmark results (Attempt 2)
id: pushBenchmarkAttempt2
continue-on-error: true
if: steps.pushBenchmarkAttempt1.outcome == 'failure'
uses: ./.github/workflows/push-results
bench: Implement first class support for callgrind (#570) Since callgrind allows to control stats collection from the guest, this allows us to reset the collection right before the benchmark starts. This change exposes this to the benchmark runner and integrates callgrind data parsing into bench.py, so that we can run bench.py with --callgrind argument and, as long as the runner was built with callgrind support, we get instruction counts from the run. We convert instruction counts to seconds using 10G instructions/second rate; there's no correct way to do this without simulating the full CPU pipeline but it results in time units on a similar scale to real runs.
2022-07-04 19:13:07 +01:00
with:
Feat: Added retry functionality to benchmarks (#667) Resolves #668 ## The problem Benchmarks jobs run concurrently for the different operating systems. This means that when it comes time to push the benchmark results to [the assigned benchmark results repo](https://github.com/luau-lang/benchmark-data), there can be two different jobs trying to push changes at the same time. In such a case, one of the pushes will fail and we end up missing some benchmark results data from the workflow run. ## The solution Whenever a push fails, we need to retry the steps leading up to the push (checking out the benchmark results repo, storing benchmark results, pushing the results to [a specific repo](https://github.com/luau-lang/benchmark-data)). ### Note There are 3 push attempts before submitting to failure. ## TL;DR This PR retries pushing benchmark results when they fail to get pushed (often due to pushing from multiple jobs concurrently) Co-authored-by: Jamie Kuppens <reshurum@gmail.com> Co-authored-by: Ignacio Falk <flakolefluk@gmail.com>
2022-09-12 16:45:55 +01:00
repository: ${{ matrix.benchResultsRepo.name }}
branch: ${{ matrix.benchResultsRepo.branch }}
token: ${{ secrets.BENCH_GITHUB_TOKEN }}
path: "./gh-pages"
bench_name: "${{ matrix.bench.title }} (Windows ${{matrix.arch}})"
bench_tool: "benchmarkluau"
bench_output_file_path: "./${{ matrix.bench.script }}-output.txt"
bench_external_data_json_path: "./gh-pages/dev/bench/data-${{ matrix.os }}.json"
- name: Push benchmark results (Attempt 3)
id: pushBenchmarkAttempt3
continue-on-error: true
if: steps.pushBenchmarkAttempt2.outcome == 'failure'
uses: ./.github/workflows/push-results
with:
repository: ${{ matrix.benchResultsRepo.name }}
branch: ${{ matrix.benchResultsRepo.branch }}
token: ${{ secrets.BENCH_GITHUB_TOKEN }}
path: "./gh-pages"
bench_name: "${{ matrix.bench.title }} (Windows ${{matrix.arch}})"
bench_tool: "benchmarkluau"
bench_output_file_path: "./${{ matrix.bench.script }}-output.txt"
bench_external_data_json_path: "./gh-pages/dev/bench/data-${{ matrix.os }}.json"
bench: Implement first class support for callgrind (#570) Since callgrind allows to control stats collection from the guest, this allows us to reset the collection right before the benchmark starts. This change exposes this to the benchmark runner and integrates callgrind data parsing into bench.py, so that we can run bench.py with --callgrind argument and, as long as the runner was built with callgrind support, we get instruction counts from the run. We convert instruction counts to seconds using 10G instructions/second rate; there's no correct way to do this without simulating the full CPU pipeline but it results in time units on a similar scale to real runs.
2022-07-04 19:13:07 +01:00
unix:
name: ${{matrix.os}}
strategy:
fail-fast: false
matrix:
Keep using ubuntu-20.04 in GitHub Actions (and clang++-10 in coverage) (#795) ### Problem ubuntu-latest was updated to 22.04 which removes clang++-10 we used for coverage stats and creates a pre-compiled binary that requires a glibc upgrade https://github.com/Roblox/luau/issues/773 ### Solution Pin to ubuntu-20.04 using multi-value matrix configurations. In coverage configuration, we use clang++-10 once again.
2023-01-06 20:17:25 +00:00
os: [ubuntu-20.04, macos-latest]
bench: Implement first class support for callgrind (#570) Since callgrind allows to control stats collection from the guest, this allows us to reset the collection right before the benchmark starts. This change exposes this to the benchmark runner and integrates callgrind data parsing into bench.py, so that we can run bench.py with --callgrind argument and, as long as the runner was built with callgrind support, we get instruction counts from the run. We convert instruction counts to seconds using 10G instructions/second rate; there's no correct way to do this without simulating the full CPU pipeline but it results in time units on a similar scale to real runs.
2022-07-04 19:13:07 +01:00
bench:
- {
script: "run-benchmarks",
timeout: 12,
title: "Luau Benchmarks",
}
benchResultsRepo:
- { name: "luau-lang/benchmark-data", branch: "main" }
runs-on: ${{ matrix.os }}
steps:
- name: Checkout Luau repository
uses: actions/checkout@v3
- name: Build Luau
run: make config=release luau luau-analyze
- uses: actions/setup-python@v3
with:
python-version: "3.9"
architecture: "x64"
- name: Install python dependencies
run: |
python -m pip install requests
python -m pip install --user numpy scipy matplotlib ipython jupyter pandas sympy nose
- name: Run benchmark
run: |
python bench/bench.py | tee ${{ matrix.bench.script }}-output.txt
Feat: Added retry functionality to benchmarks (#667) Resolves #668 ## The problem Benchmarks jobs run concurrently for the different operating systems. This means that when it comes time to push the benchmark results to [the assigned benchmark results repo](https://github.com/luau-lang/benchmark-data), there can be two different jobs trying to push changes at the same time. In such a case, one of the pushes will fail and we end up missing some benchmark results data from the workflow run. ## The solution Whenever a push fails, we need to retry the steps leading up to the push (checking out the benchmark results repo, storing benchmark results, pushing the results to [a specific repo](https://github.com/luau-lang/benchmark-data)). ### Note There are 3 push attempts before submitting to failure. ## TL;DR This PR retries pushing benchmark results when they fail to get pushed (often due to pushing from multiple jobs concurrently) Co-authored-by: Jamie Kuppens <reshurum@gmail.com> Co-authored-by: Ignacio Falk <flakolefluk@gmail.com>
2022-09-12 16:45:55 +01:00
- name: Push benchmark results
id: pushBenchmarkAttempt1
continue-on-error: true
uses: ./.github/workflows/push-results
bench: Implement first class support for callgrind (#570) Since callgrind allows to control stats collection from the guest, this allows us to reset the collection right before the benchmark starts. This change exposes this to the benchmark runner and integrates callgrind data parsing into bench.py, so that we can run bench.py with --callgrind argument and, as long as the runner was built with callgrind support, we get instruction counts from the run. We convert instruction counts to seconds using 10G instructions/second rate; there's no correct way to do this without simulating the full CPU pipeline but it results in time units on a similar scale to real runs.
2022-07-04 19:13:07 +01:00
with:
repository: ${{ matrix.benchResultsRepo.name }}
Feat: Added retry functionality to benchmarks (#667) Resolves #668 ## The problem Benchmarks jobs run concurrently for the different operating systems. This means that when it comes time to push the benchmark results to [the assigned benchmark results repo](https://github.com/luau-lang/benchmark-data), there can be two different jobs trying to push changes at the same time. In such a case, one of the pushes will fail and we end up missing some benchmark results data from the workflow run. ## The solution Whenever a push fails, we need to retry the steps leading up to the push (checking out the benchmark results repo, storing benchmark results, pushing the results to [a specific repo](https://github.com/luau-lang/benchmark-data)). ### Note There are 3 push attempts before submitting to failure. ## TL;DR This PR retries pushing benchmark results when they fail to get pushed (often due to pushing from multiple jobs concurrently) Co-authored-by: Jamie Kuppens <reshurum@gmail.com> Co-authored-by: Ignacio Falk <flakolefluk@gmail.com>
2022-09-12 16:45:55 +01:00
branch: ${{ matrix.benchResultsRepo.branch }}
bench: Implement first class support for callgrind (#570) Since callgrind allows to control stats collection from the guest, this allows us to reset the collection right before the benchmark starts. This change exposes this to the benchmark runner and integrates callgrind data parsing into bench.py, so that we can run bench.py with --callgrind argument and, as long as the runner was built with callgrind support, we get instruction counts from the run. We convert instruction counts to seconds using 10G instructions/second rate; there's no correct way to do this without simulating the full CPU pipeline but it results in time units on a similar scale to real runs.
2022-07-04 19:13:07 +01:00
token: ${{ secrets.BENCH_GITHUB_TOKEN }}
path: "./gh-pages"
Feat: Added retry functionality to benchmarks (#667) Resolves #668 ## The problem Benchmarks jobs run concurrently for the different operating systems. This means that when it comes time to push the benchmark results to [the assigned benchmark results repo](https://github.com/luau-lang/benchmark-data), there can be two different jobs trying to push changes at the same time. In such a case, one of the pushes will fail and we end up missing some benchmark results data from the workflow run. ## The solution Whenever a push fails, we need to retry the steps leading up to the push (checking out the benchmark results repo, storing benchmark results, pushing the results to [a specific repo](https://github.com/luau-lang/benchmark-data)). ### Note There are 3 push attempts before submitting to failure. ## TL;DR This PR retries pushing benchmark results when they fail to get pushed (often due to pushing from multiple jobs concurrently) Co-authored-by: Jamie Kuppens <reshurum@gmail.com> Co-authored-by: Ignacio Falk <flakolefluk@gmail.com>
2022-09-12 16:45:55 +01:00
bench_name: ${{ matrix.bench.title }}
bench_tool: "benchmarkluau"
bench_output_file_path: "./${{ matrix.bench.script }}-output.txt"
bench_external_data_json_path: "./gh-pages/dev/bench/data-${{ matrix.os }}.json"
- name: Push benchmark results (Attempt 2)
id: pushBenchmarkAttempt2
continue-on-error: true
if: steps.pushBenchmarkAttempt1.outcome == 'failure'
uses: ./.github/workflows/push-results
with:
repository: ${{ matrix.benchResultsRepo.name }}
branch: ${{ matrix.benchResultsRepo.branch }}
token: ${{ secrets.BENCH_GITHUB_TOKEN }}
path: "./gh-pages"
bench_name: ${{ matrix.bench.title }}
bench_tool: "benchmarkluau"
bench_output_file_path: "./${{ matrix.bench.script }}-output.txt"
bench_external_data_json_path: "./gh-pages/dev/bench/data-${{ matrix.os }}.json"
- name: Push benchmark results (Attempt 3)
id: pushBenchmarkAttempt3
continue-on-error: true
if: steps.pushBenchmarkAttempt2.outcome == 'failure'
uses: ./.github/workflows/push-results
bench: Implement first class support for callgrind (#570) Since callgrind allows to control stats collection from the guest, this allows us to reset the collection right before the benchmark starts. This change exposes this to the benchmark runner and integrates callgrind data parsing into bench.py, so that we can run bench.py with --callgrind argument and, as long as the runner was built with callgrind support, we get instruction counts from the run. We convert instruction counts to seconds using 10G instructions/second rate; there's no correct way to do this without simulating the full CPU pipeline but it results in time units on a similar scale to real runs.
2022-07-04 19:13:07 +01:00
with:
Feat: Added retry functionality to benchmarks (#667) Resolves #668 ## The problem Benchmarks jobs run concurrently for the different operating systems. This means that when it comes time to push the benchmark results to [the assigned benchmark results repo](https://github.com/luau-lang/benchmark-data), there can be two different jobs trying to push changes at the same time. In such a case, one of the pushes will fail and we end up missing some benchmark results data from the workflow run. ## The solution Whenever a push fails, we need to retry the steps leading up to the push (checking out the benchmark results repo, storing benchmark results, pushing the results to [a specific repo](https://github.com/luau-lang/benchmark-data)). ### Note There are 3 push attempts before submitting to failure. ## TL;DR This PR retries pushing benchmark results when they fail to get pushed (often due to pushing from multiple jobs concurrently) Co-authored-by: Jamie Kuppens <reshurum@gmail.com> Co-authored-by: Ignacio Falk <flakolefluk@gmail.com>
2022-09-12 16:45:55 +01:00
repository: ${{ matrix.benchResultsRepo.name }}
branch: ${{ matrix.benchResultsRepo.branch }}
token: ${{ secrets.BENCH_GITHUB_TOKEN }}
path: "./gh-pages"
bench_name: ${{ matrix.bench.title }}
bench_tool: "benchmarkluau"
bench_output_file_path: "./${{ matrix.bench.script }}-output.txt"
bench_external_data_json_path: "./gh-pages/dev/bench/data-${{ matrix.os }}.json"
Reference in a new issue Copy permalink