luau-lang/luau
1
0
Fork 0
mirror of https://github.com/luau-lang/luau.git synced 2024-12-13 21:40:43 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 3
luau/.github/workflows/benchmark-dev.yml

386 lines
15 KiB
YAML
Raw Normal View History

bench: Implement first class support for callgrind (#570) Since callgrind allows to control stats collection from the guest, this allows us to reset the collection right before the benchmark starts. This change exposes this to the benchmark runner and integrates callgrind data parsing into bench.py, so that we can run bench.py with --callgrind argument and, as long as the runner was built with callgrind support, we get instruction counts from the run. We convert instruction counts to seconds using 10G instructions/second rate; there's no correct way to do this without simulating the full CPU pipeline but it results in time units on a similar scale to real runs.
2022-07-04 19:13:07 +01:00
name: benchmark-dev
on:
push:
branches:
- master
Feat: Added retry functionality to benchmarks (#667) Resolves #668 ## The problem Benchmarks jobs run concurrently for the different operating systems. This means that when it comes time to push the benchmark results to [the assigned benchmark results repo](https://github.com/luau-lang/benchmark-data), there can be two different jobs trying to push changes at the same time. In such a case, one of the pushes will fail and we end up missing some benchmark results data from the workflow run. ## The solution Whenever a push fails, we need to retry the steps leading up to the push (checking out the benchmark results repo, storing benchmark results, pushing the results to [a specific repo](https://github.com/luau-lang/benchmark-data)). ### Note There are 3 push attempts before submitting to failure. ## TL;DR This PR retries pushing benchmark results when they fail to get pushed (often due to pushing from multiple jobs concurrently) Co-authored-by: Jamie Kuppens <reshurum@gmail.com> Co-authored-by: Ignacio Falk <flakolefluk@gmail.com>
2022-09-12 16:45:55 +01:00
bench: Implement first class support for callgrind (#570) Since callgrind allows to control stats collection from the guest, this allows us to reset the collection right before the benchmark starts. This change exposes this to the benchmark runner and integrates callgrind data parsing into bench.py, so that we can run bench.py with --callgrind argument and, as long as the runner was built with callgrind support, we get instruction counts from the run. We convert instruction counts to seconds using 10G instructions/second rate; there's no correct way to do this without simulating the full CPU pipeline but it results in time units on a similar scale to real runs.
2022-07-04 19:13:07 +01:00
paths-ignore:
- "docs/**"
- "papers/**"
- "rfcs/**"
- "*.md"
- "prototyping/**"
jobs:
windows:
name: windows-${{matrix.arch}}
strategy:
fail-fast: false
matrix:
os: [windows-latest]
arch: [Win32, x64]
bench:
- {
script: "run-benchmarks",
timeout: 12,
title: "Luau Benchmarks",
cachegrindTitle: "Performance",
cachegrindIterCount: 20,
}
benchResultsRepo:
- { name: "luau-lang/benchmark-data", branch: "main" }
runs-on: ${{ matrix.os }}
steps:
- name: Checkout Luau repository
uses: actions/checkout@v3
- name: Build Luau
shell: bash # necessary for fail-fast
run: |
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
cmake --build . --target Luau.Repl.CLI --config Release
cmake --build . --target Luau.Analyze.CLI --config Release
- name: Move build files to root
run: |
move build/Release/* .
- uses: actions/setup-python@v3
with:
python-version: "3.9"
architecture: "x64"
- name: Install python dependencies
run: |
python -m pip install requests
python -m pip install --user numpy scipy matplotlib ipython jupyter pandas sympy nose
- name: Run benchmark
run: |
python bench/bench.py | tee ${{ matrix.bench.script }}-output.txt
Feat: Added retry functionality to benchmarks (#667) Resolves #668 ## The problem Benchmarks jobs run concurrently for the different operating systems. This means that when it comes time to push the benchmark results to [the assigned benchmark results repo](https://github.com/luau-lang/benchmark-data), there can be two different jobs trying to push changes at the same time. In such a case, one of the pushes will fail and we end up missing some benchmark results data from the workflow run. ## The solution Whenever a push fails, we need to retry the steps leading up to the push (checking out the benchmark results repo, storing benchmark results, pushing the results to [a specific repo](https://github.com/luau-lang/benchmark-data)). ### Note There are 3 push attempts before submitting to failure. ## TL;DR This PR retries pushing benchmark results when they fail to get pushed (often due to pushing from multiple jobs concurrently) Co-authored-by: Jamie Kuppens <reshurum@gmail.com> Co-authored-by: Ignacio Falk <flakolefluk@gmail.com>
2022-09-12 16:45:55 +01:00
- name: Push benchmark results
id: pushBenchmarkAttempt1
continue-on-error: true
uses: ./.github/workflows/push-results
bench: Implement first class support for callgrind (#570) Since callgrind allows to control stats collection from the guest, this allows us to reset the collection right before the benchmark starts. This change exposes this to the benchmark runner and integrates callgrind data parsing into bench.py, so that we can run bench.py with --callgrind argument and, as long as the runner was built with callgrind support, we get instruction counts from the run. We convert instruction counts to seconds using 10G instructions/second rate; there's no correct way to do this without simulating the full CPU pipeline but it results in time units on a similar scale to real runs.
2022-07-04 19:13:07 +01:00
with:
repository: ${{ matrix.benchResultsRepo.name }}
Feat: Added retry functionality to benchmarks (#667) Resolves #668 ## The problem Benchmarks jobs run concurrently for the different operating systems. This means that when it comes time to push the benchmark results to [the assigned benchmark results repo](https://github.com/luau-lang/benchmark-data), there can be two different jobs trying to push changes at the same time. In such a case, one of the pushes will fail and we end up missing some benchmark results data from the workflow run. ## The solution Whenever a push fails, we need to retry the steps leading up to the push (checking out the benchmark results repo, storing benchmark results, pushing the results to [a specific repo](https://github.com/luau-lang/benchmark-data)). ### Note There are 3 push attempts before submitting to failure. ## TL;DR This PR retries pushing benchmark results when they fail to get pushed (often due to pushing from multiple jobs concurrently) Co-authored-by: Jamie Kuppens <reshurum@gmail.com> Co-authored-by: Ignacio Falk <flakolefluk@gmail.com>
2022-09-12 16:45:55 +01:00
branch: ${{ matrix.benchResultsRepo.branch }}
bench: Implement first class support for callgrind (#570) Since callgrind allows to control stats collection from the guest, this allows us to reset the collection right before the benchmark starts. This change exposes this to the benchmark runner and integrates callgrind data parsing into bench.py, so that we can run bench.py with --callgrind argument and, as long as the runner was built with callgrind support, we get instruction counts from the run. We convert instruction counts to seconds using 10G instructions/second rate; there's no correct way to do this without simulating the full CPU pipeline but it results in time units on a similar scale to real runs.
2022-07-04 19:13:07 +01:00
token: ${{ secrets.BENCH_GITHUB_TOKEN }}
path: "./gh-pages"
Feat: Added retry functionality to benchmarks (#667) Resolves #668 ## The problem Benchmarks jobs run concurrently for the different operating systems. This means that when it comes time to push the benchmark results to [the assigned benchmark results repo](https://github.com/luau-lang/benchmark-data), there can be two different jobs trying to push changes at the same time. In such a case, one of the pushes will fail and we end up missing some benchmark results data from the workflow run. ## The solution Whenever a push fails, we need to retry the steps leading up to the push (checking out the benchmark results repo, storing benchmark results, pushing the results to [a specific repo](https://github.com/luau-lang/benchmark-data)). ### Note There are 3 push attempts before submitting to failure. ## TL;DR This PR retries pushing benchmark results when they fail to get pushed (often due to pushing from multiple jobs concurrently) Co-authored-by: Jamie Kuppens <reshurum@gmail.com> Co-authored-by: Ignacio Falk <flakolefluk@gmail.com>
2022-09-12 16:45:55 +01:00
bench_name: "${{ matrix.bench.title }} (Windows ${{matrix.arch}})"
bench_tool: "benchmarkluau"
bench_output_file_path: "./${{ matrix.bench.script }}-output.txt"
bench_external_data_json_path: "./gh-pages/dev/bench/data-${{ matrix.os }}.json"
- name: Push benchmark results (Attempt 2)
id: pushBenchmarkAttempt2
continue-on-error: true
if: steps.pushBenchmarkAttempt1.outcome == 'failure'
uses: ./.github/workflows/push-results
bench: Implement first class support for callgrind (#570) Since callgrind allows to control stats collection from the guest, this allows us to reset the collection right before the benchmark starts. This change exposes this to the benchmark runner and integrates callgrind data parsing into bench.py, so that we can run bench.py with --callgrind argument and, as long as the runner was built with callgrind support, we get instruction counts from the run. We convert instruction counts to seconds using 10G instructions/second rate; there's no correct way to do this without simulating the full CPU pipeline but it results in time units on a similar scale to real runs.
2022-07-04 19:13:07 +01:00
with:
Feat: Added retry functionality to benchmarks (#667) Resolves #668 ## The problem Benchmarks jobs run concurrently for the different operating systems. This means that when it comes time to push the benchmark results to [the assigned benchmark results repo](https://github.com/luau-lang/benchmark-data), there can be two different jobs trying to push changes at the same time. In such a case, one of the pushes will fail and we end up missing some benchmark results data from the workflow run. ## The solution Whenever a push fails, we need to retry the steps leading up to the push (checking out the benchmark results repo, storing benchmark results, pushing the results to [a specific repo](https://github.com/luau-lang/benchmark-data)). ### Note There are 3 push attempts before submitting to failure. ## TL;DR This PR retries pushing benchmark results when they fail to get pushed (often due to pushing from multiple jobs concurrently) Co-authored-by: Jamie Kuppens <reshurum@gmail.com> Co-authored-by: Ignacio Falk <flakolefluk@gmail.com>
2022-09-12 16:45:55 +01:00
repository: ${{ matrix.benchResultsRepo.name }}
branch: ${{ matrix.benchResultsRepo.branch }}
token: ${{ secrets.BENCH_GITHUB_TOKEN }}
path: "./gh-pages"
bench_name: "${{ matrix.bench.title }} (Windows ${{matrix.arch}})"
bench_tool: "benchmarkluau"
bench_output_file_path: "./${{ matrix.bench.script }}-output.txt"
bench_external_data_json_path: "./gh-pages/dev/bench/data-${{ matrix.os }}.json"
- name: Push benchmark results (Attempt 3)
id: pushBenchmarkAttempt3
continue-on-error: true
if: steps.pushBenchmarkAttempt2.outcome == 'failure'
uses: ./.github/workflows/push-results
with:
repository: ${{ matrix.benchResultsRepo.name }}
branch: ${{ matrix.benchResultsRepo.branch }}
token: ${{ secrets.BENCH_GITHUB_TOKEN }}
path: "./gh-pages"
bench_name: "${{ matrix.bench.title }} (Windows ${{matrix.arch}})"
bench_tool: "benchmarkluau"
bench_output_file_path: "./${{ matrix.bench.script }}-output.txt"
bench_external_data_json_path: "./gh-pages/dev/bench/data-${{ matrix.os }}.json"
bench: Implement first class support for callgrind (#570) Since callgrind allows to control stats collection from the guest, this allows us to reset the collection right before the benchmark starts. This change exposes this to the benchmark runner and integrates callgrind data parsing into bench.py, so that we can run bench.py with --callgrind argument and, as long as the runner was built with callgrind support, we get instruction counts from the run. We convert instruction counts to seconds using 10G instructions/second rate; there's no correct way to do this without simulating the full CPU pipeline but it results in time units on a similar scale to real runs.
2022-07-04 19:13:07 +01:00
unix:
name: ${{matrix.os}}
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest]
bench:
- {
script: "run-benchmarks",
timeout: 12,
title: "Luau Benchmarks",
cachegrindTitle: "Performance",
cachegrindIterCount: 20,
}
benchResultsRepo:
- { name: "luau-lang/benchmark-data", branch: "main" }
runs-on: ${{ matrix.os }}
steps:
- name: Checkout Luau repository
uses: actions/checkout@v3
- name: Build Luau
run: make config=release luau luau-analyze
- uses: actions/setup-python@v3
with:
python-version: "3.9"
architecture: "x64"
- name: Install python dependencies
run: |
python -m pip install requests
python -m pip install --user numpy scipy matplotlib ipython jupyter pandas sympy nose
- name: Run benchmark
run: |
python bench/bench.py | tee ${{ matrix.bench.script }}-output.txt
- name: Install valgrind
if: matrix.os == 'ubuntu-latest'
run: |
sudo apt-get install valgrind
- name: Run ${{ matrix.bench.title }} (Cold Cachegrind)
if: matrix.os == 'ubuntu-latest'
run: sudo bash ./scripts/run-with-cachegrind.sh python ./bench/bench.py "${{ matrix.bench.cachegrindTitle}}Cold" 1 | tee -a ${{ matrix.bench.script }}-output.txt
- name: Run ${{ matrix.bench.title }} (Warm Cachegrind)
if: matrix.os == 'ubuntu-latest'
run: sudo bash ./scripts/run-with-cachegrind.sh python ./bench/bench.py "${{ matrix.bench.cachegrindTitle }}" ${{ matrix.bench.cachegrindIterCount }} | tee -a ${{ matrix.bench.script }}-output.txt
Feat: Added retry functionality to benchmarks (#667) Resolves #668 ## The problem Benchmarks jobs run concurrently for the different operating systems. This means that when it comes time to push the benchmark results to [the assigned benchmark results repo](https://github.com/luau-lang/benchmark-data), there can be two different jobs trying to push changes at the same time. In such a case, one of the pushes will fail and we end up missing some benchmark results data from the workflow run. ## The solution Whenever a push fails, we need to retry the steps leading up to the push (checking out the benchmark results repo, storing benchmark results, pushing the results to [a specific repo](https://github.com/luau-lang/benchmark-data)). ### Note There are 3 push attempts before submitting to failure. ## TL;DR This PR retries pushing benchmark results when they fail to get pushed (often due to pushing from multiple jobs concurrently) Co-authored-by: Jamie Kuppens <reshurum@gmail.com> Co-authored-by: Ignacio Falk <flakolefluk@gmail.com>
2022-09-12 16:45:55 +01:00
- name: Push benchmark results
id: pushBenchmarkAttempt1
continue-on-error: true
uses: ./.github/workflows/push-results
bench: Implement first class support for callgrind (#570) Since callgrind allows to control stats collection from the guest, this allows us to reset the collection right before the benchmark starts. This change exposes this to the benchmark runner and integrates callgrind data parsing into bench.py, so that we can run bench.py with --callgrind argument and, as long as the runner was built with callgrind support, we get instruction counts from the run. We convert instruction counts to seconds using 10G instructions/second rate; there's no correct way to do this without simulating the full CPU pipeline but it results in time units on a similar scale to real runs.
2022-07-04 19:13:07 +01:00
with:
repository: ${{ matrix.benchResultsRepo.name }}
Feat: Added retry functionality to benchmarks (#667) Resolves #668 ## The problem Benchmarks jobs run concurrently for the different operating systems. This means that when it comes time to push the benchmark results to [the assigned benchmark results repo](https://github.com/luau-lang/benchmark-data), there can be two different jobs trying to push changes at the same time. In such a case, one of the pushes will fail and we end up missing some benchmark results data from the workflow run. ## The solution Whenever a push fails, we need to retry the steps leading up to the push (checking out the benchmark results repo, storing benchmark results, pushing the results to [a specific repo](https://github.com/luau-lang/benchmark-data)). ### Note There are 3 push attempts before submitting to failure. ## TL;DR This PR retries pushing benchmark results when they fail to get pushed (often due to pushing from multiple jobs concurrently) Co-authored-by: Jamie Kuppens <reshurum@gmail.com> Co-authored-by: Ignacio Falk <flakolefluk@gmail.com>
2022-09-12 16:45:55 +01:00
branch: ${{ matrix.benchResultsRepo.branch }}
bench: Implement first class support for callgrind (#570) Since callgrind allows to control stats collection from the guest, this allows us to reset the collection right before the benchmark starts. This change exposes this to the benchmark runner and integrates callgrind data parsing into bench.py, so that we can run bench.py with --callgrind argument and, as long as the runner was built with callgrind support, we get instruction counts from the run. We convert instruction counts to seconds using 10G instructions/second rate; there's no correct way to do this without simulating the full CPU pipeline but it results in time units on a similar scale to real runs.
2022-07-04 19:13:07 +01:00
token: ${{ secrets.BENCH_GITHUB_TOKEN }}
path: "./gh-pages"
Feat: Added retry functionality to benchmarks (#667) Resolves #668 ## The problem Benchmarks jobs run concurrently for the different operating systems. This means that when it comes time to push the benchmark results to [the assigned benchmark results repo](https://github.com/luau-lang/benchmark-data), there can be two different jobs trying to push changes at the same time. In such a case, one of the pushes will fail and we end up missing some benchmark results data from the workflow run. ## The solution Whenever a push fails, we need to retry the steps leading up to the push (checking out the benchmark results repo, storing benchmark results, pushing the results to [a specific repo](https://github.com/luau-lang/benchmark-data)). ### Note There are 3 push attempts before submitting to failure. ## TL;DR This PR retries pushing benchmark results when they fail to get pushed (often due to pushing from multiple jobs concurrently) Co-authored-by: Jamie Kuppens <reshurum@gmail.com> Co-authored-by: Ignacio Falk <flakolefluk@gmail.com>
2022-09-12 16:45:55 +01:00
bench_name: ${{ matrix.bench.title }}
bench_tool: "benchmarkluau"
bench_output_file_path: "./${{ matrix.bench.script }}-output.txt"
bench_external_data_json_path: "./gh-pages/dev/bench/data-${{ matrix.os }}.json"
- name: Push benchmark results (Attempt 2)
id: pushBenchmarkAttempt2
continue-on-error: true
if: steps.pushBenchmarkAttempt1.outcome == 'failure'
uses: ./.github/workflows/push-results
with:
repository: ${{ matrix.benchResultsRepo.name }}
branch: ${{ matrix.benchResultsRepo.branch }}
token: ${{ secrets.BENCH_GITHUB_TOKEN }}
path: "./gh-pages"
bench_name: ${{ matrix.bench.title }}
bench_tool: "benchmarkluau"
bench_output_file_path: "./${{ matrix.bench.script }}-output.txt"
bench_external_data_json_path: "./gh-pages/dev/bench/data-${{ matrix.os }}.json"
- name: Push benchmark results (Attempt 3)
id: pushBenchmarkAttempt3
continue-on-error: true
if: steps.pushBenchmarkAttempt2.outcome == 'failure'
uses: ./.github/workflows/push-results
bench: Implement first class support for callgrind (#570) Since callgrind allows to control stats collection from the guest, this allows us to reset the collection right before the benchmark starts. This change exposes this to the benchmark runner and integrates callgrind data parsing into bench.py, so that we can run bench.py with --callgrind argument and, as long as the runner was built with callgrind support, we get instruction counts from the run. We convert instruction counts to seconds using 10G instructions/second rate; there's no correct way to do this without simulating the full CPU pipeline but it results in time units on a similar scale to real runs.
2022-07-04 19:13:07 +01:00
with:
Feat: Added retry functionality to benchmarks (#667) Resolves #668 ## The problem Benchmarks jobs run concurrently for the different operating systems. This means that when it comes time to push the benchmark results to [the assigned benchmark results repo](https://github.com/luau-lang/benchmark-data), there can be two different jobs trying to push changes at the same time. In such a case, one of the pushes will fail and we end up missing some benchmark results data from the workflow run. ## The solution Whenever a push fails, we need to retry the steps leading up to the push (checking out the benchmark results repo, storing benchmark results, pushing the results to [a specific repo](https://github.com/luau-lang/benchmark-data)). ### Note There are 3 push attempts before submitting to failure. ## TL;DR This PR retries pushing benchmark results when they fail to get pushed (often due to pushing from multiple jobs concurrently) Co-authored-by: Jamie Kuppens <reshurum@gmail.com> Co-authored-by: Ignacio Falk <flakolefluk@gmail.com>
2022-09-12 16:45:55 +01:00
repository: ${{ matrix.benchResultsRepo.name }}
branch: ${{ matrix.benchResultsRepo.branch }}
token: ${{ secrets.BENCH_GITHUB_TOKEN }}
path: "./gh-pages"
bench_name: ${{ matrix.bench.title }}
bench_tool: "benchmarkluau"
bench_output_file_path: "./${{ matrix.bench.script }}-output.txt"
bench_external_data_json_path: "./gh-pages/dev/bench/data-${{ matrix.os }}.json"
bench: Implement first class support for callgrind (#570) Since callgrind allows to control stats collection from the guest, this allows us to reset the collection right before the benchmark starts. This change exposes this to the benchmark runner and integrates callgrind data parsing into bench.py, so that we can run bench.py with --callgrind argument and, as long as the runner was built with callgrind support, we get instruction counts from the run. We convert instruction counts to seconds using 10G instructions/second rate; there's no correct way to do this without simulating the full CPU pipeline but it results in time units on a similar scale to real runs.
2022-07-04 19:13:07 +01:00
Feat: Added retry functionality to benchmarks (#667) Resolves #668 ## The problem Benchmarks jobs run concurrently for the different operating systems. This means that when it comes time to push the benchmark results to [the assigned benchmark results repo](https://github.com/luau-lang/benchmark-data), there can be two different jobs trying to push changes at the same time. In such a case, one of the pushes will fail and we end up missing some benchmark results data from the workflow run. ## The solution Whenever a push fails, we need to retry the steps leading up to the push (checking out the benchmark results repo, storing benchmark results, pushing the results to [a specific repo](https://github.com/luau-lang/benchmark-data)). ### Note There are 3 push attempts before submitting to failure. ## TL;DR This PR retries pushing benchmark results when they fail to get pushed (often due to pushing from multiple jobs concurrently) Co-authored-by: Jamie Kuppens <reshurum@gmail.com> Co-authored-by: Ignacio Falk <flakolefluk@gmail.com>
2022-09-12 16:45:55 +01:00
- name: Push Cachegrind benchmark results
bench: Implement first class support for callgrind (#570) Since callgrind allows to control stats collection from the guest, this allows us to reset the collection right before the benchmark starts. This change exposes this to the benchmark runner and integrates callgrind data parsing into bench.py, so that we can run bench.py with --callgrind argument and, as long as the runner was built with callgrind support, we get instruction counts from the run. We convert instruction counts to seconds using 10G instructions/second rate; there's no correct way to do this without simulating the full CPU pipeline but it results in time units on a similar scale to real runs.
2022-07-04 19:13:07 +01:00
if: matrix.os == 'ubuntu-latest'
Feat: Added retry functionality to benchmarks (#667) Resolves #668 ## The problem Benchmarks jobs run concurrently for the different operating systems. This means that when it comes time to push the benchmark results to [the assigned benchmark results repo](https://github.com/luau-lang/benchmark-data), there can be two different jobs trying to push changes at the same time. In such a case, one of the pushes will fail and we end up missing some benchmark results data from the workflow run. ## The solution Whenever a push fails, we need to retry the steps leading up to the push (checking out the benchmark results repo, storing benchmark results, pushing the results to [a specific repo](https://github.com/luau-lang/benchmark-data)). ### Note There are 3 push attempts before submitting to failure. ## TL;DR This PR retries pushing benchmark results when they fail to get pushed (often due to pushing from multiple jobs concurrently) Co-authored-by: Jamie Kuppens <reshurum@gmail.com> Co-authored-by: Ignacio Falk <flakolefluk@gmail.com>
2022-09-12 16:45:55 +01:00
id: pushBenchmarkCachegrindAttempt1
continue-on-error: true
uses: ./.github/workflows/push-results
bench: Implement first class support for callgrind (#570) Since callgrind allows to control stats collection from the guest, this allows us to reset the collection right before the benchmark starts. This change exposes this to the benchmark runner and integrates callgrind data parsing into bench.py, so that we can run bench.py with --callgrind argument and, as long as the runner was built with callgrind support, we get instruction counts from the run. We convert instruction counts to seconds using 10G instructions/second rate; there's no correct way to do this without simulating the full CPU pipeline but it results in time units on a similar scale to real runs.
2022-07-04 19:13:07 +01:00
with:
Feat: Added retry functionality to benchmarks (#667) Resolves #668 ## The problem Benchmarks jobs run concurrently for the different operating systems. This means that when it comes time to push the benchmark results to [the assigned benchmark results repo](https://github.com/luau-lang/benchmark-data), there can be two different jobs trying to push changes at the same time. In such a case, one of the pushes will fail and we end up missing some benchmark results data from the workflow run. ## The solution Whenever a push fails, we need to retry the steps leading up to the push (checking out the benchmark results repo, storing benchmark results, pushing the results to [a specific repo](https://github.com/luau-lang/benchmark-data)). ### Note There are 3 push attempts before submitting to failure. ## TL;DR This PR retries pushing benchmark results when they fail to get pushed (often due to pushing from multiple jobs concurrently) Co-authored-by: Jamie Kuppens <reshurum@gmail.com> Co-authored-by: Ignacio Falk <flakolefluk@gmail.com>
2022-09-12 16:45:55 +01:00
repository: ${{ matrix.benchResultsRepo.name }}
branch: ${{ matrix.benchResultsRepo.branch }}
token: ${{ secrets.BENCH_GITHUB_TOKEN }}
path: "./gh-pages"
bench_name: ${{ matrix.bench.title }} (CacheGrind)
bench_tool: "roblox"
bench_output_file_path: "./${{ matrix.bench.script }}-output.txt"
bench_external_data_json_path: "./gh-pages/dev/bench/data-${{ matrix.os }}.json"
- name: Push Cachegrind benchmark results (Attempt 2)
if: matrix.os == 'ubuntu-latest' && steps.pushBenchmarkCachegrindAttempt1.outcome == 'failure'
id: pushBenchmarkCachegrindAttempt2
continue-on-error: true
uses: ./.github/workflows/push-results
with:
repository: ${{ matrix.benchResultsRepo.name }}
branch: ${{ matrix.benchResultsRepo.branch }}
token: ${{ secrets.BENCH_GITHUB_TOKEN }}
path: "./gh-pages"
bench_name: ${{ matrix.bench.title }} (CacheGrind)
bench_tool: "roblox"
bench_output_file_path: "./${{ matrix.bench.script }}-output.txt"
bench_external_data_json_path: "./gh-pages/dev/bench/data-${{ matrix.os }}.json"
- name: Push Cachegrind benchmark results (Attempt 3)
if: matrix.os == 'ubuntu-latest' && steps.pushBenchmarkCachegrindAttempt2.outcome == 'failure'
id: pushBenchmarkCachegrindAttempt3
continue-on-error: true
uses: ./.github/workflows/push-results
with:
repository: ${{ matrix.benchResultsRepo.name }}
branch: ${{ matrix.benchResultsRepo.branch }}
token: ${{ secrets.BENCH_GITHUB_TOKEN }}
path: "./gh-pages"
bench_name: ${{ matrix.bench.title }} (CacheGrind)
bench_tool: "roblox"
bench_output_file_path: "./${{ matrix.bench.script }}-output.txt"
bench_external_data_json_path: "./gh-pages/dev/bench/data-${{ matrix.os }}.json"
bench: Implement first class support for callgrind (#570) Since callgrind allows to control stats collection from the guest, this allows us to reset the collection right before the benchmark starts. This change exposes this to the benchmark runner and integrates callgrind data parsing into bench.py, so that we can run bench.py with --callgrind argument and, as long as the runner was built with callgrind support, we get instruction counts from the run. We convert instruction counts to seconds using 10G instructions/second rate; there's no correct way to do this without simulating the full CPU pipeline but it results in time units on a similar scale to real runs.
2022-07-04 19:13:07 +01:00
static-analysis:
name: luau-analyze
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest]
bench:
- {
script: "run-analyze",
timeout: 12,
title: "Luau Analyze",
cachegrindTitle: "Performance",
cachegrindIterCount: 20,
}
benchResultsRepo:
- { name: "luau-lang/benchmark-data", branch: "main" }
Feat: Added retry functionality to benchmarks (#667) Resolves #668 ## The problem Benchmarks jobs run concurrently for the different operating systems. This means that when it comes time to push the benchmark results to [the assigned benchmark results repo](https://github.com/luau-lang/benchmark-data), there can be two different jobs trying to push changes at the same time. In such a case, one of the pushes will fail and we end up missing some benchmark results data from the workflow run. ## The solution Whenever a push fails, we need to retry the steps leading up to the push (checking out the benchmark results repo, storing benchmark results, pushing the results to [a specific repo](https://github.com/luau-lang/benchmark-data)). ### Note There are 3 push attempts before submitting to failure. ## TL;DR This PR retries pushing benchmark results when they fail to get pushed (often due to pushing from multiple jobs concurrently) Co-authored-by: Jamie Kuppens <reshurum@gmail.com> Co-authored-by: Ignacio Falk <flakolefluk@gmail.com>
2022-09-12 16:45:55 +01:00
bench: Implement first class support for callgrind (#570) Since callgrind allows to control stats collection from the guest, this allows us to reset the collection right before the benchmark starts. This change exposes this to the benchmark runner and integrates callgrind data parsing into bench.py, so that we can run bench.py with --callgrind argument and, as long as the runner was built with callgrind support, we get instruction counts from the run. We convert instruction counts to seconds using 10G instructions/second rate; there's no correct way to do this without simulating the full CPU pipeline but it results in time units on a similar scale to real runs.
2022-07-04 19:13:07 +01:00
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v3
with:
token: "${{ secrets.BENCH_GITHUB_TOKEN }}"
- name: Build Luau
run: make config=release luau luau-analyze
- uses: actions/setup-python@v4
with:
python-version: "3.9"
architecture: "x64"
- name: Install python dependencies
run: |
sudo pip install requests numpy scipy matplotlib ipython jupyter pandas sympy nose
- name: Install valgrind
run: |
sudo apt-get install valgrind
- name: Run Luau Analyze on static file
bench: Implement luau-analyze and luau --compile benchmarks (#575) This change adds another file for benchmarking luau-analyze and sets up benchmarks for both non-strict/strict modes for analysis and all three optimization levels for compilation performance. To avoid issues with race conditions on repository update we do all this in the same job in benchmark.yml. To be able to benchmark both modes from a single file, luau-analyze gains --mode argument which allows to override the default typechecking mode. Not sure if we'll want this to be a hard override on top of the module-specified mode in the future, but this works for now.
2022-07-05 16:23:09 +01:00
run: sudo python ./bench/measure_time.py ./build/release/luau-analyze bench/other/LuauPolyfillMap.lua | tee ${{ matrix.bench.script }}-output.txt
bench: Implement first class support for callgrind (#570) Since callgrind allows to control stats collection from the guest, this allows us to reset the collection right before the benchmark starts. This change exposes this to the benchmark runner and integrates callgrind data parsing into bench.py, so that we can run bench.py with --callgrind argument and, as long as the runner was built with callgrind support, we get instruction counts from the run. We convert instruction counts to seconds using 10G instructions/second rate; there's no correct way to do this without simulating the full CPU pipeline but it results in time units on a similar scale to real runs.
2022-07-04 19:13:07 +01:00
- name: Run ${{ matrix.bench.title }} (Cold Cachegrind)
bench: Implement luau-analyze and luau --compile benchmarks (#575) This change adds another file for benchmarking luau-analyze and sets up benchmarks for both non-strict/strict modes for analysis and all three optimization levels for compilation performance. To avoid issues with race conditions on repository update we do all this in the same job in benchmark.yml. To be able to benchmark both modes from a single file, luau-analyze gains --mode argument which allows to override the default typechecking mode. Not sure if we'll want this to be a hard override on top of the module-specified mode in the future, but this works for now.
2022-07-05 16:23:09 +01:00
run: sudo ./scripts/run-with-cachegrind.sh python ./bench/measure_time.py "${{ matrix.bench.cachegrindTitle}}Cold" 1 ./build/release/luau-analyze bench/other/LuauPolyfillMap.lua | tee -a ${{ matrix.bench.script }}-output.txt
bench: Implement first class support for callgrind (#570) Since callgrind allows to control stats collection from the guest, this allows us to reset the collection right before the benchmark starts. This change exposes this to the benchmark runner and integrates callgrind data parsing into bench.py, so that we can run bench.py with --callgrind argument and, as long as the runner was built with callgrind support, we get instruction counts from the run. We convert instruction counts to seconds using 10G instructions/second rate; there's no correct way to do this without simulating the full CPU pipeline but it results in time units on a similar scale to real runs.
2022-07-04 19:13:07 +01:00
- name: Run ${{ matrix.bench.title }} (Warm Cachegrind)
bench: Implement luau-analyze and luau --compile benchmarks (#575) This change adds another file for benchmarking luau-analyze and sets up benchmarks for both non-strict/strict modes for analysis and all three optimization levels for compilation performance. To avoid issues with race conditions on repository update we do all this in the same job in benchmark.yml. To be able to benchmark both modes from a single file, luau-analyze gains --mode argument which allows to override the default typechecking mode. Not sure if we'll want this to be a hard override on top of the module-specified mode in the future, but this works for now.
2022-07-05 16:23:09 +01:00
run: sudo bash ./scripts/run-with-cachegrind.sh python ./bench/measure_time.py "${{ matrix.bench.cachegrindTitle}}" 1 ./build/release/luau-analyze bench/other/LuauPolyfillMap.lua | tee -a ${{ matrix.bench.script }}-output.txt
bench: Implement first class support for callgrind (#570) Since callgrind allows to control stats collection from the guest, this allows us to reset the collection right before the benchmark starts. This change exposes this to the benchmark runner and integrates callgrind data parsing into bench.py, so that we can run bench.py with --callgrind argument and, as long as the runner was built with callgrind support, we get instruction counts from the run. We convert instruction counts to seconds using 10G instructions/second rate; there's no correct way to do this without simulating the full CPU pipeline but it results in time units on a similar scale to real runs.
2022-07-04 19:13:07 +01:00
Feat: Added retry functionality to benchmarks (#667) Resolves #668 ## The problem Benchmarks jobs run concurrently for the different operating systems. This means that when it comes time to push the benchmark results to [the assigned benchmark results repo](https://github.com/luau-lang/benchmark-data), there can be two different jobs trying to push changes at the same time. In such a case, one of the pushes will fail and we end up missing some benchmark results data from the workflow run. ## The solution Whenever a push fails, we need to retry the steps leading up to the push (checking out the benchmark results repo, storing benchmark results, pushing the results to [a specific repo](https://github.com/luau-lang/benchmark-data)). ### Note There are 3 push attempts before submitting to failure. ## TL;DR This PR retries pushing benchmark results when they fail to get pushed (often due to pushing from multiple jobs concurrently) Co-authored-by: Jamie Kuppens <reshurum@gmail.com> Co-authored-by: Ignacio Falk <flakolefluk@gmail.com>
2022-09-12 16:45:55 +01:00
- name: Push static analysis results
id: pushStaticAnalysisAttempt1
continue-on-error: true
uses: ./.github/workflows/push-results
bench: Implement first class support for callgrind (#570) Since callgrind allows to control stats collection from the guest, this allows us to reset the collection right before the benchmark starts. This change exposes this to the benchmark runner and integrates callgrind data parsing into bench.py, so that we can run bench.py with --callgrind argument and, as long as the runner was built with callgrind support, we get instruction counts from the run. We convert instruction counts to seconds using 10G instructions/second rate; there's no correct way to do this without simulating the full CPU pipeline but it results in time units on a similar scale to real runs.
2022-07-04 19:13:07 +01:00
with:
repository: ${{ matrix.benchResultsRepo.name }}
Feat: Added retry functionality to benchmarks (#667) Resolves #668 ## The problem Benchmarks jobs run concurrently for the different operating systems. This means that when it comes time to push the benchmark results to [the assigned benchmark results repo](https://github.com/luau-lang/benchmark-data), there can be two different jobs trying to push changes at the same time. In such a case, one of the pushes will fail and we end up missing some benchmark results data from the workflow run. ## The solution Whenever a push fails, we need to retry the steps leading up to the push (checking out the benchmark results repo, storing benchmark results, pushing the results to [a specific repo](https://github.com/luau-lang/benchmark-data)). ### Note There are 3 push attempts before submitting to failure. ## TL;DR This PR retries pushing benchmark results when they fail to get pushed (often due to pushing from multiple jobs concurrently) Co-authored-by: Jamie Kuppens <reshurum@gmail.com> Co-authored-by: Ignacio Falk <flakolefluk@gmail.com>
2022-09-12 16:45:55 +01:00
branch: ${{ matrix.benchResultsRepo.branch }}
bench: Implement first class support for callgrind (#570) Since callgrind allows to control stats collection from the guest, this allows us to reset the collection right before the benchmark starts. This change exposes this to the benchmark runner and integrates callgrind data parsing into bench.py, so that we can run bench.py with --callgrind argument and, as long as the runner was built with callgrind support, we get instruction counts from the run. We convert instruction counts to seconds using 10G instructions/second rate; there's no correct way to do this without simulating the full CPU pipeline but it results in time units on a similar scale to real runs.
2022-07-04 19:13:07 +01:00
token: ${{ secrets.BENCH_GITHUB_TOKEN }}
path: "./gh-pages"
Feat: Added retry functionality to benchmarks (#667) Resolves #668 ## The problem Benchmarks jobs run concurrently for the different operating systems. This means that when it comes time to push the benchmark results to [the assigned benchmark results repo](https://github.com/luau-lang/benchmark-data), there can be two different jobs trying to push changes at the same time. In such a case, one of the pushes will fail and we end up missing some benchmark results data from the workflow run. ## The solution Whenever a push fails, we need to retry the steps leading up to the push (checking out the benchmark results repo, storing benchmark results, pushing the results to [a specific repo](https://github.com/luau-lang/benchmark-data)). ### Note There are 3 push attempts before submitting to failure. ## TL;DR This PR retries pushing benchmark results when they fail to get pushed (often due to pushing from multiple jobs concurrently) Co-authored-by: Jamie Kuppens <reshurum@gmail.com> Co-authored-by: Ignacio Falk <flakolefluk@gmail.com>
2022-09-12 16:45:55 +01:00
bench_name: ${{ matrix.bench.title }}
bench_tool: "roblox"
bench_output_file_path: "./${{ matrix.bench.script }}-output.txt"
bench_external_data_json_path: "./gh-pages/dev/bench/data-${{ matrix.os }}.json"
- name: Push static analysis results (Attempt 2)
if: steps.pushStaticAnalysisAttempt1.outcome == 'failure'
id: pushStaticAnalysisAttempt2
continue-on-error: true
uses: ./.github/workflows/push-results
bench: Implement first class support for callgrind (#570) Since callgrind allows to control stats collection from the guest, this allows us to reset the collection right before the benchmark starts. This change exposes this to the benchmark runner and integrates callgrind data parsing into bench.py, so that we can run bench.py with --callgrind argument and, as long as the runner was built with callgrind support, we get instruction counts from the run. We convert instruction counts to seconds using 10G instructions/second rate; there's no correct way to do this without simulating the full CPU pipeline but it results in time units on a similar scale to real runs.
2022-07-04 19:13:07 +01:00
with:
Feat: Added retry functionality to benchmarks (#667) Resolves #668 ## The problem Benchmarks jobs run concurrently for the different operating systems. This means that when it comes time to push the benchmark results to [the assigned benchmark results repo](https://github.com/luau-lang/benchmark-data), there can be two different jobs trying to push changes at the same time. In such a case, one of the pushes will fail and we end up missing some benchmark results data from the workflow run. ## The solution Whenever a push fails, we need to retry the steps leading up to the push (checking out the benchmark results repo, storing benchmark results, pushing the results to [a specific repo](https://github.com/luau-lang/benchmark-data)). ### Note There are 3 push attempts before submitting to failure. ## TL;DR This PR retries pushing benchmark results when they fail to get pushed (often due to pushing from multiple jobs concurrently) Co-authored-by: Jamie Kuppens <reshurum@gmail.com> Co-authored-by: Ignacio Falk <flakolefluk@gmail.com>
2022-09-12 16:45:55 +01:00
repository: ${{ matrix.benchResultsRepo.name }}
branch: ${{ matrix.benchResultsRepo.branch }}
token: ${{ secrets.BENCH_GITHUB_TOKEN }}
path: "./gh-pages"
bench_name: ${{ matrix.bench.title }}
bench_tool: "roblox"
bench_output_file_path: "./${{ matrix.bench.script }}-output.txt"
bench_external_data_json_path: "./gh-pages/dev/bench/data-${{ matrix.os }}.json"
- name: Push static analysis results (Attempt 3)
if: steps.pushStaticAnalysisAttempt2.outcome == 'failure'
id: pushStaticAnalysisAttempt3
continue-on-error: true
uses: ./.github/workflows/push-results
bench: Implement first class support for callgrind (#570) Since callgrind allows to control stats collection from the guest, this allows us to reset the collection right before the benchmark starts. This change exposes this to the benchmark runner and integrates callgrind data parsing into bench.py, so that we can run bench.py with --callgrind argument and, as long as the runner was built with callgrind support, we get instruction counts from the run. We convert instruction counts to seconds using 10G instructions/second rate; there's no correct way to do this without simulating the full CPU pipeline but it results in time units on a similar scale to real runs.
2022-07-04 19:13:07 +01:00
with:
Feat: Added retry functionality to benchmarks (#667) Resolves #668 ## The problem Benchmarks jobs run concurrently for the different operating systems. This means that when it comes time to push the benchmark results to [the assigned benchmark results repo](https://github.com/luau-lang/benchmark-data), there can be two different jobs trying to push changes at the same time. In such a case, one of the pushes will fail and we end up missing some benchmark results data from the workflow run. ## The solution Whenever a push fails, we need to retry the steps leading up to the push (checking out the benchmark results repo, storing benchmark results, pushing the results to [a specific repo](https://github.com/luau-lang/benchmark-data)). ### Note There are 3 push attempts before submitting to failure. ## TL;DR This PR retries pushing benchmark results when they fail to get pushed (often due to pushing from multiple jobs concurrently) Co-authored-by: Jamie Kuppens <reshurum@gmail.com> Co-authored-by: Ignacio Falk <flakolefluk@gmail.com>
2022-09-12 16:45:55 +01:00
repository: ${{ matrix.benchResultsRepo.name }}
branch: ${{ matrix.benchResultsRepo.branch }}
token: ${{ secrets.BENCH_GITHUB_TOKEN }}
path: "./gh-pages"
bench_name: ${{ matrix.bench.title }}
bench_tool: "roblox"
bench_output_file_path: "./${{ matrix.bench.script }}-output.txt"
bench_external_data_json_path: "./gh-pages/dev/bench/data-${{ matrix.os }}.json"
bench: Implement first class support for callgrind (#570) Since callgrind allows to control stats collection from the guest, this allows us to reset the collection right before the benchmark starts. This change exposes this to the benchmark runner and integrates callgrind data parsing into bench.py, so that we can run bench.py with --callgrind argument and, as long as the runner was built with callgrind support, we get instruction counts from the run. We convert instruction counts to seconds using 10G instructions/second rate; there's no correct way to do this without simulating the full CPU pipeline but it results in time units on a similar scale to real runs.
2022-07-04 19:13:07 +01:00
Feat: Added retry functionality to benchmarks (#667) Resolves #668 ## The problem Benchmarks jobs run concurrently for the different operating systems. This means that when it comes time to push the benchmark results to [the assigned benchmark results repo](https://github.com/luau-lang/benchmark-data), there can be two different jobs trying to push changes at the same time. In such a case, one of the pushes will fail and we end up missing some benchmark results data from the workflow run. ## The solution Whenever a push fails, we need to retry the steps leading up to the push (checking out the benchmark results repo, storing benchmark results, pushing the results to [a specific repo](https://github.com/luau-lang/benchmark-data)). ### Note There are 3 push attempts before submitting to failure. ## TL;DR This PR retries pushing benchmark results when they fail to get pushed (often due to pushing from multiple jobs concurrently) Co-authored-by: Jamie Kuppens <reshurum@gmail.com> Co-authored-by: Ignacio Falk <flakolefluk@gmail.com>
2022-09-12 16:45:55 +01:00
- name: Push static analysis Cachegrind results
if: matrix.os == 'ubuntu-latest'
id: pushStaticAnalysisCachegrindAttempt1
continue-on-error: true
uses: ./.github/workflows/push-results
with:
repository: ${{ matrix.benchResultsRepo.name }}
branch: ${{ matrix.benchResultsRepo.branch }}
token: ${{ secrets.BENCH_GITHUB_TOKEN }}
path: "./gh-pages"
bench_name: ${{ matrix.bench.title }}
bench_tool: "roblox"
bench_output_file_path: "./${{ matrix.bench.script }}-output.txt"
bench_external_data_json_path: "./gh-pages/dev/bench/data-${{ matrix.os }}.json"
- name: Push static analysis Cachegrind results (Attempt 2)
if: matrix.os == 'ubuntu-latest' && steps.pushStaticAnalysisCachegrindAttempt1.outcome == 'failure'
id: pushStaticAnalysisCachegrindAttempt2
continue-on-error: true
uses: ./.github/workflows/push-results
with:
repository: ${{ matrix.benchResultsRepo.name }}
branch: ${{ matrix.benchResultsRepo.branch }}
token: ${{ secrets.BENCH_GITHUB_TOKEN }}
path: "./gh-pages"
bench_name: ${{ matrix.bench.title }}
bench_tool: "roblox"
bench_output_file_path: "./${{ matrix.bench.script }}-output.txt"
bench_external_data_json_path: "./gh-pages/dev/bench/data-${{ matrix.os }}.json"
- name: Push static analysis Cachegrind results (Attempt 2)
if: matrix.os == 'ubuntu-latest' && steps.pushStaticAnalysisCachegrindAttempt2.outcome == 'failure'
id: pushStaticAnalysisCachegrindAttempt3
continue-on-error: true
uses: ./.github/workflows/push-results
with:
repository: ${{ matrix.benchResultsRepo.name }}
branch: ${{ matrix.benchResultsRepo.branch }}
token: ${{ secrets.BENCH_GITHUB_TOKEN }}
path: "./gh-pages"
bench_name: ${{ matrix.bench.title }}
bench_tool: "roblox"
bench_output_file_path: "./${{ matrix.bench.script }}-output.txt"
bench_external_data_json_path: "./gh-pages/dev/bench/data-${{ matrix.os }}.json"
Reference in a new issue Copy permalink