*etabench* is an attempt to make a benchmark for comparing ETA algorithms. This
is useful not just to pick the best one with more than just a guess work, but
also to improve them by making changes and evaluating their effects.
See [this post][post] for more detailed explanation of what this is and how it
works.
### Prerequisites ###
* C++ compiler that supports a suitable subset of C++20 (e.g., GCC 11)
* [xmake][xmake] build system
* (optional) [gnuplot][gnuplot] (for `--plot` and `--montage`)
* (optional) [ImageMagick][ImageMagick] (for `--montage`)
### Building and running ###
Simple
```
$ xmake
```
in the root directory should be enough. The first time it will ask to confirm
installation of library dependencies, agree to continue the build.
`etabench` executable will be copied to the root directory as part of the build
process and can be run without arguments:
```
$ ./etabench --algs \*
Profile \ Alg #1 #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 #13 #14
Constant 98.01:4 93.68:5 100.00:1 100.00:1 100.00:1 100.00:1 100.00:1 100.00:1 100.00:1 99.76:2 99.65:3 100.00:1 100.00:1 100.00:1
Falling 15° 75.79:1 75.74:2 66.29:5 63.47:9 63.70:8 63.31:10 60.21:13 61.98:11 64.47:6 60.98:12 64.02:7 60.21:13 68.92:4 70.03:3
Falling 30° 75.42:1 68.02:2 57.79:5 52.90:9 52.93:8 52.70:10 50.73:12 52.37:11 55.27:6 49.10:13 53.65:7 50.73:12 58.12:4 61.57:3
Falling 45° 75.16:1 65.75:2 55.07:4 47.67:10 47.58:11 47.94:8 46.57:12 47.72:9 50.65:6 43.28:13 48.75:7 46.57:12 51.17:5 57.72:3
Falling 60° 70.96:1 69.13:2 56.47:4 47.50:10 47.04:11 47.74:8 47.50:10 47.61:9 51.16:5 43.28:12 48.13:7 47.50:10 49.16:6 57.62:3
Falling 75° 66.85:2 73.95:1 59.81:3 49.29:7 47.25:10 50.62:5 49.29:7 47.99:9 50.01:6 44.38:11 48.47:8 49.29:7 49.29:7 58.75:4
Raising 15° 90.86:1 11.11:11 14.43:8 15.95:6 13.24:10 17.25:4 14.30:9 15.41:7 16.70:5 17.68:3 9.99:12 5.47:13 5.47:13 19.71:2
Raising 30° 91.51:1 7.51:11 11.42:7 12.35:5 9.69:9 13.13:3 11.33:8 9.56:10 12.75:4 12.27:6 6.11:12 3.21:13 3.21:13 14.29:2
Raising 45° 84.61:1 5.46:11 8.68:7 9.25:5 7.30:9 9.79:3 8.63:8 6.63:10 9.57:4 8.85:6 4.22:12 2.30:13 2.30:13 10.52:2
Raising 60° 83.56:1 6.84:11 11.11:6 11.77:5 9.24:9 12.48:3 11.04:7 8.52:10 12.26:4 9.80:8 5.43:12 3.27:13 3.27:13 13.51:2
Raising 75° 74.67:1 7.24:11 12.34:6 13.02:5 10.12:8 13.87:3 12.25:7 9.19:9 13.60:4 8.87:10 6.01:12 4.09:13 4.09:13 14.96:2
Random 78.92:9 89.69:6 98.50:3 98.77:1 90.38:5 86.91:8 98.73:2 95.30:4 61.25:11 54.54:13 59.02:12 86.94:7 72.85:10 38.52:14
Replay 85.43:7 96.81:4 79.42:11 82.05:9 98.29:2 63.55:13 80.39:10 92.52:6 72.13:12 98.25:3 94.37:5 99.21:1 82.32:8 19.87:14
Saw 56.60:1 20.94:11 31.24:5 33.14:3 26.63:8 34.75:2 31.32:4 24.83:10 30.95:6 26.07:9 15.85:12 9.25:13 8.95:14 28.28:7
Square 73.38:11 87.23:5 88.40:4 91.27:2 89.06:3 91.58:1 84.09:7 84.76:6 56.75:13 76.56:9 79.01:8 76.27:10 59.41:12 23.32:14
Step 73.32:1 9.13:11 11.69:9 14.90:4 12.88:6 15.10:3 11.53:10 12.81:7 13.74:5 12.29:8 6.62:12 3.75:13 3.75:13 15.81:2
Algorithm ranking
1. 78.44% - Acceleration
2. 49.26% - Firefox
3. 47.67% - Switch
4. 46.46% - Window 20 1
5. 45.33% - Smoothing 0.10
6. 45.05% - LookBack 20
7. 44.87% - Average
8. 44.83% - Gravity
9. 41.95% - Combined[ Average Immediate ]
10. 41.62% - Exponential 1
11. 40.58% - Slowness 1
12. 40.50% - AveChangeLimit
13. 38.89% - ImmChangeLimit
14. 37.78% - Immediate
```
### Selecting algorithms ###
Pass `--list-algs` to see numbered list of available algorithms. Then use
`--algs` to specify subset of algorithms. Argument to `--algs` is a
comma-separated list of numbers or special value `*` (all algorithms) or an
empty string (default set of algorithms).
Some algorithms are considerably worse than others and enabling them by default
only clutters output and especially the plots. Use `--algs \*` to enable all
algorithms.
```
$ ./etabench --list-algs
Algorithms:
1. Acceleration
2. Average
3. Firefox
4. Gravity
5. Immediate
6. LookBack 20
7. Smoothing 0.10
8. Switch
9. Window 20 1
10. AveChangeLimit
11. Combined[ Average Immediate ]
12. Exponential 1
13. ImmChangeLimit
14. Slowness 1
$ ./etabench --algs 4,7,12
```
### Selecting profiles ###
Similar to algorithms, but the options are called `--list-profs` and `--profs`
and no profiles are disabled by default.
```
$ ./etabench --list-profs
Profiles:
1. Constant
2. Falling 15°
3. Falling 30°
4. Falling 45°
5. Falling 60°
6. Falling 75°
7. Raising 15°
8. Raising 30°
9. Raising 45°
10. Raising 60°
11. Raising 75°
12. Random
13. Replay
14. Saw
15. Square
16. Step
$ ./etabench --profs 12,13
```
### Plotting to separate files ###
Running
```
$ ./etabench --plot plots
```
will create `plots` directory with a set of directories that correspond to the
set of profiles. Each directory will contain a PNG plot and its `gnuplot`
source.
### Plotting to a single file ###
Running
```
$ ./etabench --montage montage.png
```
will do `--plos /some/temp/dir` and then combine result into a single PNG using
`montage` command from ImageMagick.
### Debugging ###
Manually:
```
$ ./etabench --profs 6 --algs 4 --verbose
```
With [gdb]:
```
$ xmake f -m debug
$ xmake
$ gdb --args ./etabench --profs 6 --algs 4
```
### License ###
Version 3 of the GNU General Public License.
[post]: https://reversed.top/2022-12-31/benchmarking-io-eta-algorithms/
[xmake]: https://xmake.io/#/
[gnuplot]: http://www.gnuplot.info/
[ImageMagick]: https://imagemagick.org/
[gdb]: https://www.sourceware.org/gdb/
Before first commit, do not forget to setup your git environment:
git config --global user.name "your_name_here"
git config --global user.email "your@email_here"
Clone this repository using HTTP(S):
git clone https://code.reversed.top/user/xaizek/etabench
Clone this repository using ssh (do not forget to upload a key first):
git clone ssh://rocketgit@code.reversed.top/user/xaizek/etabench
You are allowed to anonymously push to this repository.
This means that your pushed commits will automatically be transformed into a
pull request:
... clone the repository ...
... make some changes and some commits ...
git push origin master