etabench is an attempt to make a benchmark for comparing ETA algorithms. This is useful not just to pick the best one with more than just a guess work, but also to improve them by making changes and evaluating their effects.

See this post for more detailed explanation of what this is and how it works.

Prerequisites

• C++ compiler that supports a suitable subset of C++20 (e.g., GCC 11)
• xmake build system
• (optional) gnuplot (for --plot and --montage)
• (optional) ImageMagick (for --montage)

Building and running

Simple

$ xmake

in the root directory should be enough. The first time it will ask to confirm installation of library dependencies, agree to continue the build.

etabench executable will be copied to the root directory as part of the build process and can be run without arguments:

$ ./etabench --algs \*
Profile \ Alg        #1        #2        #3        #4        #5        #6        #7        #8        #9        #10       #11       #12       #13       #14
Constant        98.01:4   93.68:5  100.00:1  100.00:1  100.00:1  100.00:1  100.00:1  100.00:1  100.00:1   99.76:2   99.65:3  100.00:1  100.00:1  100.00:1
Falling 15°     75.79:1   75.74:2   66.29:5   63.47:9   63.70:8   63.31:10  60.21:13  61.98:11  64.47:6   60.98:12  64.02:7   60.21:13  68.92:4   70.03:3
Falling 30°     75.42:1   68.02:2   57.79:5   52.90:9   52.93:8   52.70:10  50.73:12  52.37:11  55.27:6   49.10:13  53.65:7   50.73:12  58.12:4   61.57:3
Falling 45°     75.16:1   65.75:2   55.07:4   47.67:10  47.58:11  47.94:8   46.57:12  47.72:9   50.65:6   43.28:13  48.75:7   46.57:12  51.17:5   57.72:3
Falling 60°     70.96:1   69.13:2   56.47:4   47.50:10  47.04:11  47.74:8   47.50:10  47.61:9   51.16:5   43.28:12  48.13:7   47.50:10  49.16:6   57.62:3
Falling 75°     66.85:2   73.95:1   59.81:3   49.29:7   47.25:10  50.62:5   49.29:7   47.99:9   50.01:6   44.38:11  48.47:8   49.29:7   49.29:7   58.75:4
Raising 15°     90.86:1   11.11:11  14.43:8   15.95:6   13.24:10  17.25:4   14.30:9   15.41:7   16.70:5   17.68:3    9.99:12   5.47:13   5.47:13  19.71:2
Raising 30°     91.51:1    7.51:11  11.42:7   12.35:5    9.69:9   13.13:3   11.33:8    9.56:10  12.75:4   12.27:6    6.11:12   3.21:13   3.21:13  14.29:2
Raising 45°     84.61:1    5.46:11   8.68:7    9.25:5    7.30:9    9.79:3    8.63:8    6.63:10   9.57:4    8.85:6    4.22:12   2.30:13   2.30:13  10.52:2
Raising 60°     83.56:1    6.84:11  11.11:6   11.77:5    9.24:9   12.48:3   11.04:7    8.52:10  12.26:4    9.80:8    5.43:12   3.27:13   3.27:13  13.51:2
Raising 75°     74.67:1    7.24:11  12.34:6   13.02:5   10.12:8   13.87:3   12.25:7    9.19:9   13.60:4    8.87:10   6.01:12   4.09:13   4.09:13  14.96:2
Random          78.92:9   89.69:6   98.50:3   98.77:1   90.38:5   86.91:8   98.73:2   95.30:4   61.25:11  54.54:13  59.02:12  86.94:7   72.85:10  38.52:14
Replay          85.43:7   96.81:4   79.42:11  82.05:9   98.29:2   63.55:13  80.39:10  92.52:6   72.13:12  98.25:3   94.37:5   99.21:1   82.32:8   19.87:14
Saw             56.60:1   20.94:11  31.24:5   33.14:3   26.63:8   34.75:2   31.32:4   24.83:10  30.95:6   26.07:9   15.85:12   9.25:13   8.95:14  28.28:7
Square          73.38:11  87.23:5   88.40:4   91.27:2   89.06:3   91.58:1   84.09:7   84.76:6   56.75:13  76.56:9   79.01:8   76.27:10  59.41:12  23.32:14
Step            73.32:1    9.13:11  11.69:9   14.90:4   12.88:6   15.10:3   11.53:10  12.81:7   13.74:5   12.29:8    6.62:12   3.75:13   3.75:13  15.81:2

Algorithm ranking
  1. 78.44% - Acceleration
  2. 49.26% - Firefox
  3. 47.67% - Switch
  4. 46.46% - Window 20 1
  5. 45.33% - Smoothing 0.10
  6. 45.05% - LookBack 20
  7. 44.87% - Average
  8. 44.83% - Gravity
  9. 41.95% - Combined[ Average Immediate ]
 10. 41.62% - Exponential 1
 11. 40.58% - Slowness 1
 12. 40.50% - AveChangeLimit
 13. 38.89% - ImmChangeLimit
 14. 37.78% - Immediate

Selecting algorithms

Pass --list-algs to see numbered list of available algorithms. Then use --algs to specify subset of algorithms. Argument to --algs is a comma-separated list of numbers or special value * (all algorithms) or an empty string (default set of algorithms).

Some algorithms are considerably worse than others and enabling them by default only clutters output and especially the plots. Use --algs \* to enable all algorithms.

$ ./etabench --list-algs
Algorithms:
 1. Acceleration
 2. Average
 3. Firefox
 4. Gravity
 5. Immediate
 6. LookBack 20
 7. Smoothing 0.10
 8. Switch
 9. Window 20 1
10. AveChangeLimit
11. Combined[ Average Immediate ]
12. Exponential 1
13. ImmChangeLimit
14. Slowness 1
$ ./etabench --algs 4,7,12

Selecting profiles

Similar to algorithms, but the options are called --list-profs and --profs and no profiles are disabled by default.

$ ./etabench --list-profs
Profiles:
 1. Constant
 2. Falling 15°
 3. Falling 30°
 4. Falling 45°
 5. Falling 60°
 6. Falling 75°
 7. Raising 15°
 8. Raising 30°
 9. Raising 45°
10. Raising 60°
11. Raising 75°
12. Random
13. Replay
14. Saw
15. Square
16. Step
$ ./etabench --profs 12,13

Plotting to separate files

Running

$ ./etabench --plot plots

will create plots directory with a set of directories that correspond to the set of profiles. Each directory will contain a PNG plot and its gnuplot source.

Plotting to a single file

Running

$ ./etabench --montage montage.png

will do --plos /some/temp/dir and then combine result into a single PNG using montage command from ImageMagick.

Debugging

Manually:

$ ./etabench --profs 6 --algs 4 --verbose

With gdb:

$ xmake f -m debug
$ xmake
$ gdb --args ./etabench --profs 6 --algs 4

License

Version 3 of the GNU General Public License.