Noah is a Ruby Fellow at AppFolio and has authored countless articles and given many presentations on benchmarking Ruby and Rails. He has also built benchmarking tools, including an official benchmark for Ruby 3x3. We're grateful that he's taken the time to answer a few questions today.
What first inspired you to start benchmarking Ruby so fervently?
When I was chosen for the Ruby Fellow role, I knew I needed to work on performance. That's what they were specifically looking for! I tried a few things including different Ruby compile options, different kinds of profiling... You can see some of that in my early articles. But what worked (for me, for readers, for the Core team) turned out to be the benchmarking. Sometimes you try a few things and find a sweet spot!
What do you see as the largest impact item in Ruby 3x3 when it comes to better performance?
It's hard to narrow Ruby 3x3 performance down - there are so many little fixes! But for non-Rails code, MJIT is huge. For Rails, probably a change to how Ruby handles TraceFunc in Ruby 2.5 that added nearly 10% to the overall speed.
What's the most challenging aspect of your benchmarks?
The most challenging aspect, always, is being sure enough (but never 100%!) that I'm right. Measuring and interpretation are hard, and often I can't be sure what a result means. Re-running tests and extra measurements help, but you're never utterly sure. You've probably noticed that a lot of my conclusions sound nuanced ("X is probably true," "Y seems very unlikely.") That's why!
What's the most worrisome item on the current Ruby roadmap, in your opinion?
I'm not sure I'd call any item on the roadmap "worrisome" - progress is pretty good and we have 18 months left. Static type tools took longer to get really started, but new tools like Sorbet are in surprisingly good shape. We'll get there. I think real-world Rails performance is going to finish at more like double speed than triple. [editor: emphasis is ours] It's at about 172% speed in Ruby 2.6. That should make sense - a lot of Rails time is in the database, or libev or other C extensions and Ruby's speed doesn't affect it. In those cases I'm pretty happy with "only" doubling the end-to-end speed. Especially in a mature 20+-year-old language!