Crafting RubyBench #1
Recently I have started contributing to Ruby Bench project. I wrote a few words about it in my previous post.
In this post I want to summarize contributions I've made during last month.
Sequel benchmarks
Finally we have Sequel benchmarks set up and running. Some results are already onsite. Last year GSoC student got most of work done here, I was happy to finish it up and make it onsite.
Comparing results between libraries
So far you could have tracked performance of various libraries and platforms present on Ruby Bench. Though you couldn't tell which one performs better or even how changing library dependencies affect its performance. Being able to compare performance between ORM libraries can be useful when deciding which one to use in your stack. How much performance is left on table when using one? You may want to take that insight into account.
Following are some comparison highlights:
- ActiveRecord and Sequel performs better with postgres.
- Sequel performs generally faster than ActiveRecord.
Further we need some comparison insight how much performance is being left on table. Therefore we need to have these benchmarks running against raw mysql and pg gems. Maybe ruby implementation comparisons would be interesting to see?
Other contributions
During last month I've made number of other smaller contributions.
- Fixed Rails benchmarks which were broken for some time.
- We can now run benchmark suite manually for number of last commits.
Next steps
As Sam Saffron suggested, we need to make our benchmarks more honest and put them in order. Next step is to add real world bench. Here's WIP regarding this.