Using `git bisect` to debug regressions with Rails codebase

Aug 16, 2016 - Prakash Murthy

git bisect is a very useful tool for identifying the offending commit when you know one commit where the functionality doesn't work as expected and another where it works.

This short blogpost on the Thoughtbot site gives a good introduction to how git bisect can be used.

However, given that the ruby on rails repository is constantly evolving, resulting in hundreds if not thousands of commits between the two reference commits, the process to use git bisect with the Rails codebase is a bit more involved. This blogpost explains the process with the help of two examples.

First Example:

Issue # 26108 - Decimals set to values over precision and scale get truncated to 0.0

Initial Analysis about issue #26108:

The Original Poster has included an executable test case script in the comment to reproduce the behavior. The OP has also mentioned that it is a regression in Rails 5.0.0 from the previous version Rails 4.2.1

Further, the OP has linked to another issue (issue # 23400) which explains that this is a behavior change between Rails versions 4.2.4 and 4.2.5.

The executable script for reproducing this issue:

Running the above script with ruby decimal_rounding_error.rb while changing the version of Rails in the script confirms the following behavior:

Rails Version Script execution status
5.0.0fail
4-2-stablefail
4.2.5fail
4.2.4pass
4.2.1pass

Setup for bisecting with git for issue #26108:

The commit that introduced this behavior change is between 4.2.4 and 4.2.5 releases. The commit shas for each of these releases can be identified from the Rails releases page as follows:

The executable script can be used to add a new test in the Rails codebase by adding the following two files (activerecord/test/models/thing.rb and activerecord/test/cases/decimal_rounding_test.rb :

The first two files above mimic the test file structure for activerecord in the Rails codebase. The third file above has the script to run the newly included test in the Rails codebase. The script can be either run by itself from the command line or invoked from the file as ./z_run.sh after making it executable.

Running the `git bisect` steps for issue #26108:

With the above in place, the git bisect process can be started off as follows:

  • git bisect start
  • git bisect good dac822ef58ae05f0e805222fa8744116080165ac
  • git bisect bad 73521d586981279a99d3ba038d62e2414125df7a

The below screenshot shows how the git bisect process proceeds till it identifies the first bad commit. Note: I am running the test script in a separate window; based on the test script execution status, I specify whether it is `good` or `bad` in this window.

Rails

Conclusion about issue #26108:

The first bad commit identified was 0b94384be. This is actually fixing a bug where the precision and scale wasn’t being respected before. So the behavior the OP is complaining about - decimal being rounded based on precision and scale provided in database - is the correct behavior. The OP also found that the correct way to validate this type of data is with attribute_before_type_cast.

Second Example

Issue # 26122 - Rails 5 ActiveRecord::Attribute stack level too deep error on save

Initial Analysis about issue #26122:

The OP has provided an executable test case script for reproducing the behavior, and has specified that it happens on Rails 5.0.0. No mention of whether it was working in previous versions of Rails.

The executable script for reproducing this issue:

Running the above script with ruby thousands_of_assignments.rb while changing the version of Rails in the script confirms the following behavior:

Rails Version Script execution status
masterfail
5.0.0fail
4.2.7.1pass

Setup for bisecting with git for issue #26122:

The commit that introduced this behavior change is between 4.2.7.1 and 5.0.0 releases. The commit shas for each of these releases can be identified from the Rails releases page as follows:

The executable script can be used to add a new test in the Rails codebase by adding the following two files (activerecord/test/models/stat.rb and activerecord/test/cases/my_assignment_test.rb :

The first two files above mimic the test file structure for activerecord in the Rails codebase. The third file above has the script to run the newly included test in the Rails codebase. The script can be either run by itself from the command line or invoked from the file as ./z_run.sh after making it executable.

Running the `git bisect` steps for issue #26122:

With the above in place, the git bisect process can be started off as follows:

  • git bisect start
  • git bisect good 26dadfb187b1cf13dc2f783198b91b32d141c3ab
  • git bisect bad 2efddadd6cba4e2129acedf1d402d11abcc03996

The below screenshots shows how the git bisect process proceeds till it identifies the first bad commit. Note: I am running the test script in a separate window; based on the test script execution status, I specify whether it is `good` or `bad` in this window.

Rails Rails

Conclusion about issue #26122:

The first bad commit identified is 07723c23a7 which is a huge change in the way attribute changes are tracked; so does look relevant. Aaron Patterson fixed the bug in a simple and elegant manner.

Additional notes about the `git bisect` process with Rails:

  • In my initial attempts, I was using the executable test case (decimal_rounding_error.rb & thousands_of_assignments.rb files above) to verify whether the Rails codebase status is good or bad while bisecting. This led to the process being hijacked frequently by unrelated errors. The executable test case file is very useful in quickly verifying the functionality across different Rails versions, but is not a good thing to use while bisecting Rails codebase.
  • Prathamesh Sonpatki helped me in this regard by showing how to convert the executable script code to a new test in the Rails code base. Working with a new test in the Rails codebase makes it for a smoother process of bisecting.
  • Prathamesh and I initially added the new test in an existing test file on the Rails code base. Turns out this is not desirable while bisecting as the contents of the file could change across versions, and that could lead to merge conflicts when the codebase changes between each step of the bisecting process. Best to add new files in the Rails codebase while bisecting to avoid this problem.
  • A few steps would require running the bundle install command when the codebase changes significantly and all the required components are not installed.
  • It is important to run the test with bundle exec to ensure that the test is using all the components within the current version of rails codebase. Without bundle exec, there could be cases when a different version of certain gems are used, leading to inconsistencies.
  • Gemfile.lock - which has been included in the recent versions of Rails codebase - could cause merge conflicts during the bisect process. Best to check that the contents of the file has not changed from the previous bundle install / bundle exec .. commands before specifying whether the bisect process is good or bad.
  • Sometimes specifying a commit at the start of the process may throw a fatal: Needed a single revision error message. This means that the relevant commit is not available in your local version of the Rails codebase. It can be fixed by pulling the specific branch that includes the relevant commit. More info about the error message here.
  • Sometimes there could be an error message complaining about not finding a compatible version of arel gem. In such cases, the way forward is to find the correct commit sha on the arel repository that corresponds to (has the same date as) the commit on the rails repository, and specify that commit reference in the Gemfile.

Conclusion:

Hope the above documentation make it easy to bisect the Rails codebase while trying to help with issues. If you know of any other tips that I missed, please let me know in the comments below, or by email at prakash@rubyappcare.com.

Have fun working with Rails issues.

Back to Blog