Microsoft, in collaboration with Delft University of Technology, has developed a "Nudge" system designed to speed up the code review process by sending targeted notifications to the person blocking progress.
This system has been deployed across 8K+ repositories and sent over 210 K nudges in the span of a year.
The Nudge System: Key Components
1. Predictions Model
A machine learning-based effort estimation model predicts the time it will take to complete a pull request (PR), based on 2 years of historical data.
The model considers 28 features of PRs, highlighting the top 4 factors that slow down and speed up reviews.
2. Activity Detection
The activity detection module monitors if recent activity has taken place by the author or the reviewer of the PR, such as:
- PR state changes (e.g., approve or reject)
- Comments from the author or reviewer, including when a comment is resolved or marked as "won’t fix."
- Updates where authors push new commits to respond to feedback or add improvements. If any activity is detected within the last 24 hours, the notification process halts, ensuring only meaningful nudges are sent.
3. Actor Identification
This module identifies the "change blockers" and relevant actors who need to act on the PR.
- If the author needs to act, they are notified when:
- The PR is approved and ready to merge
- Not all review comments are addressed (the author can resolve them or mark them as "won’t fix")
- The author has pending action items from the reviewer - If the reviewer needs to act, they are notified when:
- The review hasn’t started yet
- The author has addressed comments and now the reviewer needs to re-verify the changes
The system ensures the most critical actors are notified, prioritizing the author if both parties need to take action. This prevents unnecessary delays in the code review cycle.
Finally, notifications are sent to the list of actors identified in the previous step in the form of pull request review comments and email messages.
This system design led Microsoft to achieve a 71% satisfaction rate regarding the usefulness of notifications, based on feedback from 22K pull requests. More importantly, it contributed to a 60% reduction in pull request completion time, significantly accelerating the development workflow.
Reference: Nudge: Accelerating Overdue Pull Requests toward Completion
Link: https://arxiv.org/abs/2011.12468
ReviewNudgeBot for Slack implements core ideas outlined in referenced paper.
Check it out if you are looking for a tool to accelerate your team's code reviews.
