OpenSource.net

I know it can be exciting to hit some big numbers and celebrate your project’s hundredth or even thousandth star or fork. While popularity measurements might be interesting and exciting, they don’t often provide any real insight into project health or indicate areas of improvement for your project. So feel free to celebrate those milestones, but don’t fall into the trap of thinking that they indicate anything meaningful about your project.

A better approach is to focus on project health, instead of popularity. Project health metrics can help us understand which specific aspects of a project are healthy. If they are not, we can make improvements and then measure the impact of those improvements to make sure that they made the difference that we were expecting, and if not, we can try something else. Equally important, project health metrics also help indicate where you don’t need to make changes. If some area of your project is already healthy and doing well, you’re better off focusing on other areas where you can improve.

One of the hardest things about measuring project health, and metrics in general, is that you could look at so many different metrics that a lot of people get overwhelmed. To help people get over this hurdle, I created a Starter Project Health Metrics Model with four simple metrics. This model and the metrics contained in it are defined within CHAOSS, an Open Source project focused on understanding Open Source community health on a global scale. This metrics model gives people a starting point that they can build on over time.

Contributor data

This metric is often called the Bus Factor or Lottery Factor based on the idea that if one person disappears after winning the lottery, your project may no longer continue to be healthy and viable. There are a couple of things this graph can tell you. First, how big of an issue is your current contributor situation? If one person is making most of the contributions, you should focus on ways to distribute the load and have more people involved in the project. Second, you also might find that there are people who are contributing more than you realized, which is another reason that this is a good metric. This can help you think about who you can encourage to contribute more or maybe find someone who could move into a leadership role. Reaching out to someone and acknowledging their work while encouraging them to do more can help you recruit more people who could take on increasing responsibilities within your project.

With respect to leadership roles, you might look at that person making ten percent of commits and decide that they are ready to become a maintainer. People who are making around five to six percent of commits might be good candidates for mentorship or becoming reviewers with an eye toward making them maintainers after they gain a bit more experience.

The catch here and with many metrics is that we don’t want to just think about people who are making code contributions. This is a good start, but you should also be thinking about how you can move people into leadership positions to be responsible for things that might not show up in GitHub or GitLab, like documentation, community management, marketing and other important roles.

Responsiveness

It’s important for projects to respond to pull requests in a timely manner because a quick response can help you retain contributors who otherwise might become discouraged if they don’t receive a timely response. Timely, thoughtful and kind responses to contributors indicate that you appreciate their work.

While quick responses are important, it’s also important to keep up with change requests (pull requests/merge requests) and resolve them in a timely manner, even if the response is closing requests that will not be merged. It’s easy to get behind on incoming contributions, and we all get behind sometimes, but not addressing these contributions promptly creates technical debt and reduces the chances that they will ever be merged because older change requests are likely to have so many merge conflicts, becoming too difficult to accept.

In both of these responsiveness metrics, it’s important to focus on the trends. If responsiveness is already improving, keep up the good work. However, if you see responsiveness declining, then it might be time to find ways to improve it, including recruiting more contributors and maintainers for your project – perhaps from your contributor data discussed earlier.

Releases

Finally, it’s important to look at the frequency of releases including all of the releases, even the tiny point releases. It’s critical that security updates and bug fixes land in a release in a timely manner and it’s important to get those new features out too. An appropriate release frequency for your project is influenced by the size of your project and how many dependencies you have on other projects that are releasing fixes. 

Important considerations for measuring and using project health data

When looking at project health, it’s important to remember that every project is a little different, so be sure to interpret the metrics in light of a project’s needs. Project health metrics should never be used to punish people, instead, they should be used to help teams make improvements and become as healthy as possible. Keep in mind that data and metrics dashboards won’t be perfect; you’ll find inconsistencies and inaccuracies, so you’ll always want to apply a bit of common sense and a reality filter based on your knowledge of the project. Additionally, it’s important to consider all of the various types of contributions happening in and around your project that occur beyond the repository and might not be reflected in the metrics. Remember that there are actual people included in many metrics, so consent and ethical treatment of the data and results are important considerations.

In summary, every project is a little different, so the way that the metrics in the Starter Project Health Metrics Model (and most other metrics) should be interpreted will vary by project based on the size of the project, its maturity, and many other factors. Always remember that no metric is perfect, but if something looks out of place, it’s probably worth investigating.

Note: All internal images are from the CHAOSS project and are found in our repo under the MIT license: https://github.com/chaoss/wg-metrics-models/blob/main/LICENSE

Cover photo by patricia serna on Unsplash