The Story

Where were you in February 2024? Here’s where I was:

• I worked at Hotjar.
• My son is about to be born and I headed out on parental leave on the 26th.
• On the 28th I attend a training course because my son hadn’t arrived yet. This was my last contact with my team on Slack.
• On the 29th my son is born.
• On the 1st of March I logged back on to tell my team.

I saw that nearly no deployments had happened and I found a 60+ message long Slack thread of people attempting to diagnose the broken build. This was clearly serious because Hotjar was always shipping. Easily 10-20 deployments happened on a normal day and it was commonplace for our team (Developer Experience) to get complaints if for any reason the deployment pipeline was closed for more than an hour.

I scrolled to the bottom of the chat to see what the resolution was and what caused it and it turns out… it was my fault.

My fault

I joined Hotjar in May of 2021 as a Backend Engineer, shipping features to our customers, until Jan 2023 where I shifted to the Developer Experience team, where we kept our engineers shipping fast and happy.

At some point during those three years I wrote a very standard looking test that checked whether a repository method (one that interacted with Postgres) returned the expected rows.

This method in particular would filter the rows based on a date range you passed as a paramter, something like this:

def get_foo_between_dates(start_date: datetime, end_date: datetime) -> list[Foo]:
    # Code goes here

Aligning with best practices and the strong testing culture at the time, I wrote both happy path and non-happy path tests. One of them went something like this:

1. Setup the test: create a Foo for every date in February
2. Assert that 28 Foos exist
3. Query for all Foos between 1st and 5th February (inclusive)
4. Assert that I receive 5 Foos

Pretty simple test, it did its job for three years straight… until finally there were 29 Foos instead of 28 Foos, failing the first assert.

Just to make sure I never forget this, my lovely team at the time wrote this in the card that came with the flowers celebrating my son’s birth:

Worth it.