PlayHQ Outage

First, we are sorry. This is the worst outage in PlayHQ’s history. We pride ourselves on our 99.9% uptime, enterprise-level infrastructure, and modern codebase, which supports over 1 billion page views annually for over 2 million participants in four of the top five sports in Australia.  This should not have happened (especially on the first week of the season), and we are determined to ensure we don’t drop the ball again.

What went wrong?

PlayHQ encountered a site-wide system outage on Saturday 13th April between 6 pm (AEST) and 11:30 pm (AEST).  Around one-third of users visiting  playhq.com, PlayAFL, MyHoops and NetballHQ apps saw a 404 message instead of the fixtures and match centres they sought. Those visiting the admin portal and registration flows would have found it slow and frustrating. Some users would have been affected when downloading games to score on the scoring portal, but scoring functioned without issue otherwise.

It was incredibly frustrating for those trying to print game sheets and check fixtures or scores on Saturday evening. For five hours we fell short of our goal of saving volunteers time and making the experience easy for participants and their families.

What caused it?

A change to mobile apps deployed in January 2024, were not being cached correctly. This issue was exacerbated by the significant growth in usage of our mobile apps on our biggest traffic weekend ever.

This error with app caching should have been detected in January or through further pre-weekend testing. We forecasted the weekend’s database usage assuming our caching was working as expected.

What are we doing about it?

We have not had an outage for 395 days.  We are focused on ensuring it doesn't happen this weekend or ever again with the following actions this week:

1. We have fixed the caching issue with our mobile apps.
2. As an insurance policy, we have increased processing power for this weekend
3. We are improving automated forecasting, monitoring & alerts.
4. We are increasing caching and specific request optimisations across different experiences to improve site speed.
5. We are fixing our deployment & testing processes to ensure caching is implemented properly.

Let’s fix these issues and return to our mission of unleashing the life-changing impact of community sport.