Taking the risk out of certificate renewals for a major public sector agency

One expired certificate can take a critical web portal, application or server offline until someone fixes it. For a large Australian government agency running more than 3,000 servers, every renewal relied on manual tickets, emailed files and copy-and-paste installs.

Expired certificates had already caused loss of access in the past. Avocado Consulting and 2Steps replaced that process with automated certificate retrieval, installation and verification.

About our client

Our client is a large Australian public sector organisation. Its websites and customer and staff portals all depend on SSL certificates to stay secure and available.

The organisation runs more than 3,000 servers. Avocado Consulting manages its Splunk environment, including the Splunk web interface. Every website, portal and web application across the enterprise relies on a valid certificate to run.

The challenge

Splunk alerted the team when a certificate was due to expire, but everything after that alert was manual. Each renewal took three to four days, so the team had to start the process 30 days before expiry to be safe.

This was not a theoretical risk. Expired certificates had already caused users to lose access. Every renewal meant a ServiceNow ticket, a wait for the certificate, an email to an individual, and a manual install and double check on every server.

This created four distinct risks:

• Key person risk. Certificates were emailed to individuals. If that person was off sick, the whole process had to start again.
• Outage risk. When a certificate expired, the service behind it either ran with no security or stopped working. Splunk could not run without a valid certificate, so the platform was down until the certificate was fixed.
• Lockout risk. Depending on the certificate, an expiry could lock the team out of the server itself. Without a local account, there was no way in.
• Silent security risk. Installing a certificate was a simple file copy. If it landed on the wrong server, the service still worked, but with the wrong certificate. Nothing broke, so nothing alerted.

The manual process also left the team blind. With no automation, there were no metrics on how many certificates existed, how many were about to expire, or whether each one was actually in use.

Scale turns a routine task into an operational event

Certificate expiry is predictable, yet it still causes outages, because the work to prevent it depends on people, inboxes and calendars. The risk grows with every server added.

Scale turns a routine task into an operational event. If 100 certificates expired on the same day, our entire team would have to stop all other work to replace them. Across an enterprise of 3,000 servers, that exposure multiplies. Large managed service providers that run thousands of servers face the same problem many times over.

Monitoring expiry dates is only half the job. The other half is confirming the replacement was installed on the right server, for the right application. That deployment gap is where outages and hidden security exposures sit.

For any organisation running customer-facing portals, regulated services or a large server enterprise, three questions are worth asking:

• Who receives our renewed certificates, and what happens if they are away?
• How long does a renewal take end to end, and how much lead time do we need?
• How would we know if a certificate was installed on the wrong server?

Our approach

Avocado started at the source: the system where certificates are requested and created. For this client that was ServiceNow; however, the approach works the same way with any tool that organises certificate requests.

1. Connect to the source of truth. Avocado mapped where certificates were requested, stored and sent. The platform used API calls to retrieve each certificate's details from ServiceNow or any other tool: server name, the certificate itself, who requested it, when, and its expiry date. It also captured the application each certificate belongs to.
2. Automate installation. Where the platform had access to the location the certificate is stored, it retrieved the certificate and installed it on the right servers automatically. No manual copy and paste.
3. Verify every deployment. Because the platform knows which certificate was raised for which application, it checks the live application or website to confirm the correct certificate is in place. If it is not, an alert is triggered.
4. Report in one dashboard. The platform gives the team a single view of how many certificates exist, which are due to expire and whether each one is in use. It can also send that data to tools the organisation already uses, such as Splunk or Dynatrace.

The one manual step that remains is deliberate. The security team that generates certificates still validates each request before issuing it. Once issued, certificates are placed where the platform can access them, and automation takes over from there.

Where certificate automation applies

The engagement began with 2Steps pushing certificate data into Splunk, but 2Steps is not limited to one platform or one type of server. It covers every server and application that relies on a certificate.

• Websites and portals. Monitors, installs and verifies the certificates behind customer and staff-facing sites.
• Any web application. Applies the same process to any application using a certificate.
• Servers across the enterprise. Installs certificates automatically wherever the platform has access to where they are stored.
• Deployment verification. Confirms the right certificate is installed on the right server for the right application, and alerts if not.
• Certificate inventory. Tracks how many certificates exist, when each expires and which application it belongs to.
• Usage reporting. Shows whether each certificate is actually being used.
• Request tracking. Records who requested each certificate and when, pulled from the request system by API.

Technology agnostic by design

The solution works with the tools an organisation already has, rather than replacing them.

• Upstream. It connects by API to ServiceNow or any other tool used to request and create certificates.
• Downstream. Its dashboard can run in the platform itself, or the data can be sent to Splunk, Dynatrace or any other monitoring tool.

For organisations already invested in Dynatrace, that means certificate health can sit alongside the rest of their observability data, in the platform their teams already use.

The result

With automation in place, work that took three to four days of tickets, emails and manual installs happens automatically from the moment a certificate is issued.

For the client, this means:

• Fewer outage and lockout risks across its websites, portals and internal platforms.
• No single point of failure in the renewal process.
• Assurance that the right certificate is on the right server, not just that a certificate is present.
• Time back for the team to focus on higher-value work instead of chasing renewals.
• Metrics for the first time, giving security and operations a clear view of the certificate enterprise.

Once a certificate is generated, it's installed automatically and checked to confirm it's the right certificate on the right server. That's the part we can't guarantee when it's all done by hand.

Why Avocado

Avocado brings the delivery experience: mapping how certificates move through an organisation, connecting certificate automation to existing tools, and feeding the results into platforms such as Splunk and Dynatrace.

Find out where your certificate risk sits

Start with a conversation with Avocado. From there, a Discovery engagement scans and catalogues your certificate enterprise, including shadow IT, so you know exactly what you have, when it expires and where it is installed.

Start a conversation