Certificates, Expiry and the Morning Everyone Fails at Once

Monday, 08:05, at Northgate Retail. Rosa opens the queue and finds three tickets with the same sentence in them: "the VPN says the connection is not trusted". By 08:12 there are twelve. By 08:20 there are thirty, and the regional managers are phoning the desk directly. They do their Monday stock review from home before driving to their stores. The ticket form is, from their point of view, not answering. Rosa searches the desk's own notes and finds a ticket from the spring, marked resolved, titled "renew gateway certificate". It was renewed. The new certificate was issued, downloaded, and saved to a folder. The ticket was closed on the strength of the download. Nobody installed it. The old one expired at midnight, which it had been scheduled to do for exactly one year, and every laptop in the organisation noticed at once.

A certificate, in plain words, is the signed identity document the gateway shows to every client that connects. The certificate proves that the gateway is the office and not something pretending to be the office. It is issued by an authority the laptop already trusts, and it carries an expiry date. When that date passes, every client refuses to connect, correctly, within the same minute. The morning that follows is the most predictable incident a help desk will ever have. The date was printed on the document the day it was issued. This article is part of our VPN & Remote Access Tickets series. It is about turning that morning into a calendar entry, a change record and a runbook. It is also about what to do on the morning when all three were missed.

It covers what a certificate is and why everyone fails at once, then expiry as a calendar event rather than an incident. Next come the other certificates that expire and client version drift. We cover the renewal as a change, the renewal runbook, then the notice before the change. We finish with the morning it was missed and the incident shape, a short war story, and the certificate calendar to copy, in that order.

What a Certificate Is, and Why Everyone Fails at Once

Think of the certificate as an identity card the gateway holds up every time a laptop arrives at the door. The card has the gateway's name on it, an expiry date, and the signature of an issuer. The laptop accepts the card only if it already trusts the issuer and the date has not passed. That is the one analogy this article uses, and it covers the three ways the card fails. It may have expired. It may have been signed by an issuer the laptop has never heard of. Or the name on it may not match the door. All three produce a warning on the client that says, in one wording or another, that the connection cannot be trusted. And the warning is the laptop doing its job.

The reason everyone fails at once is that the card is one card. A password expires per person, a laptop breaks per laptop, a home connection fails per house, and the tickets from those arrive one at a time across a month. The gateway certificate is checked by every client on every connection. So when it expires, the next connection attempt from every remote worker fails within the same few minutes. The queue goes from quiet to forty in the time it takes to make a coffee. There is no gradual version of this fault. It is the only ticket category in the series that arrives as a crowd.

The warning must never be clicked past, and users should be told so in the setup guide, in one sentence: the laptop is refusing to trust something, and the fix is to make the something trustworthy again, not to teach the laptop to stop asking. Some clients do not even offer the choice, which is correct. Those that do are offering the user a way to connect to anything that claims to be the office. That is precisely what the certificate exists to prevent. A desk that tells users to click through in order to clear the queue has solved Monday by creating a worse Tuesday.

Expiry Is a Calendar Event, Not an Incident

The expiry date of a certificate is known on the day it is issued, to the minute. That makes it the one fault in this whole library that can be scheduled out of existence. A certificate calendar is a register of every certificate the organisation's remote access depends on. It records each certificate's expiry date, its owner, a backup owner, the lead time the renewal needs, and reminder dates before expiry. It is kept somewhere the whole desk can see and reviewed at a fixed point each month. It is a one-page document, and it is the template at the end of this article. A desk that has it does not have the Monday morning. A desk that has it in one person's head has the Monday morning when that person leaves.

The register needs an owner and a backup owner by role, not by name. The reason the certificate at Northgate expired was not that nobody cared. It was that the calendar reminder went to the mailbox of a technician who had left in the spring, and nobody had thought to move it. Owner by role, with the role's current holder written beside it and updated when people change, means the reminder lands on a person who is still employed. It is a small thing. It is the whole thing.

The reminders should be set at three points. The fictional desk in this article uses ninety days, thirty days and seven days before expiry, which is a choice rather than a standard. A desk whose renewals need an external issuer and a purchase order might start earlier. The ninety-day reminder opens a ticket to plan the renewal as a change. The thirty-day reminder is when the new certificate should already be in hand. The seven-day reminder is the last check that it has been installed and tested, not merely obtained, which is the distinction that Northgate's spring ticket missed. Each reminder opens a ticket rather than sending an e-mail, so that it sits in the queue until someone closes it with a note saying what was done.

Review the register monthly, in a fixed slot. Have the review consist of three questions: is every certificate on it, is every date right, and is every owner still here. The first question is the one that matters most, because the certificates that expire by surprise are never the ones on the list. They are the ones somebody set up for a project, with a long expiry, and forgot. Walk the gateway owner through the register once a quarter and ask what else has a date on it. Something always does.

The Other Certificates That Expire

The gateway's own certificate is the famous one, but the same tunnel may depend on three or four others, and each has a date. Some organisations issue a certificate to every laptop, or to every user. That lets the gateway check the laptop's identity card as well as showing its own. Those certificates expire per device or per person, so they produce a trickle of individual tickets rather than a crowd. Each one looks like a cannot-connect ticket until someone reads the warning. The issuer that signs all of them has a certificate of its own, usually with a long life and an expiry that nobody alive remembers setting. And the gateway's management interface, its sign-in portal if it has one, and the service that checks the second factor may each carry one.

The per-device certificate produces one particular ticket worth knowing by sight: the laptop that was in a drawer. A device switched off for four months, lent to a contractor, or kept as a spare, comes back with a certificate that expired while it slept. The user who picks it up cannot connect and has no idea why, because the laptop was "fine when it was put away". The same laptop usually also has an old client, which the next section covers. Put a line in the known-issue list for it. Add a line in the loaner process that says a loaner laptop is connected to the VPN before it is handed out, not after.

The contractor's certificate, where one is issued, should expire with the contract, and that is a feature rather than a fault. A contractor whose access ends on the date it should is the offboarding series working as designed. The ticket to expect is the contractor whose contract was extended by a manager who told nobody. And the fix is the extension in the right place, not a longer certificate. Say so in the note, kindly. The manager was not trying to break anything; the system was doing what the paperwork said.

Client Version Drift

Client version drift is the gap that opens between the version of the VPN client the desk supports and the versions actually installed on the laptops out there. It is the reason a certificate renewal done perfectly can still produce a Monday morning for part of the organisation. A renewed certificate may come from a newer issuer, or use a newer method, or arrive alongside a gateway update. And a client that is old enough will not trust it, or will not speak to the updated gateway at all. The desk renews on time, tests on a desk laptop with a current client, and declares success. The eleven laptops with the client from two years ago fail on Monday anyway.

The defence is to know what is out there before the renewal. That means asking the gateway owner for a count of connecting clients by version from the gateway's own connection records. Most gateways keep those in some form. The exact way to get the count is the gateway owner's business. The desk's business is to ask for it at the ninety-day reminder and to turn the answer into a list of laptops to update before the change. A desk that cannot get the count can at least ask its own setup guide what version it publishes. That desk can also ask, at the thirty-day mark, every remote user to check theirs against it. The reply rate will be poor. The non-repliers are the list.

A renewal, therefore, is also a client-update project, and the two should be planned as one change with the client update first. Update the published version on the setup guide, and push or ask for the update over a fortnight. Renew the certificate once the count of old clients is small enough to handle as individual tickets on the morning. Laptops in drawers will still be missed. Handle them as the per-device section above says, with a line in the known-issue list and a loaner process that connects before lending. The drawer is where version drift goes to wait.

The Renewal Is a Change

Renewing the gateway certificate changes one thing that every remote worker depends on. That is the definition of a change that deserves a record. The record is one page, with what is changing, when, who is doing it, how it will be tested, how it will be undone, and who was told. The one-page form and the reasoning behind each line are in The One-Page Change Record. The certificate renewal is the cleanest example a small desk will ever have of a change that is routine, low-effort and high-impact at the same time. The impact line reads "every remote worker" and that line alone decides the window.

The window is a time when a failure would be noticed and fixed before it hurt, which rules out the two times people instinctively choose. Not Friday evening, because a renewal that breaks old clients will be discovered on Monday morning by the whole organisation at once, with the person who did it unreachable. Not Monday at eight, for the same reason in the other direction. A fictional desk picks a mid-week morning, early enough that the desk is in and the gateway owner is at a keyboard. The old certificate is still valid for days. The desk picks that window because a renewal done early has a rollback and one done on the last day does not.

That is the point of renewing early: the old certificate is the rollback. Renew with three or four weeks still to run. Then, if the new certificate is refused by half the clients, the old one goes back on the gateway in ten minutes. Everyone reconnects while the desk finds out why. Renew on the last day and there is nothing to go back to. The change record's rollback line should say, in plain words, "reinstall the previous certificate, which remains valid until the date on the calendar". If it cannot say that, the change is late and the record should say that too.

Two people do the renewal: one at the gateway and one testing from outside, on a known-good connection. The outside tester uses a current client and the oldest client the organisation still has. The test is not "it works for me"; it is "it works for the oldest laptop we found in the count". Write both results in the change record with the time. The record is what Rosa will search for in a year, and "done" is not a result.

The Renewal Runbook

The runbook is the procedure the change record points at, written for a competent stranger at the gateway. Every action is followed by its check and its expected output, in the form Writing the Runbook sets out. The outline below is generic, because every gateway has its own screens and the runbook must use yours. The shape, the checks and the order are the part that carries over. Keep it agents-only, keep it to a page, and write the date it was last run at the bottom.

• Confirm the calendar entry: the certificate name, where it is installed, its expiry date, and that the old one still has at least three weeks to run.
• Obtain the new certificate from the issuer. Check that the name on it matches the gateway's public name exactly, and that the issuer is one current clients trust.
• Save a copy of the current certificate and the current gateway settings somewhere the second person can reach; this is the rollback.
• Post the planned-maintenance notice to remote users and put the known-issue line on the desk's internal page.
• Install the new certificate on the gateway; check the gateway reports the new expiry date.
• Test from outside with a current client: connect, open the internal test page, disconnect. Expected: no warning, test page loads.
• Test from outside with the oldest client in use. Expected: the same; if a warning appears, roll back and open a client-update task.
• Test from a phone with the client installed, if the organisation uses one. Expected: the same.
• Update the calendar entry with the new expiry date and the reminder dates; update the change record with the test results and times.
• Post the all-clear, and watch the queue for an hour for warnings from laptops the count missed.

The runbook has a stop line, and it is the oldest-client test. A desk that passes the first test and skips the second has tested the renewal on the one laptop guaranteed to work. Everything the runbook asks for before the install costs ten minutes: the copy of the old certificate, the notice, and the known-issue line. Those preparations are the difference between a failed renewal that is a quiet rollback and one that is a morning.

The Notice Before the Change

Remote users should hear about the renewal before it happens, in a short notice that says when, what they might see, and what to do. The form, tone and timing of a notice people will read rather than delete are in Planned-Maintenance Notices People Actually Read. For a certificate renewal the content is three lines. "On Wednesday morning between eight and nine we are renewing the security certificate on the remote access gateway. You may be asked to reconnect once. If you see a warning that the connection is not trusted, do not click past it; disconnect, wait five minutes, try again, and if it persists tell the desk which message you saw." Nobody needs the word certificate explained to them in the notice. They need to know that a reconnect is expected and a warning is not.

The desk itself gets a notice too, which is a line on the internal page or the queue. It says the renewal is happening, what the warning will look like if it goes wrong, and who to call. A technician who picks up the first "not trusted" ticket at 08:10 on Wednesday should recognise it from that line in a second. They should know that the ticket goes to the change, not up the can't-connect ladder. The known-issue line is the cheapest piece of communication on the whole desk and the one most often skipped. The person doing the change knows what is happening and forgets that nobody else does.

Pair the notice with the self-check that Reducing VPN Tickets describes. That gives a user who sees the warning one page to read before raising a ticket. That page says "if it is Wednesday morning, wait five minutes". A notice that goes out on Tuesday afternoon with that link in it removes most of the tickets a renewal would otherwise produce. That includes the ones from people who will never read the notice, because their colleague did.

The Morning It Was Missed

When the calendar failed, or did not exist, the morning has a shape, and the shape is an incident rather than a run of tickets. Recognise it in the first five: the same words, within the same ten minutes, from people in different places. At that point the desk stops answering tickets one at a time. Every reply it sends will be out of date in a minute. And every minute spent on one requester is a minute the other thirty are not told anything. Declaring, and what the first fifteen minutes should contain, is the subject of Declaring an Incident and the First Fifteen Minutes. The certificate morning is its textbook case, because the cause is almost always obvious and the communication is almost always the hard part.

One ticket becomes the incident, and every report is linked to it. That way, one update reaches every requester and the month's count shows one cause rather than thirty-eight symptoms. If you run QuerySolv, the reports can be linked to the incident ticket and answered from it. The event history on the incident shows the minute the first report arrived, which the review will want. The first notice goes out within five minutes of recognising the pattern, before the cause is confirmed. It says only what is known: remote access is failing for everyone, the desk knows, the next update is at a stated time. The wording and the discipline of saying something quickly are in The First Notice: Say Something in Five Minutes. The regional managers phoning the desk are not angry that the VPN is down. They are angry that nobody has said so.

The fix itself is usually quick, if the new certificate exists: install it, and the clients reconnect. If it does not exist, the gateway owner obtains one, which can take minutes or most of a day depending on the issuer. The desk's job in that gap is the update cadence, a message at the promised time whether or not there is news. Clients that have cached the refusal may need a disconnect and reconnect, and a few will need a restart. The all-clear should say so. Then the review, in the shape Closing the Incident and the Review gives it, with one question at its centre: how did the date get lost? The answer is always a person, a mailbox or a ticket closed too early, and the fix is always the calendar. The morning cost Northgate about forty technician hours and a Monday of thirty store reviews done late. The calendar entry that would have prevented it took four minutes to write.

A Short War Story: Eleven Laptops at Bellweather Logistics

Kofi, who ran the desk at Bellweather Logistics, renewed the gateway certificate on time, from a calendar he had built himself. He tested it from his own laptop before closing the change. The new certificate came from a different issuer than the old one. The eleven depot laptops that had never been updated because they lived in loading bays with no wireless did not trust it. On Monday morning, eleven depot managers could not scan deliveries into the stock system. The depots ran on paper manifests until a technician had driven to each one with an update on a memory stick. The cost was a day of paper at eleven depots, six hours of driving, and an evening of re-keying manifests into the system. Kofi's calendar had worked perfectly. The change record had one line in the test section, which read "tested, OK". The laptop it was tested on was the newest in the building. The lesson he added to the runbook, in capitals, was that the renewal is tested on the oldest client. The lesson also said that a count of client versions is the first step of the change, not an afterthought.

The Certificate Calendar

The calendar below is one page, kept where the whole desk can see it and reviewed in a fixed monthly slot. Use one block per certificate. The gateway's own goes first, then any per-device or per-user issuer. Next come the management and portal certificates, then anything else with a date the gateway owner can think of. Owner and backup are roles with the current holder beside them. The three reminder dates each open a ticket, and the last line of each block records the last renewal with its change record number. That lets the next person read how it went. Fill it in with the gateway owner in the room; they know dates the desk has never heard of.

CERTIFICATE AND EXPIRY CALENDAR — ONE PAGE
Reviewed monthly on: ____________   Last reviewed by: ______________   Date: ______________

BLOCK 1 — THE GATEWAY CERTIFICATE
  What it secures: remote access gateway, public name ______________________________
  Installed on: ______________________   Issued by: ______________________
  Expires on: ______________   Lifetime: ____ months   Renewal lead time: ____ weeks
  Owner (role / current holder): ______________________ / ______________________
  Backup (role / current holder): ______________________ / ______________________
  Reminders (each opens a ticket):  ____ days before   ____ days before   ____ days before
  Clients affected by renewal: all remote users   Oldest client version in use: ______
  Renewal runbook: ______________   Last renewed: ______________   Change record: ______
  Tested on oldest client: yes / no   Notice sent: yes / no   Notes: ______________________

BLOCK 2 — THE DEVICE OR USER CERTIFICATE ISSUER (if used)
  What it secures: ______________________________   Installed on: ______________________
  Expires on: ______________   Owner / holder: ______________________ / ______________
  Backup / holder: ______________________ / ______________   Reminders: ____ / ____ / ____
  What happens at expiry: ______________________________________________
  Last renewed: ______________   Change record: ______

BLOCK 3 — PER-DEVICE OR PER-USER CERTIFICATES (if used)
  Lifetime: ____ months   Issued at: enrolment / setup / other: ______________
  Expiry handling: automatic renewal / manual, by ______________________
  Known-issue line for the laptop that was in a drawer: yes / no
  Loaner laptops connected before handing out: yes / no

BLOCK 4 — OTHER CERTIFICATES WITH A DATE
  Name: ______________________  Secures: ______________________  Expires: __________  Owner: __________
  Name: ______________________  Secures: ______________________  Expires: __________  Owner: __________
  Name: ______________________  Secures: ______________________  Expires: __________  Owner: __________

MONTHLY REVIEW — THREE QUESTIONS
  [ ] Is every certificate on this page?  (ask the gateway owner what else has a date)
  [ ] Is every expiry date right?  (read it from the gateway, not from memory)
  [ ] Is every owner and backup still here, and is the reminder going to them?

Rosa built the page on the Tuesday after the morning, with the gateway owner beside her. She found four certificates she had not known existed, one of which was due in six weeks. The spring ticket's closing note was amended to say that a certificate is renewed when it is installed and tested, not when it is downloaded. The runbook's step five now says so in bold. The regional managers got an apology, a plain explanation, and the date of the next renewal. That is the only kind of apology that stops the next phone call.

Frequently Asked Questions