The Slow Application: Is It You, Us or Them?

Tuesday, 15:20, at Meridian Parts. A search in the parts catalogue takes twenty-five seconds in the warehouse office. Sana has the stopwatch figures to prove it. They are twenty-five, twenty-four and twenty-six on Grace's machine. They are twenty-five on the machine beside it, twenty-five on the spare laptop plugged into the same socket. So it is not the machine. The vendor's support desk has replied that their monitoring "shows all services healthy". The network contractor has replied that the line "is performing within contract". Grace, who runs the warehouse and has been waiting a week, has replied that the desk is "passing this around". Sana has three replies, a number nobody disputes, and no idea whose problem it is. That last part is the subject of this article.

The slow application ticket is the one where the number does not stay with a machine. The spare at the same desk is as slow as the user's, so the cause is somewhere behind the screen. Behind the screen there are three parties. You is the user's side, the machine and the habits, mostly ruled out by the spare. Us is the organisation's own equipment: the application server, its database, the file service it reads from, and the network between the desk and all of them. Them is the vendor, who hosts the application on their own servers or supports the one on yours. There is also a fourth suspect that belongs to nobody and is blamed by nobody: the data. It has been growing since the application was installed. This article is part of our The Vague Slowness Complaint series. This article starts where the spare laptop test in the first article ends. It is about finding which of the four it is, with evidence the owner of that one will accept. It leaves the network's own diagnosis to the slow-internet series, which owns it.

The article covers the three suspects and the fourth, then the vendor-hosted application and the server in the cupboard. Next come the network branch and why this article stops at its edge, the data that grew for years, and the report that was always slow. Then come the evidence an application owner will accept and who owns the ticket while it waits. A short war story and the application slowness evidence sheet to copy finish the article.

The Three Suspects and the Fourth

Each suspect has a test, and the tests are cheap. So the article runs them in the order of cost. "You" is tested by the spare laptop. The ticket arrived here because the spare was slow too, so the user's machine is innocent. The only part of "you" still in play is the task itself, which is where the data comes in. "Us" is tested by timing the task from as close to the server as the desk can get. That means at the server's own console if the owner will do it, or from a wired machine in the same room. "Them" is tested by timing the task from outside the organisation's network altogether, on a phone's mobile connection or from a home connection. That way, none of "us" is in the path. The data is tested by making the task smaller and timing it again.

The table below is the sorting once those timings exist. It will not always resolve to one row. That is because a slow server on a slow network behind a slow vendor is possible and has happened to me. But the sorting nearly always resolves to one row first. The first row is where the ticket goes.

The Vendor-Hosted Application: "Them"

A vendor-hosted application runs on the vendor's servers, reached over the internet. The desk can see exactly none of it: not the server, not the database, not the load from the vendor's other customers. What the desk can see is the time a task takes from two places, and that is enough. The first place is the user's desk, where the number already exists. The second is a connection that does not go through the organisation's network at all. That could be a phone on its mobile connection, a laptop tethered to it, or a colleague working from home. If the task is slow from both, the slowness is on the far side of the internet and belongs to the vendor. If it is slow from the office and quick from the phone, the slowness is in the office's path to the internet and the vendor is innocent.

Ask the vendor two questions in writing, and ask them before the ticket has aged a week. First, is the slowness affecting all customers right now? Their status page or their support desk can answer that, and it turns the ticket into waiting. Second, what figure do they consider normal for the task, measured from where? Ask because "all services healthy" is a statement about their monitoring. Their monitoring measures their servers from inside their own building. Suppose a vendor's own figure is four seconds for a search. That search takes twenty-five from every machine in your warehouse and twenty-five from a phone on the pavement outside. That vendor has a problem on their side. The two timings are the sentence that gets it escalated past the first-line reply.

Go through the account manager when the support desk stalls. Support desks at vendors are measured on tickets closed, and "monitoring shows healthy" closes a ticket. Account managers are measured on renewals. A customer with an evidence sheet and a renewal date is a different conversation. Keep the tone plain. You are not accusing anyone; you are reporting two numbers and asking what theirs is. (Vendor status pages have never, in my experience, shown a problem before the first customer reported one. That is not cynicism. It is how status pages are updated.)

The Server in the Cupboard: "Us"

An application hosted on the organisation's own server is "us". The first question is who owns the server. On a small desk the answer may be the same person who is reading this. In that case, the owner conversation is with the mirror. The server has its own view of itself, the equivalent of the task monitor on a desktop. It shows processor, memory, disk activity and network, over time. The slowness ticket's time of day is matched against that view, and the match is usually obvious. It might be the backup running at nine, or the report that a department schedules at the start of every hour. Or it might be the anti-malware scan on the server itself, or the second application that was moved onto the same host last spring "temporarily".

Time the task at the console. If the owner opens the application on the server itself, or on a wired machine beside it, and the search takes twenty-five seconds there too, the network is innocent and the server is the suspect. In that case, its database has grown past what its memory can hold, its disk is full or slow, or it is sharing a host with something hungry. If the search takes two seconds at the console and twenty-five from the warehouse, the server is innocent and the path is the suspect. The next section covers that. That one timing, taken by the owner in a minute, is the single most useful number in the whole ticket. It is the one most desks never ask for. That is because asking feels like an accusation. Ask anyway. Owners are rarely offended by a stopwatch.

The server that is quietly starving is the hardest "us" case because nothing is broken. The application was sized for the data it had when it was installed and for a server that had nothing else to do. Both have changed by degrees. The evidence is the server's own view over a month, with memory full, disk busy and the application's database the thing that is busy. The fix is a conversation about memory, a faster disk, moving the squatter application off, or archiving, which is the data section. None of those is a desk fix. All of them start with the desk's timings and the server's chart on the same page.

The Network Branch, and Why This Article Stops at Its Edge

The sign that the network is the suspect is a task that is quick near the server and slow far from it. Or it can be quick from one site and slow from another, or quick on a wired desk and slow on wireless in the same room. Any of those takes the ticket out of this article and into the slow-internet series. It has an isolation card for proving which branch the slowness is in. It also has an evidence sheet for measuring the connection with the three numbers a network owner or provider will accept. Use the measurement article there for the readings. Do not try to measure a network with a stopwatch, because a stopwatch measures the task and a network is measured in other units.

What this article keeps is the handoff. The application timings go across with the ticket. The network owner needs to know that the task is two seconds at the console and twenty-five on the floor, and that it is twenty-five on the spare too. That pair of numbers is what stops the network owner from asking the desk to rebuild the user's machine first. The two series meet in the ticket, and the ticket carries both sets of evidence. Neither series needs the other's instruments, and neither should rebuild them.

The Data That Grew for Years

An application that searches, sorts or reports over everything it holds gets slower as it holds more. The growth is invisible because it is gradual. A search that took one second in the application's first year, over a thousand parts, takes twenty-five in its eighth, over four hundred thousand. And no single month was the month it broke. The machine is innocent, the server may be innocent, and the vendor's "normal" was measured on a demonstration database the size of a spreadsheet. The suspect is the task, and the task is bigger than it was.

The test is to make the task smaller and time it again. The same search limited to the current year's parts; the same report over one month instead of all time; the same list filtered to one warehouse before it loads. If the smaller task is quick, the data is the cause, and the two timings side by side are the whole case. The fix is one of three things, none of them the desk's. One is archiving old records out of the live set, which the data's owner decides. Another is an index or a setting on the vendor's side that makes the search cheaper, which is a vendor case with the two timings attached. Or it is a change of habit, so that the search is filtered before it runs rather than after. The desk shows the numbers and offers the conversation.

Whose data it is matters, and the answer is never the desk. The parts catalogue belongs to the warehouse. The sales history belongs to sales. The patient records belong to the clinic. The desk's job is to show that the task is slow because of its size, to show what the smaller task costs, and to ask the owner what can be archived. Give the plain warning that archived is not deleted. Ask the plain question of who will need the old rows and when. Owners who hear "your data is the problem" argue. Owners who see twenty-five seconds beside two seconds and are asked "which of these would you like?" usually choose.

The Report That Was Always Slow

Some tasks were slow on the day the application was installed and will be slow until it is retired. The month-end report that runs over everything is the usual one. It is not a defect; it is a report doing a great deal of work. The test from Expectation or Defect applies. If it took four minutes last year and takes four minutes now and the vendor's guide says "several minutes", it is an expectation. In that case, the ticket's work is the conversation and the known-slow list rather than a repair. If it took four minutes last year and takes twelve now, it is the data section, and the growth is the suspect.

The practical fix for a report that was always slow is to run it when nobody is waiting for it. It can be scheduled overnight, or started before the coffee. Or it can be run by one person for the whole department instead of by eleven people at nine on the first of the month. The desk can suggest the schedule and the application owner can usually set it. What the desk cannot do is make the report fast. The honest line to the requester says so: "this report reads every order since the system went in, it has always taken about four minutes, and the quickest way to have it at nine is to have it run at six."

Evidence the Application Owner Will Accept

Whoever the ticket goes to, the vendor, the server's owner, the data's owner or the network owner, will ask the same six things. The evidence sheet exists so that the answers are already in the ticket before they ask. The exact task, in words a stranger could repeat. The timings: three at the user's desk with the time of day, one on the spare, and whichever of the console timing and the outside-the-network timing applies. From where: the site, the room, wired or wireless. The smaller-task timing, if the data is suspected. Since when, and what changed around then. And what else has been tried, with the result, so that nobody is asked to restart the user's machine for the fourth time.

The sheet is boring on purpose. An application owner who receives "the warehouse say it's really slow and they're fed up" has been given a feeling and will reply with one. An owner who receives a task, five timings, two places and a date has been given something they can reproduce. Most will reproduce it before replying. That is the moment the ticket starts moving. Keep opinions out of it. Keep the user's frustration out of it. And keep the desk's own theory out of it or put it in one line at the bottom marked as a theory. The numbers persuade; the theory gives the owner something to disagree with.

Who Owns the Ticket While It Waits

The ticket stays with the desk until the requester's task is fast or the requester has been told, in plain words, that it will not be and why. Sending it to the vendor does not send it away. The vendor's case has a reference. The reference goes into an internal note with the date it was raised and the date the vendor promised a reply. That way, the next technician can chase it without re-reading the thread. The requester gets an update at an honest cadence whether or not there is news. That is because a week of silence while the vendor "investigates" is a week in which Grace decides the desk has forgotten her. The format for the update when the fix is a long way off is in When the Fix Is Weeks Away. The slowness version is the same with the timing in it.

Link the other tickets to it. A slow application produces several tickets from several people. Each one closed separately as "raised with vendor" is a count the desk has thrown away. Linked to one parent, they are the evidence of how many people the slowness touches. That is the number the vendor's account manager and your own budget holder both respond to. In QuerySolv the parent and the linked tickets are ordinary tickets with an ordinary history. A requester's reply to any of them reopens it. That is the right behaviour for a problem that comes and goes. Most tools can link or merge in some fashion. The test is whether one technician can see the whole set in one place.

Close it with the numbers. When the vendor's fix lands, or the data is archived, or the server gets its memory, time the task again from the same desk. Put the before and after in the resolving reply: "the search took about twenty-five seconds and now takes three". The requester gets a receipt, the vendor's case gets its proof, and the known-slow list gets an entry removed. A ticket closed with "vendor says resolved" and no number is a ticket that will be reopened the first morning the search takes nine.

A Short War Story: The Morning Job Board at Calder Print

Calder Print is a commercial printer with a desk of one. A technician called Marek spent six weeks on a ticket from the production manager, Eileen. It said the job-ticketing system was unusable every morning. During those weeks, the vendor blamed the connection, and the connection's contractor blamed the vendor's server. Marek relayed each reply to the other and to Eileen, who began copying the managing director on her replies. Nobody had timed the task from anywhere but the production floor. When Marek finally timed the job board at the server's console it opened in two seconds. On the wired machine in the office next to the server it opened in three, on the production floor's wired machine in four. On the floor's wireless handsets it opened in forty, at nine, when the printing presses started. That was when the floor's one access point began to drop frames behind them. The slowness had been the warehouse access point from the first day. The slow-connection isolation card would have found that in a morning. Neither the vendor nor the contractor had been wrong about their own equipment. The cost was six weeks, a vendor escalation that went nowhere because it had nothing to escalate, and Eileen's confidence, which the managing director's involvement did not help. Timing the task from both ends before ringing anyone would have taken fifteen minutes and one walk.

The Application Slowness Evidence Sheet

The sheet below is the internal note for any slowness ticket where the spare was slow too. Section A restates the task and the desk timings from the intake card. B is the three sorting timings, console, outside the network and narrowed task, with the row of the sorting table each one points at. C is the handoff, with the owner, the reference and the dates. D is the close. The sheet is sent to the owner as it stands, without the theory line unless you want to be argued with. It stays in the ticket until the after-timing in D is filled.

APPLICATION SLOWNESS — EVIDENCE SHEET
Ticket: ________   Application: ______________   Hosted by: us / vendor   Owner: ______________
Requester: ______________   Site / room: ______________   Wired / wireless   Linked tickets: ____

A. THE TASK AND THE DESK TIMINGS (from the intake card)
   Task, in the user's words: ______________________________________________
   Watch starts at: ______________   stops at: ______________ (usable)
   At the user's desk: ____ s / ____ s / ____ s   middle ____ s   time of day ______
   On the spare at the same desk: ____ s
   Since when: ______________   What changed then: ______________

B. THE SORTING TIMINGS
   1. Near the server (console, or wired beside it; taken by ______): ____ s
      Quick here, slow at the desk ............ the path: slow-connection isolation card
      Slow here too, server is ours ........... us: the server (section C, owner on our side)
   2. From outside our network (phone's mobile connection or home): ____ s
      Slow here too, vendor-hosted ............ them: the vendor (section C, vendor case)
      Quick here, slow from the office ........ us: office path to the internet (slow-connection sheet)
   3. The task made smaller (range / filter: ______________): ____ s
      Quick when smaller ...................... the data: owner conversation on archiving or indexing
   4. Earlier timing of the same task, if any: ____ s on ________
      Same number, vendor's guide agrees ...... expectation: known-slow list and the conversation
   What else has been tried, with the timing after each: ______________________________

C. THE HANDOFF
   Sent to: vendor / server owner / data owner / network owner   Name: ______________
   Reference: ________   Raised on: ________   Reply promised by: ________   Chased on: ________
   Requester update cadence: every ____ days, with or without news   Next update due: ________
   Other requesters linked to this ticket: ____   Workaround given: ______________________

D. THE CLOSE
   Fix applied by the owner: ______________________   on: ________
   Same task, same desk, after: ____ s   (before: ____ s)   timed by: ______
   Before and after sent to the requester: [ ]   Known-slow list updated: [ ]   Linked tickets closed: [ ]

Sana's sheet had twenty-five seconds in A, two seconds at the console in B1, and twenty-four from her own phone in the car park in B2. It had three seconds for a search limited to the current year's parts in B3. That answered the question before anyone had to be rung at all. The catalogue held every part the company had ever stocked, back to parts for machines nobody owned. The warehouse archived the discontinued lines in a week. The search now takes four seconds. Grace timed it herself, and has not copied anyone on anything since.

Frequently Asked Questions