Imagine for a moment two companies have an identical problem – the website is slow. Perhaps this is a huge problem; the company bet it’s future on a new product, that customers sign up for on the web, and performance is bad enough that people are abandoning the site and going to the competition. In a few more months, the competition will ‘own’ the market, and the percentage of customers that are even worth fighting over will be a small percentage of the total – more like 9% than the 90% of market share hoped for in the initial offering.
Like I said, there are two companies. Both appoint a point man on performance improvement and give him the speech that the future of the company is in his hands. Both companies find a war-room, an easel to write on, some markers and stickynotes. Management asks for daily reports, and steps away. Knowing the PMI playbook backwards and forwards, plus a half-dozen other techniques, both newly appointed ‘performance project managers’ are competent, smart people, and they have a plan: Figure out what component(s) are the bottleneck, and speed them up. It’s not rocket science, right?
Then differences show up.
The Response
The next thing the project managers are going to do is to schedule a meeting. They’ll want to meet with a technical person from at least four different teams: The database administrators, the system administrators, someone from network operations, and a programmer. That team will start by looking at raw performance data (from production monitoring), and work backwards to figure out what the problem is and how to fix it.
At company A, that is exactly what happens.
At company B, it’s more … complicated.
You see, at company B, Bill the project manager had to schedule a meeting, but he didn’t know who to invite. After a week, he was able to get the right people in the room, but they wanted to know what specific questions he needed to ask. The DBA brought his defense – database response times were fine; if there is a problem with a query, just ask for some help tuning it by scheduling a meeting. Network had the same comment; if there was a problem with packets, give them the data for a trace, and they’d figure it out. The programmer was convinced it was a database problem.
By the end of the meeting, all Bill has is a list of defenses.
Of course, the problem is a lot more complex than that.
The Reality
The database is a little slow, especially if you throw an underperforming query at it. And the programmer did screw up – instead of one query with a loop for the results, he executes a query to get the row ID’s, then a query for each row id to be processed. Worse than that, his program recalculates the running value of an account by adding up all transactions from the beginning of time. With these kinds of errors, each individual transaction can be fast, but the whole is a mess.
It’s not just the programmer’s fault. The network is not optimal either.
These are kinds of problems that are solvable. Get the right people in the room, find ways to profile the code, to find out where it is spending it’s time, and eliminate repetitive trips to a server.
But it just won’t happen if the staff is in four different buildings communicating by tracking tickets.
With everyone in one room, 100% dedicated, project manager A has a chance.
Project Manager B was doomed before he started.
If It Happens To You
A few times in my career I’ve been in a position like this — the company wants me to fight a battle with both hands tied behind my back. In all my years, I have learned a couple of things I can do to improve the odds.
A) Refuse the assignment. As a contractor, if I can’t sniff out success, I have the option of refusing the assignment. Regular employees have this option too, though they seldom realize it.
The thing is, project B above was a “dead fish” project. Anybody in the organization already, anyone but a new hire, would know by the second day (and probably before the assignment was made), that the project just did not have the attention, focus, or priorities of supervisors and line management, and wasn’t going to get fixed. Project Manager B could have made getting a staff member from each department, 100% dedicated to the project, a core requirement of his accepting the position. When management said “we can’t do that, Bill”, he could have said “well, then, I’m afraid I can’t be successful. Perhaps you can find someone else to take over the project.”
The key here is to realize, by sheer force of will, that you have the option to reject an assignment. This is the kind of thing you can’t fake, you must believe.
B) Negotiate for the resources you need. The other possibility is that when Bill insists on staff, he actually gets it – that senior management says “Of course, this is our priority.” If that happens, I try to send an email making our agreement clear, and follow it up the next day by asking who will be on the project team, and when can they start.
C) Recognize Failure as an option. If the organization can’t give a performance problem time or attention, then performance won’t get better. If Bill the project manager gets blocked, one thing he can do is create the daily status email. Then when the quarterly sales estimates are missed and big big meeting occurs, Bill can at least say “I’ve been reporting on the issue for six weeks. Every day I send an email on what we need to do to solve the problem and how I am blocked. I remain blocked on those issues. What’s the surprise?”
Somehow, though, I find that … unsatisfying.
My first point here is that performance improvement – reaction and fixing – is an entirely different kind of problem than performance testing. Second is that beyond tools, beyond method and process, sheer attention and focus on the problem will get you pretty far … and a lack of it will fail. If you recognize that you are headed toward becoming company B early in a project, you have more options to do more about it, to influence the team to steer toward a better outcome.
In software, we get geeked out about metrics, about tools and automation. Over the next few weeks I want to talk about that, but first, I wanted to mention three critical things for a performance project: time, attention, and a willingness to figure it out over having specific, pre-defined tasks.
More to come.