Measuring mobile: How to zero in on performance

When most people hear the phrase “mobile analytics,” they may assume it is related to measuring usage of web-enabled or web-only mobile sites. But that’s just part of the story.

Information on web traffic and other traditional usage metrics might give you an idea of how many people are coming to your site through mobile devices. What it won’t tell you, however, is the quality of their experience.

Increasingly, organizations are turning to analytics and other user-supplied data to measure the effectiveness and usability of their mobile apps and their overall mobile strategy.

“Agencies are constantly asking the question: How do we measure success?” says Kyra Fussell, Deltek’s senior research analyst. “It’s a really difficult challenge. These are long-term initiatives and when you’re dealing with technologies that are rapidly evolving success can be a bit of a moving target.”

Agencies need a way to evaluate apps and initiatives in real-time, she says. Otherwise, they risk progressing toward an end goal that might have become outdated.

Starting with the mission
But it’s not always easy for the IT department to measure the success of a mobile strategy. That is because approximately two-thirds of all mobile initiatives are being driven by the line of business or mission stakeholders, not IT, according to a November study by J. Gold Associates.

Often, the IT department has little insight into the actual mission goals that must be served by mobile apps and connectivity, while the line of business doesn’t understand – or necessarily care – about the technology problems that must be overcome by IT, says Jack Gold, an IT consultant and founder of J. Gold Associates.

To solve this disconnect, Gold suggested that every organization should create a feedback loop spearheaded by members of a mobile advisory council. This group would have representatives from human resources, legal, end users, IT, help desk and procurement. Their task would be to plot out their organizations mobile goals for the next six months to three years.

“What do we want our mobile strategy to look like, how is our mission evolving? How do we make it happen from a technology, architecture, and business planning perspective? Also, you’ll need to match it up with any mandates or initiatives that are coming down from the executive branch or federal CIO,” he says.

Gold suggests meeting once a week in the beginning and then, as the mobile policies and implementations move forward, once a month, and eventually every three months. “Strategy is never chiseled in stone.”

David Eads, founder of Mobile Strategy Partners, says it will be up to IT and the advisory board to come up with specific key performance indicators for all mobile development, especially when it comes to creating apps in-house or purchasing apps from vendors.

Whatever the source of the apps, you need to make sure the workflow they create actually supports the goals of the organizations and that they don’t support any stumbling blocks. This will require user testing that includes hands-on testing and constant user feedback.

“If you are paying employees by the hour you want to make sure they’re not wasting time stumbling through a mobile app path,” Eads says.

Still, metrics can be difficult to capture if an organization is heavily compartmentalized, says Alex Rossino, principal research analyst at Deltek. This is especially a problem when it comes to measuring the overall return on investment.

“The ROI for analytics depends on the network environment it’s being stored on. If you still have containerized data that’s siloed, you really can’t monitor what’s going on across your mobile strategy,” he says. “You need a unified network environment to use it effectively.”

This may take longer in an agency setting since many are still slowly working toward common environments, he says, but it will happen eventually.