Great post arguing that tech manuals should be written by marketers not engineers.

Why do so many companies treat potential users so much better than existing users? Think about it. The brochure is a thing of beauty, while the user manual is a thing of boredom.

Like everything else I post, originally found on Seth Godin

As anyone who has applied for a loan or tried to get through a co-op board knows, there is a lot of paperwork required to verify your finances. Getting a financial packet together was my task today, and I figured it would be quick work, what with all of my financial information organized and online, it would be just a matter of click, click, click.

Backing up one second. Since the earliest days of the Internet era I've been an online-only type of guy, organizing all my accounts, employee benefits, etc through the web. My wife (as always) has been skeptical, but I insisted. "We can trust eTrade," I said. "It's just like a normal bank." Well, yes. But not in the way they intended.

Following is a brief summary of how eTrade, among other financial companies that I do business with, have failed to design their web presences to allow me to assemble my financial packet. Keep in mind, that as an online-only customer, I represent the most profitable segment of their businesses, yet their boneheaded decisions on usability, technology and security have prevented me from using their sites for common activities. Instead, I will spend quite some time on the phone with a highly paid customer service rep, and then get paper statements printed (at their expense) and sent to me by mail (on their dime). Come on guys, get with it!

Brokerage (eTrade):

• Attempt to print out recent statement, new browser window opens with nothing in it.


• Refresh browser, it takes me to the non-logged in homepage.


• Close browser.


• Click again on recent statement, new browser opens with statement. Instead of any numbers the columns are all filled with the string "NaN".

Second Account with eTrade

• Log out of first eTrade account. (eTrade has a "linking" feature so you can see multiple accounts, but you need to print out a form and get it notarized for it to actually work.)


• Repeat steps above to print out statements. Same results.

401K Account

• Login successfully.


• Navigate to "statements", click on most recent statement.


• Statement appears in a frame below the site's navbar. Prints successfully.


• Click back button to get back to list of statements (so I can print previous one). Sent to site homepage, logged-out.


• Repeat steps above for each statement to print.

Paychecks Site

• Login with correct username and password.


• Get the dreaded "password expired" screen. (When will people learn that frequently changing passwords is a security risk not a security benefit.)


• Enter old password, followed by new one. New one is 8 characters long and includes a number.


• "New password must be at least 8 characters long and include a number. Try again."


• Try again, and again, and again.


• Give up, try the "trouble logging in" link.


• "Please enter your social security number and your employee id number".


• Having no idea what my employee id number is, am forced to give up. Keep in mind that I had my correct username and password to start with and was unable to use the site.

Student Loan Site

• Login successfully.


• Search for 5 minutes for anything resembling a "Statement." Every time I click the back button the browser gives me a "Do you want to send information again" warning dialog. The site is using HTTP POST to send user data between pages. This is probably their solution for avoiding the dreaded cookies which, as we all know, are a serious privacy problem!


• Eventually find a link to "Mortgage Letter" which has the information I want. This was probably considered a usability improvement since users often want to get loan information for a mortgage, however, they forgot the part about people who might want the information for other purposes.


• While I'm here, decide that I might want to accelerate some of my loan payments. Click on "Change Payment Options."


• Nope. "Change" apparently only refers to lowering your payments. The information on this page refers to consolidation.


• Click on "Automatic Payments" since I am enrolled in auto debit. OK, this looks good. It shows me my current monthly debit along with information about debit and auto-payment.


• Click, click, click, click.


• Back, "Send", Back, "Send", Back, "Send."


• Give up. It may be possible to change the payment amount but I can't figure out how.

Business Checking Account

• Haven't logged in here lately so I look up my username and password.


• Expired! Shit.


• New password...


• "Please enter your web passcode. You should have received this from your bank manager."


• Forget it.

Don't you guys realize how much money this is costing you?

Oliver Travers pointed me to a review of the worst website ever.

Good reference for the next time someone tells me that my application is hard to use...

I'm not the first to notice that once the concept of usability has entered your brain, you have a tendency to see more and more of the world as impossibly broken. Having just got back from a business trip to Colorado, I couldn't help but notice what "worked" and what was "broken" at each stage of my trip. Herewith, a rant...

Check-in for United: The e-ticket kiosks are actually quite well-done. There seems to be an increasing effort to make these things actually work. The sneaky bastards ask me whether I would like to upgrade for only $39 and I almost click the "Yes" button out of sheer OK-momentum.

Security Check: My drivers license is expired so I can't get to the gate. I have to go back to the United check-in and get a new boarding pass, this one with large capital "S"s all down the side. "S" is for security, I imagine. Uh oh. OK, now I've got the boarding pass marked TERRORIST, I can go right through and into the special line for some cavity searching. Why couldn't they have just shunted me into that line at security? Why did I need a new boarding pass? If I was a real terrorist and got caught with an out of date ID at security, I'd hop a cab out of there and try again another day.

Side note: The TSA people are incredibly polite and nice. They seem to actually like their jobs.

Meals Not Included: I buy a $6 sandwich from the Au Bon Pain kiosk at LGA. On the plane the stewardesses offer to sell me the exact same sandwich for $10. Good thinking United, that's really the way to compete with JetBlue.

Limited Carry-Ons: When is the first airline going to buy planes specifically designed for carry-on luggage? Why not have less room in the cargo hold and double-decker carry-on bins?

Small Seats: It is virtually impossible to use a laptop in coach, even though the plane is continuing to the bay area after Colorado and virtually everyone on board wants to use a laptop.

System Tray Insanity: I'm 40,000 feet above the nearest internet connection, yet my Windows system tray wants to alert me every 5 minutes that "Outlook cannot be reached", "Wireless connection unavailable", "Network cable unplugged." I'm offline. Deal with it.

Exact Change Please: Will they ever give stewardesses a change drawer? For 40 years they have been collecting cash for drinks and they still have to run around like idiots looking for change. Here's an idea: Why not let me pay for drinks on the plane with frequent flyer miles?

Denver Airport: Get off the plane in the unfathomably huge Denver Airport. The terminal stretches endlessly in both directions. The sign on my right says "Terminal", the sign on my left says "Baggage." Hmmm.

Hotel Light Switches: Nicer hotels tend to have multiple desk lamps for a soft home-like effect. However, to the guest who has never been in the room and is arriving after dart, a single light switch at the front of the room would be far more usable.

Bath Products: What genius put high-end shampoo in a bottle with a metal screw top. A top which is essentially un-openable with soapy, wet hands.

Cabs with Cupholders: Cabs in Denver have cupholders! Why hasn't anyone else thought of this in New York?

Painkillers in Vaults: Not the first to notice this, but when you've got a monster hangover and have just been through a cavity search, you really shouldn't have to spend 5 minutes trying to open a security container for pain relief.

Isn't traveling fun?

The innocuous bookmark has been a fixture in my life for going on 6 years now. I find it interesting how many reporters and business people seem to suddenly "discover" that bookmarks are ineffective and that people have a hard time simply getting back to the sites they like.

The Times has a non-article on this subject today titled New Ways to Revisit Web Sites which can be summarized as "bookmarks suck and there are a lot of untested hypotheses for what is better."

The fundamental problem with bookmarks is that they're trying to solve two very different problems with one tool. The more common usage of browser-based bookmarks is to quickly navigate to the user's favorite sites on a regular basis. The less common usage is for creating a personal archive. These two uses have very different user profiles and are currently being accomplished in very different ways.

Quick Navigation

Although there is little research on this subject, from personal experience watching people interact with their browsers I believe a large percentage of users see bookmarks as a method of quick navigation. This usage profile is characterized by:

Bookmarks placed within the "root" directory of the list (e.g. not organized).

Bookmarks placed within a top folder labeled something like "Daily" or "Hot"

Little need for advanced tools, annotation, sharing, etc.

High satisfaction with the tools built into the browsers

A number of other solutions satisfy the quick navigation requirement, including the browser's links toolbar and autocomplete features (which is discussed in the Times article).

As far as I'm concerned, this problem is solved. It is the second problem that causes all the confusion.

Personal Archiving

Personal archiving is a much harder problem to solve and to date none of the browsers has made much progress. There was a study at some point (reference?) that found that 1/3 of all Internet users have more than 100 bookmarks; this has always been my guesstimate for the archiving audience. This archiving usage profile is characterized by:

A large number of bookmarks organized into nested folders

Annotation needs

Search needs

Desired portability between computers

Desired backup

Sharing with colleagues (rare)

Publishing on a website/XML (rare)

The archiving audience has been the subject of business plans for some time. There are a ton of shareware plugins for the browsers, online bookmark sites like my company, Blinkpro, and random ideas like the Brain, all of which are essentially about archiving stuff, organizing it, and accessing it at a later point.

I believe that the emergence of blogging and XML will effectively kill the archiving functions of bookmarks. In a sense, blogging is a form of bookmarking, with the most common format for a blog entry consisting of a link, its category, and some annotation. With creative modifications of currently-available blogging software a motivated user can create a bookmark archive which satisfies all of the user requirements listed above within a very attractive web-based interface.

I also believe that this convergence of bookmarks and blogging is the real impetus behind Google's purchase of Blogger. Imagine every link on Google including an "Add" button, linking directly to a hosted private blog organized by category. Links or categories could be marked as "Public" and displayed on a custom blog page (google.com/members/ari) with advanced features for paid members only.

With the standardization around XML and the heavy competition in the blog tool arena there's also a good chance that multiple offerings for blog/bookmarking and whole new ideas will develop.

January 22, 2004 01:23 PM | Comments (0) | Topics: Usability and Design

Boxes and Arrows has published a thorough review of their experience using WebEx for
remote online usability testing. They review WebEx since it's what they used. I've had some experience using Vividence so I thought it would be useful to add to the conversation.

Vividence offers online usability testing based around a downloaded Windows application which controls the user's browser and thereby prompts the user through a series of desired actions and related questions. In the process, the download tracks all clickstreams and records time lags between clicks.

The core product of Vividence is the Vividence XMS Suite, a powerful set of tools which allows you to test up to 800 users, include conditional logic (e.g. if the user answers "Yes" to question 1, they can skip to question 3), and professional services. When I was evaluating Vividence for a corporate client I was given a very high price quote for XMS - I think it was $30,000, but I could be mistaken. This puts XMS out of the range for the vast majority of web projects.

The more affordable solution is Vividence Express, a scaled down version of the same technology with a considerably lower cost of $1,500. Express panels are limited to 50 users and two "tasks" for testing. I used Express for my project so that's where my experience is coming from.

Caveat: The project took place about 6 months ago and I don't have access to my original documents so some of the details are approximated. Also Vividence may have updated their product in the interim, Express was fairly new when my test was completed.

Panel Selection
Because Vividence's methodology requires the use of a downloadable application, panel selection offers a difficult choice. They recommend use of their panel (who presumably have the download already), and for Express they include the recruitment cost in the fee. Many clients, however, would prefer to use their own customer list or intercept screens on their websites to recruit panelists. Although Vividence supports customer-generated lists, respondents have to download and install a program to take part in the testing, which is a considerable barrier to getting a clean sample group.

Vividence's list is very convenient to use, with segmentation based on numerous criteria accomplished through the web interface. I was able to select up to 4 criteria for list restrictions, which I limited to:

• Age range: 25 - 50


• Career: Non-professional, no PhD's


• Internet Experience: Mid, no Advanced


• Country: US

There were two problems with this list selection process. My client wanted a panel which represented the product's audience which was roughly 75% female. Vividence Express only offered me the ability to select all female or all male, no percentage make-up. Not that big of a deal, but it illustrates the sort of compromises you have to make when using someone else's list.

The more important problem with Vividence's list was its accuracy. Since the criteria were provided by the panelists themselves on sign-up, there is always going to be a problem with inaccuracies. In particular, though, I found a big disparity between my desire for a panel of average Internet usage and the panel that was actually recruited. This is probably a self-selection problem since only advanced Internet users are likely to download an application and take part in interactive surveys. Whatever the reason, the panel recruited for my survey was much more advanced in terms of Internet usage than our client's target market or probably the overall US average. When I asked the open-ended question:

"How does the design of the site make you feel?"

I got a response:

You should use more JavaScript mouse-overs.

Not exactly what I wanted from a user panel for a consumer product!

Methodology
Vividence Express offers a very nice online interface for building your testing script. The system lets you create different types of survey questions (free response, radio buttons, check-boxes, etc) without any programming and offers nice extras such as the ability to randomize the order of the multiple-choice responses and include or exclude an "other" choice. Not much room for improvement here.

The tasks users are asked to complete, however, is another story. Express scripts are limited to two tasks. In our test we first asked users to

"Go to http://thewebsite.com, and without clicking anywhere answer the following questions..."

The clickthru analysis provided by Vividence showed that more than half of the respondents actually clicked around quite a bit before answering the following questions. Users who don't follow instructions are going to be a limitation to any online testing not guided by a human tester.

The second task went more smoothly, but I felt very limited by the two task limit. Yes, I was able to get some good insights into the usability of the front page and the key function of the site, but the lesser functions remained wholly untested.

Reporting
Vividence's reporting system was excellent, offering raw data, graphs, and cross-tabs as required. The click-thru analysis left a little to be desired as the system provided each user's path information without any meaningful summaries. Clickthru data can be hard to analyze because users will come to the same pages from different routes and may back-up through previously-visited pages. It would be very helpful if they offered some better visualization tools to help give better insights in this area.

Conclusion
Vividence Express is probably the cheapest online usability option available but it comes with a number of fatal flaws. The panel recruitment process requires the use of a downloadable application, their internal panel has serious bias problems, and the restriction to two tasks limits the testing usefulness to very simple websites or applications.

I'd be very interested to hear from anyone else with experiences with Vividence.

Microsoft's announcement that there will be no new versions of IE for the Mac is no doubt sending shivers through many product managers bones. It is yet another sign that the golden age of IE dominance is coming to an end and that anyone offering an online service will have to worry about cross-browser compatibility once more.

This is an annoyance for informational sites like this blog, or your average corporate website. A little more thought needs to go into the HTML, some Javascript customization and other tweaks will be required.

But for sites or downloadable products with complex user interactions, multiple browsers mean a significant amount of additional work throughout the development, testing, and customer service processes.

In the past two weeks customers of the subscription bookmarking business I own, BlinkPro, have discovered browser-specific problems with Mozilla, Firebird, and Safari. Some of these bugs were frankly our fault -- that is, caused by out-of-date code. But others were simple minor differences between browsers which forced us to change code, patch our releases, and otherwise incur costs which were avoidable in the IE-only era.

As an example, we have a browser-specific page on our site which gives instructions on how to export your bookmarks to a file. This page is generated dynamically, depending on your browser and version. The instructions for both Netscape 6 and Mozilla said:

Click 'Bookmarks'...
Select 'Manage Bookmarks...'
Select 'File -> Export'

This worked great for Netscape for two years without complaint. Well, the Mozilla people moved the Export function from the 'File' menu to the 'Tools' menu so those users were confused. Worse yet, I heard that Firebird moved it back!

Look, no one likes Microsoft's arrogant dominance of the marketplace, and in many ways the new browsers are improvements over IE. We should all just be clearly aware that there are costs to divergent platforms, and that these costs are borne by the companies creating applications and content.