Why we don't have comments on our product pages

A customer recently wrote:

I think you really need to add comments from users under each product (like sparkfun does). Makes it easy to review good/bad about a project and ask a question not in docs.

I rejected the idea when some of our developers pushed for comments on our web site a few years ago, and upon reconsideration, I am still firm in my opposition. However, I am interested in what others think, so rather than just giving the customer a quick reply, I spent a bit more time to present my view here. Several of my thoughts on this particular topic are based on broader philosophies of how people and organizations should operate, but I’ll try to focus on comment systems without getting too sidetracked.

First off, I use and like the customer review or feedback systems on sites like Amazon and Newegg, and I think it’s clear that the feature benefits customers and the sites. However, there are also many sites like CNN, where even minor reports can be appended with thousands of reader posts expounding on everything from the story not being news to some kid in the accompanying picture looking cute to some other reader being an idiot; the only thing missing in the comments is anything useful to anyone but psychologists and future anthropologists. Aside from some possible reductions in page load times, at least there is no real harm in the useless comments, and although part of me thinks that many of these sites are putting in user comment systems just because everyone else has one, I hope they know their business well enough to conclude that the cost of including a bunch of worthless content is outweighed by the benefit of enticing readers to participate in the site and thereby capturing a bit more of their attention. It might not be making the world better, but there are worse places than news sites for people to waste their time.

The spectrum of comment system value does not extend merely from obviously positive to approximately harmless; for a web site like ours, bad content is certainly harmful, and the likelihood of the user-generated content being bad, as a whole, is almost guaranteed. The second claim, that comment-based content on a site like ours will necessarily be bad content, is the bulk of my argument, and I’ll get to that after elaborating on the first point, that bad content is harmful to a site like ours. I see four reasons for bad user content to be more damaging to Pololu product pages, and by extension to our customers, than to product pages on Amazon:

First, unlike sites like Amazon or Newegg, we do not carry almost everything. When we make something like a motor controller, we might make a few versions to cover different current and voltage options, but we are not in the business of carrying fourteen different models of AVR programmer: if we think we have a good offering for a problem or application, we would rather spend our effort developing or looking for solutions to other problems rather than slightly different solutions for the same problem. If you are looking for a graphics card on Newegg and see a bad review for one, you are still probably going to get some other graphics card at that site. With our selection, a bad comment by a disgruntled customer could not just sway a potential customer toward another product we carry, but rather drive them from our site altogether. This isn’t meant to be some kind of sob story, but just my assessment of the situation, and it should not be surprising that I am against putting work into something that is against our own interest. (For those thinking that a site welcoming negative comments is a badge of openness or integrity, I’ll get to that later.)

Second, our products generally require much more expertise to use and evaluate than the typical Amazon product, making a review by an incompetent user harder to spot. If we see a review saying something like, “I would’ve given this hammer a higher rating but it keeps missing the nails, especially when I use it with my left hand”, we know right away to discount it. A more specialized site like Newegg might have the problem to some extent, too, but installing a graphics card is generally much easier than controlling a motor driver from a microcontroller. An electronics or robotics beginner might be competent enough to work through the documentation we provide but not necessarily be able to tell apart good advice or comments from bad. The bad comments could cause unnecessary frustration that might not only hurt our relationship with the customer but also the customer’s budding enthusiasm for engineering.

Third, for products like ours, it is difficult to separate the personal, subjective experience from objective facts, by which our products should be judged. If someone comments that a book is poorly written, that a pen is awkward to hold, that some shoes did not fit well, that a TV’s stand is ugly, or that some speakers sound bad, we are very aware that these are opinions, and it’s easier to take them with a grain of salt or go with two good impressions over one bad one, and those subjective opinions are still generally useful. However, it’s much more difficult to resolve conflicting statements in objective-sounding comments: you might see mixed reviews on the look of a sweater and still get it for your wife if you personally like how it looks; if you see some people saying a servo controller doesn’t work and others saying it works fine, it’s not as easy to dismiss the negative comments as just opinions or results in some particular applications that might not represent a typical application. This is not to say there is no room for subjective evaluation of our products or that comments cannot be helpful for products like ours; rather, the point is that there is a lot more room for comments to be disproportionately damaging.

These first three reasons are not just pie-in-the-sky, theoretical musings about why comment systems might be bad, and they do not apply only to small sites like ours. Just this past weekend, after I started writing this post, I went to the Lowe’s site to look into a blow torch for burning weeds in my front yard. They had one model, and it had a single customer review that claimed that contrary to what a Lowe’s staffer told him, the torch did not work with standard propane tanks. Of course, I wouldn’t be bringing this example up if the comment was correct: I eventually bought the thing at a physical store, and it worked just fine with the tanks they had there. However, if I were just shopping online, I would not have ordered it. In the store, I wasted a bunch of time trying to figure out why the thing would not fit the tanks since it sure looked like it would. There was no reason to expect the review to just be flat-out wrong, and no real way to tell that it was wrong. Since I was writing this post, I felt a greater need to address this particular problem, so I wrote a review of my own. I think it’s the first time I’ve written an online review for a product like that.

Reviews on Lowe’s torch product page.

So now, there are two reviews sitting there, claiming opposite things, and I have a feeling that if I had originally come across those two reviews there, I still would not have ordered the torch online. As soon as that first review was posted, anyone who looked at it either avoided ordering the torch or wasted time trying to understand it, and even with my comment there, the result is not as good as if the first comment were never there.

My fourth and final reason for thinking bad comments would be damaging to Pololu’s web site is more personal: I am trying to create good content and a useful resource for other engineers and future engineers, and hosting a bunch of crap on our site would be contrary to that mission. I have had arguments with customers who cited user comments from other sites when I was trying to explain why some of what they were doing was poor practice. There’s no shortage of places on the web for people who don’t know what they’re talking about to spout their ill-informed nonsense, and unlike CNN, I don’t think it’s okay to have a bunch of worthless content on my site just because it’s easy to tell that it’s worthless.

So, why am I so sure the comments would be bad? One simple way to see it is that if the comments contained anything good, we should put them in the main content on our site, making the comments redundant at best. From this argument, we can see that the value of a comment system is dependent on how much effort we put into maintaining a web site. If we put up minimal information and never modified it, comments could be better than nothing. But, that is not our situation, so let’s consider some individual claims of comment system proponents, starting with our customer’s message: “Makes it easy to review good/bad about a project and ask a question not in docs.” I think this sentiment is fairly representative in that it shows three basic desires: easy evaluation of a product, access to more information than is available in the (non-comment) documentation, and interaction with others, possibly including the site operators.

There’s almost no limit to the alternatives consumers have ready access to these days, so it’s understandable that they look to the advice and judgment of others to help evaluate their many options. In many cases, going with what works for others can be a good strategy, and that’s the main reason user reviews work at all. However, the “go with the crowd” approach depends on relatively consistent application scenarios and reviewers who have a good understanding of the material or are representative of the next user. In other words, for a review to be useful to me, my intended use for a product has to be similar to that of the reviewer, and my general background (as it relates to the product) needs to be similar to the reviewer’s. That turns out to be the case for most consumer applications, where the intended use of the product and the target customer is well defined.

For instance, for most customers, the expectation of a set of speakers is roughly the same (assuming they’re not some fancy audiophile-specific version), and a bus driver or professional golfer could be fairly representative of a typical user: after actually using the speakers, they could know about as much as there is to know about them for basic use. Reviews of speakers are mostly going to center on whether they subjectively sound good and maybe some other aspects like ease of use. The utility of the reviews would quickly go down if every reviewer had a different application, with one talking about how easy the speaker magnets were to detach and another commenting about how well the cases work to scare off pigeons. And this is still assuming the posts offer accurate information; once the comments get flooded with questions and inconclusive answers, there’s no hope of finding good information. Yet this is the reality for customers using our components, which can be used in countless combinations for any number of applications, and the customer’s experience with the product depends on the details of the combination. Our motor controllers might be used to turn a turret, drive a robot, propel an underwater ROV, spin a fan, or power LEDs, all from a multitude of different power supplies and interfaced to all kinds of different microcontrollers or other control signal sources.

Another fundamental problem with reviews from different customers is that everyone has a different scale, and I’m not just talking about opinions like “good” and “bad”. Even if a person is being completely honest and consistent, his sense of “small” can be different from someone else’s. Many adjectives like “small” are relative to the problem domain, and it’s difficult even for us to maintain some consistent descriptions as our product selection grows.

Of course, getting to the consistent and relevant user-generated content is important for mainstream retail sites, too, and various sites have different approaches for trying to rate the ratings and evaluating the credibility of reviewers. However, many of these approaches boil down to hoping that a high volume of reviews can be refined into the “right answer”, which is usually some kind of overall numerical rating. For our market size, we probably do not have a sufficient number of customers to even get a statistically valid number of ratings, and the concept of a “right answer” is in itself misleading. Viewer reviews might give you an idea of whether a movie is worth seeing, but strangers’ views should not be very relevant if your choice is whether you should read a good book or watch a good movie.

An important final point regarding user comments simplifying product evaluation is that even if user content is relevant and accurate, it becomes bad if it is poorly organized. There are successful user-generated content sites, like Wikipedia and Stack Overflow, that do not just reduce everything to a single number. However, we see that those sites have far more elaborate content management and organization tools than just a chronological list of comments.

The second benefit people want in comments is more fundamental information (as opposed to opinions or judgments) about a product. Ultimately, the hope people have for comments is that they will quickly illuminate some crucial aspect of a product; unfortunately, for complicated products like most of the ones we carry, there are many important aspects, and the likelihood that some detail you care about will be in a handful of comments or that you will find it in hundreds of comments gets vanishingly small. If there is fundamentally a lot of material to consider, an organized book with a table of contents and index is going to beat a pile of post-it notes.

As I said before, if the web site operator does not put effort into maintaining the content on a product page, the pile of post-it notes is better than nothing. For a large reseller like Amazon, it’s not just about effort: there is only so much we can expect its employees to know about a product, and if a product has thousands or millions of users, it is reasonable to expect those users to develop more expertise than the people maintaining the page. In that situation, the web site editors do not have the expertise to edit the metaphorical book, and we are left to make sense of the notes ourselves.

That is not our case, at least for the majority of our products. Our product pages and documentation are made by the people who design the products, and we do have the expertise to modify and maintain the pages as necessary. Rejecting a comment system therefore represents our commitment to providing good content, rather than hoping someone else will happen to do it for us. Please note that this is not some claim of infallibility or omniscience on our part; I’m just saying that if there is a mistake or missing information, it’s better to fix it directly than to make people rely on the comments.

The third common appeal for comments centers on terms like communication, feedback, and community. Once again, my problem is not with those goals; rather, my point is that comments on a product page are a poor means of achieving the goal. I think it’s quite easy to reach us: our phone number is on every page, along with a link for giving us feedback. If you call about a product, chances are that you can talk to one of the people that designed it. If you email, you should get a reply within a day. Our engineers also constantly monitor our forum, where customers can help each other and where we do not have any restrictions on linking to products from other vendors. Participating in discussions is also a good way to learn (both for us and for customers), but looking at a bunch of conversations is not an efficient way to learn if you don’t know ahead of time if the conversation you are looking at is a good one. Keeping around a log of outdated or otherwise irrelevant comments makes the likelihood of finding a good conversation lower.

My objections to comments might seem more valid if we consider some actual examples. Besides citing SparkFun, the customer was specifically interested in one of our stepper motor driver carriers, so I took a look at the page for a comparable SparkFun product. (The product actually got replaced since I started this post, but I’ll stick to the older version since the newer product’s comments, though also generally useless, might not be representative of what accumulates.)

SparkFun’s EasyDriver has over 100 comments, which probably represent more than 80% of the content on the page. If I am a user of the product, is it my responsibility to read all of the comments? The first several discussion threads look like they are discussing mistakes in the design that are probably no longer relevant or are separately called out in the product description. Then there are several comments praising the designer (“yeeeeeeeeeeesssss!! I love you!!” is the entirety of one post). Some comments are from customers that do not have the board yet. There are some comments about the parts being out of stock back in October. There are a bunch of unanswered questions. Need I go on? (By the way, the point here is not to pick on SparkFun, but rather to illustrate what I think is the inevitable outcome of user comments on sites like ours.)

Comments on SparkFun’s EasyDriver page.

A final aspect of comment systems that I want to discuss has to do with integrity or transparency. I have heard claims that comment systems increase trust in a site and that allowing public criticism of products shows humility or willingness by the web site operator to admit mistakes. While I am a fan of those two qualities, I am skeptical about the integrity of the execution of such systems. Unless a web site operator allows for absolutely anything in comments (which would lead to even more junk on it), some lines need to be drawn. Outright spam might be out. What about links to competitors’ products? What about comments that are of the form, “This product sucks because X” when X is objectively false? Ultimately, the web site administrator has to end up micromanaging the comments, and thus potentially changing the picture they portray, or accept that the comments will become a disorganized trash heap of inaccurate, misleading, and otherwise unsavory content. My impression is that the latter outcome is the more common one, with the mechanism that was supposed to lead to increased communication and accountability instead leading to indifference and even abandonment of the basic goal of providing the customer with good content.

My views on product comments might change as we continue offering more items we did not design and as I hear back from other customers. And of course, part of the point of this blog is to welcome comments: what do you think?