For the best experience use full HD.

Friday, March 22, 2013

Why the CSM Concept is Broken: Part 3 - The Fix

In Part 1 of this series, I established that the three major categories of CSM supporters actually see the role of the CSM in different ways. There is no consensus on the purpose of that role and therefore all three groups express dissatisfaction with the concept.

In Part 2 of this series, I concluded that the dissatisfaction derives from a disconnect between the abstract concept of the CSM being a player-elected body, that meaning all players, and the reality that only a small minority of players actually elects the CSM. Thus, the CSM does not qualify for the adjective "player-elected" nor is it representative of the capsuleer community as a whole. It is in fact a lobbying group with the pinpoint focus and limits of expertise such groups invariably have.

Now the question is what should we collectively do about it? What follows is my suggestion. I have no delusions that they represent the majority opinion, if one even exists. Before I roll out the suggestion though, you need to understand what each of the three categories I've schlepped everyone into wants from "the process," a general term for everything CSM related. I will save you all the analysis. Here is what I concluded each group wants.

CCP
CCP wants feedback on their ideas for Eve Online expansions. They want expansions that will grow the business. They don't want to be told how to run their game, though they are not opposed by any stretch of the imagination to good ideas. I think they get plenty of "good" ideas through both the CSM and those who actually participate. But the people giving that feedback are the same 16% or so who vote. They are not the majority of Eve Online players by any stretch of the imagination.

The Players 
Players just want Eve Online to be fun to play. That's an incredibly nebulous desire. What is fun to me is not fun to everyone. We all play Eve Online for different reasons. We all want different things from that experience. About the only thing we share in common is the desire to fly Internet spaceships. Some think the bomb is PvP. I don't. As I told a friend the other day, I don't exist to facilitate anyone's PvP. This was said not as a recipient of PvP, but as a potential participant. I'm still not into that sort of game play. I did it for real for too many years and frankly I'm tired of death and destruction - even virtually. But that's me, not anyone else. In the end, we all just want Eve Online to be fun for us.

The CSM Member
These folks genuinely want to represent the Eve Online player base. They also want to help mold the future of Eve Online. They want to present to CCP good ideas, and help them plan expansions the player base will love. I do not really believe they do this for free trips to Iceland. They would gladly do it over Skype if that was the way of it. But the CSM can't provide good input on things outside their experience. The null-sec representative is not as qualified to speak on high-sec carebear issues as the high-sec carebear representative for example - and vice versa. They each have their own expertise and lobby constituent. And IMO, none of them had the expertise to pass sound judgement on Incarna. It was outside the realm of any Eve player experience. That's why the wave-off never happened and Incarna crashed and burned.

Now before continuing, let's get one thing straight for all those who brought it up. Just because you hold an election doesn't mean you understand what the electorate really wants - or that you know what the elected have in mind. If you voted for a Chancellor who then sent your sons and daughters to war, did you vote for war? Don't forget that Hitler was originally elected democratically  I've had the opportunity to talk to more than a few Germans who lived through Hitler's regime. They did not vote for Hitler because he wanted to start World War II. They voted for him because he promised to end the crushing economic situation that arose with the depression; a situation made even worse by the Treaty of Versailles. Hitler used that promise to increase his military capacity, which is a no-brainer in perfect hindsight. But it also raised the economic prosperity of the German people. That's generally known as stimulus, and it worked well for Germany during the 1930s. The point is very few (I'm certain there were some) who initially voted for Hitler voted for World War II. Just like most who voted for the CSM didn't vote for Incarna.

Also, you can't assume those who voted are a representative sample if they are not a majority. The fact that they all participate when most don't also belies that misconception. To have a representative sample, you have to follow the rules governing statistical analysis. When done right, statistics are a very powerful predictor. They can also propel a person to stardom. Just ask Nate Silver. The current CSM process does not conform to such rigorous requirements. It is not representative - period.

What "the process" lacks is that rigorous statistical requirement. The proof is Incarna. CCP thought it would be awesome and the CSM let it happen. That was a double fail when the general player population had it dropped on them. Through all the review and all the discussion, it was a complete miss with the player base at large. To be sure, there were indications of dissent, but without a mechanism to quantify that dissent and represent it in an unquestionable manner, the only thing CCP really had to make the decision with was an Icelandic gut feel. I personally don't work an entire project based on a gut feel. Yeah, I know they had plenty of game industry feedback that said Incarna was the way to go. Guess what, Eve Online isn't your standard game with your standard herd oriented player. Who knew? Well, actually, there was a way to know but it wasn't through "the process" as currently implemented.

You see where I'm going now don't you? In television land it's called Nielsen Ratings. In politics it's called polling. In marketing it's called focus groups which lead to test markets. Sometimes they go wrong as happened in the Truman election and the Fabreze product, but generally they work and work well. As an aside, if you have not read "The Power of Habit: Why We Do What We Do in Life and Business", read it. You will be fascinated.

Businesses do surveys all the time. My company recently had every employee take a detailed web-based survey on how the company was doing. It was third party. It was anonymous. It had 87% participation from the over 10,000 associates employed in the United States. It told them things the management chain would never have admitted to upper management. I can't talk about specifics. You understand the scope now? But as much as it was negative in many regards, it will make our business much, much stronger so long as the company follows through on the recommendations the survey firm presented.

So finally, here's my suggestion. CCP should use the Eve Online client to poll the player population at large - randomly and according to proper statistical methods. Make sure an professional polling firm is engaged to shape the questions appropriately  CCP can already distinguish real person from multiple characters pretending to be real persons. Their war on bots indicates this to me rather bluntly. They should leverage this ability to create a polling routine within the client. Don't make it optional. Force the player to answer before they play. It's done all the time now. Just go read an article at Christian Science Monitor. They make me answer questions before I get to read the article. Many 'zines give you advertising first and articles second. This keeps their 'zines free. You may, and I stress may, drive a few players away. You will lose far fewer accounts though than because of Incarna. Hell, award ISK or Aurum to those who answer questions. That too is done all the time. Entice them and they will give you feedback.

Then make the CSM process the survey results. Have the CSM quantify and qualify the feedback. Make it part of THEIR presentation to CCP. CCP should not do this internally. Let the CSM apply their experience and expertise as players to those results so CCP gets a player's perspective rather than a developer's perspective of them. The CSM is uniquely qualified to bring that sort of interpretation to the table. Give them some facts and figures to back them up when they say something like, "Incarna's going to be a real flop." It will provide them and CCP with inarguable facts rather than best guesses and unsupported assumptions. Oh, and make sure the raw survey results are published BEFORE the CSM goes to Iceland. Make the Nate Silvers of Eve Online jump for joy. That way everyone knows what the player base really wants before any decisions are made. That would make the term "player-elected" absolutely moot, wouldn't it?

Fly Careful

14 comments:

  1. I'm pretty sure that CCP already have metrics for various activities in the game - they will have the numbers for the amount of missions run, or ships killed or complexes completed etc - but knowing what people do doesn't always mean that the activity is fun.

    A lot of the problem with EVE is that people will do things because they have to. If they just used simple metrics to say that missions are fun because a lot of players run missions then they would be misleading themselves because most players who run missions will say that they are boring.

    Its like saying that 'Aliens: Colonial Marines' was a 'good game' because of its sales numbers - it got good sales because of the aliens brand but the game itself sucked; but making assumptions based on numbers would argue it the other way.

    The next problem would be that even if you could identify what the problems are you'd need someway to FIX those problems - a poll will not give you that. Yes you could probably find out the various issues that players have with a mechanic but it doesn't mean that you can fix the system now that you know its faults.

    With any MMO but with especially EVE it tends to struggle with putting changes into context with its players and each little sub group of players believes that its more important than the others and its issues need to be fixed - anything other than complete co-operation will be seen as neglect and players will start to leave.

    If CCP decided to re-write missions completely the Null sec players would say that they are being left out in the cold. If they fix null sec the high sec players will feel neglected and new players could suffer due to the poor PvE content.

    Either one of those changes are not easy to achieve and just as likely to backfire, be ignored or simply be ineffective.

    Personally I think the only way to ensure a proper distribution of knowledge within the CSM would be to elect a representative for each gameplay style. CCP would then work with each of those representatives individually and collectively to firstly provide 'some' content for each play style but also to ensure that one change doesn't step on the toes of another.

    ReplyDelete
  2. Polling more regularly and within the client would help a lot. Even if it was done optional. At least a "delay" button must be there as it often happens someone just logs in to throw in a skill and is gone a second later.

    Just a simple notification "you are invited" and a menu in the neo com to get to the poll. Not invited, can't participate. Track the invited whether they answer or not within 7 days. If they can't be bothered to do so, there interest in the game might be different from the interest of the game design.

    ReplyDelete
  3. Were you aware that CCP not only conducts surveys of this nature, but uses the detailed statistical results as part of their decision-making process as well as the presenting the data to the CSM which is discussed at the summits?

    This is on top of the additional crowdsourcing data which has accumulated over the years on just about every topic imaginable by Trebor Daehdoow, data which is also considered in the development process - and the resulting list of player-demanded features has been dwindling with each expansion release since he began the initiative.

    The problem here is that anyone with an even passing familiarity with statistics understands their inherent limitations in terms of applicability (its very easy to toss around "scientific" and "objective" when referring to poll results, which are often anything but), and the fact that both from a business perspective as well as creatively, CCP cannot always simply follow crowd popularity in terms of what feature to work on next or exactly how the mechanics are shaped.

    However, statistical data is nonetheless very much part of both how the CSM interacts with CCP as well as how CCP seeks out quantifiable feedback from its customer base.

    ReplyDelete
    Replies
    1. Now if I knew that I certainly would have included it. I've never been surveyed and the fact I didn't know they did this is a problem. They need to release such surveys so the honest brokers of the world can keep them on the straight and narrow. Not making this information public does as much harm as good. Group think is always more prevalent in closed than open information systems. As my little story about my company survey implied, it will do no good if upper management doesn't act and they are not about to release it in total. I feel that is a huge mistake. It lessens our faith in their fidelity. So it is with CCP. I encourage you to encourage CCP to make those surveys public.

      Delete
  4. Oh, and one other thing to point out - in the CSM winter summit minutes CCP pointed out that they DON'T currently have the ability to distinguish between number of human players and number of accounts.

    Some players have questioned this, of course, and assume that they do anyways - but CCP themselves is currently saying that this is outside of their capability. CCP Seagull has said its essential for them to come up with a method that allows them to do this in order to refine their polling and demographic data even further.

    Reference:
    http://community.eveonline.com/council/transcripts/2012/CSM_CCP_Meetings_December_2012.pdf
    (Page 10)

    ReplyDelete
    Replies
    1. Hans, I'm an IT professional. That's a load of horse-hockey. They run what is in effect an authorized trojan on my system. They have access to any information they want and frankly there's not a lot I can do about it. If they can tell there are 3rd party hooks intercepting their client's communication stream in the RAM of my PC, they can analyze anything in my PCs memory they want. Let's stop with the official line. I know better. Now, "can do" and "does do" are two different things. I'm not saying they do, just that they can.

      Delete
    2. Hey now, why do you think I specifically used the words "saying", "claimed", and "said". :) I'm not arguing with you in the least, I also think its highly unlikely that they lack the capability. I am NOT, however, an IT professional and thus not very well equipped to argue with them on the matter.

      Delete
  5. Sorry, Mab, but as long as even just one player votes, the CSM will be player-elected. Representative, well, that's another kettle of fish entirely.

    ReplyDelete
  6. If CCP moves to your polling system, Mab, why have a CSM at all?

    Let's not forget why the CSM was created in the first place, to hold CCP's feet to the fire because the company proved untrustworthy. The real question that needs answering is CCP sufficiently trustworthy now that the CSM is no longer needed or relevant to restoring trust?

    To be honest, if I was a business owner, I'd be polling/surveying all the time on my own instead of farming it out to customers. So again, do we really still need the CSM?

    ReplyDelete
    Replies
    1. You have a CSM for two main reasons: it's a free work force and they are the honest brokers. They help break group think within CCP and it's a lot harder to tell someone "no" to their face. There are a few other reasons but those are the main ones.

      Delete
    2. That's probably the best (and most accurate) description of the CSM and its value that I've read in ages. Well said.

      It also happens to be a really *good* pair of reasons for having a CSM.

      Tinfoilers will always point towards the PR element, and maybe that was the original intention - but CCP does a really horrible job of promoting the CSM. Unless I really just don't understand the PR profession, it strikes me as paying for a really expensive tool that never gets pulled out of the bottom of the drawer. I *suppose* it could be PR in the terms of making players feel safer, but it doesn't that I can tell. Players distrust and complain about the CSM, and they haven't stopped distrusting or complaining about CCP, either. Maybe there's fancy metrics that would convince me otherwise, but I don't see the CSM working well for them in this manner given the current system.

      Delete
  7. Interesting stuff. I like your polling idea. Even if all it does is add more Aurum that I will never use, that's fine. I wouldn't mind using a poll to answer some questions.

    Also, I was getting ready to call Godwin's Law on you, but I think you may have made one of the few references to a certain dictator that was relevant to the process being discussed. You win +1 internets.

    To the point of CSM with polls, the CSM could help inform the numbers. I work in a field that does a lot of behavioral research, and you the quantitative and the qualitative data to really understand both, even if both are only a part of the picture. I have issues with every voting mechanism that CSM has used, but every voting system has to balance many factors. Some voting is better than no voting at all.

    ReplyDelete
  8. I enjoy playing eve, but I'm not one of those people who follow the forums and supports ideas that I think are neat. On the other hand in another game that I'm playing which is currently in beta, the companies whole focus has been on player interaction. They have their overall goal that they want to achieve with the game, (which won't be changed by the players) but there are tons of little details that can be enhanced for a better user experience. Several times this game has tossed out a system to implement a newer one based on player feedback. But the thing about it is, I have been sent surveys asking about what I think, while in the survey they also present some ideas that they are playing around with (eve has yet to do this). I then get to agree or explain why I think that's a bad idea. Of course this requires a balance between all aspects of the game so that you don't exclude an entire play style, but it gives you perspective on those aspects which may need to be changed.

    Are players always right? No, just take a look at the balancing of SCII: HoTS. Constant cries for nerfs abound before people have spent awhile trying to figure out how to overcome something. But player comments give the company something to look into.

    ReplyDelete
  9. As Hans said above, CCP does do surveys. I've filled out a couple of them over the years. They are also completely and utterly stupid.

    They work on the rate 1-5 metric of things I rarely care about.

    How much time do you think CCP should devote to art and graphics? 1-5?

    Or statements.. do you strongly disagree, disagree, don't care, agree, strongly agree.

    Blah blah blah.

    I have to wait till the last page for the comment box and then write a War and Peace sized novel about my actual issues with the game, whatever they may be, or what I think is doing well. They never fall within the questions asked before that, ever. I don't think I've ever told them what I want them to do about it. Over the years I've given them complex feedback in those little poles. Not one thing I've ever written in that comment box has ever been improved. Work may have been done on systems, but to the detriment of them(Dominion, I'm looking at you).

    So mostly, I've stopped filling them out when I get them.

    ReplyDelete

Be civil, be responsible and most of all be kind. I will not tolerate poor form. There will be no James Hooks here. We are all better than that.