According to the MTA, its riders are quite satisfied with service. (Source: Customer Satisfaction Survey)
Here’s the $64,000 question for all 5.3 million subway-riding New Yorkers: Are you satisfied with service? Are you happy with your commute? Do you think the MTA is doing a good job? For the second year in row, the authority claims that nearly three-fourths of all riders are satisfied with subway service, but I remain skeptical.
The MTA last year revealed its customer satisfaction survey, and astute SAS readers questioned the methodology. The survey is now back for a second year, and although the MTA is trumpeting across-the-board improvements, problems remain. The sampling is appropriately random, but the scale remains skewed. Whether or not riders are truly satisfied is a question we could debate.
On a top-line level, the results are promising. Overall, 74 percent of riders are satisfied with subway service, and that total is up from 71 percent last year. Furthermore, 78 percent of riders are satisfied with the overall station environment, and that figure too is up from 71 percent next year. If anything, I’m personally less satisfied with station environment these days. I’ve found subway stations to be dirtier and less well maintained recently that at any time in the recent past.
So what then does it mean to be satisfied? In the presentation of findings, the MTA discusses its sample. From June 20-30, the authority conducted 1200 adults who had taken at least one subway trip in the past 30 days. These sample was culled from landlines (86%) and cell phones (14%). The margin of error is +/- 3 percent, and it seems as though the respondents represent a valid survey sample.
The results, though, are skewed to make the MTA look good. The MTA asked respondents to rate over 50 attributes on a scale of 1-10 where 1 and 2 represented very dissatisfied, 3-5 represented just dissatisfied, 6-8 meant riders were satisfied and 9-10 were very satisfied. Generally, anything less than 8 isn’t usually considered “satisfactory,” but the MTA stretched the scale. Thus, 74 percent of riders are willing to give the authority at least a 6 on a 10-point scale. Only 13 percent of riders said the MTA’s subway service was very satisfactory.
By and large, every major metric improved this year over last. According to the MTA, 84 percent of respondents find the MTA comfortable and convenient while 80 percent are satisfied with the courtesy and helpfulness of subway conductors and 79 percent are satisfied with service frequency. Those totals were at 78 percent, 77 percent and 72 percent last year, and yet, those seem to be the topics of more complaints than anything else.
Throughout the survey, the MTA reports increased satisfaction with nearly everything. Riders are more content with station cleanliness this year than they were last, and rush hour crowding on platforms and trains isn’t as bad it was in 2010 before the service cuts and fare hikes. Despite a bump in crime and fewer station gents, riders say they feel more secure on platforms and find platforms cleaner. If you too raised your eyebrow skeptically at this news, well, the MTA thinks it has an answer.
Countdown clocks are driving the bump in customer satisfaction. (Source: MTA)
As the authority notes, the bump in satisfaction may not be due to better service or cleaner platforms. Rather, the countdown clocks are driving the perception of better service. As the authority noted in the survey presentation, “All 54 subway service and station attributes were rated higher by those with countdown clocks in their station than those without a countdown clock in station.”
With countdown clocks, customers are far more satisfied with wait times and predictability of travel time and even seem to appreciate frequency and reliability of subway service even more. By taking the surprise out of wait times, the MTA can create the perception of better control over one’s commute, and thus, riders are predisposed to be more relaxed about their rides. Overall, 96 percent of riders were satisfied with countdown clocks; the other 4 percent are probably annoyed that they don’t work sometimes.
So what then can we conclude from these results? The MTA says that “continued investments in information technology, station maintenance and cleaning, and maintaining reliable service will continue to address customer concerns into the future.” Right now, though, the money seems to be in place to improve only information technology. Is the authority then essentially tricking us all into thinking our commutes are better? I personally like the countdown clocks and find myself less anxious over my waits. But when all is said and done, I’d rather have more train service. Personally, I’m satisfied with my daily commute but barely so. Are you?
17 comments
a 6 should not mean satisfied it should mean its not bad but in definate need of improvement
The methodology is completely flawed because the stations with countdown clocks were not selected randomly. They are comparing “A” Division stations with clocks to “B” Division stations without clocks. Therefore, any difference in perceptions could easily be related not to countdown clocks but instead to other differences between the divisions.
They should have instead compared people at countdown clock stations with people at *similar* stations lacking countdown clocks. Without this control group, the findings are meaningless.
Except for the L, none of the B Division has countdown clocks. And almost all of the A Division has countdown clocks. So it’s pretty hard to avoid a general divisional split.
That said, I see no reference to the divisions in the survey presentation. What gives you the idea that nobody on the L or at A Division stations without clocks were surveyed?
[…] Countdown Clocks Linked to Higher Straphanger Satisfaction (WSJ, 2nd Ave Sagas) […]
1200 subway riders in a city of 5.3 million of them is hardly a decent sample size. You need a better cross section to draw ANY conclusions, & the MTA knows it. This I’m sorry is nothing more than PR bullshit.
While I’m not putting much weight into these results, you’re statistically incorrect. If selected properly, 1200 out of 5.3 million can easily be a simple random sample. These results have a confidence level of 95 percent with a margin of error of +/- 3 percent. There’s no problem with the sample size.
The problem is saying that someone who rates subway service a 6 is “satisfied” with it.
The only problem with the sample size is that you can only see the broad picture. If you read the questionnaire you will see that they collected data on every bus and subway route. Since the sample was so small, they may have missed some bus routes entirely or else had a sample of one or two for most of the routes since there are something like 600 routes and only 1200 responses. Therefore, the data is not statistically reliable on route by route basis.
With a random sample you bias the results to routes with the best service that are most frequently used. Therefore your results will show that most people are satisfied with the service. If they tallied the results for routes with infrequent service separately, which of course they wouldn’t do, you would see quite different satisfaction levels when it comes to service.
Add to this the confusion most people would have with the ratings system being asked to remember for 25 minutes what the numbers stand for. Since, no distinction is made between a 1 or 2, or a 9 and 10 in the results, it is meaningless to have a 10 point system.
Just asking people to respond with words instead of numbers would ensure that at least the respondents say what they mean to say. Making 5 and 6, “Neither Satisfied or Dissatisfied” would lower the satisfaction levels further. But as usual the MTA slants results the way they want you to see them. Accuracy has never been their concern.
I’m agreeing with Ben. In fact, most presidential election samples are maybe only a few thousand people, yet at least 150m people can vote. It’s a matter of matching them to the demographics that’s important.
The problem is saying that someone who rates subway service a 6 is “satisfied” with it.
That brings up another set of problems. What does 6 mean. I’m OK with the service? Does it refer to something else?
Phil, your example is exactly what is wrong with political servays. 1. It is a snapshot at the moment i, e what do you think right now? 2. 1000 respondence really isn’t as large a sample size as one might think. 3. Any question depending on phrasing could skew responces. I have conducted plenty of these servay experaments in both college & in grad school, so I think I can call bullshit on these satisfaction servays. Lets be honest they really tell us nothing of substance & waist valuable finantial resources that could go towards putting more busses & trains out there.
No one is arguing that the survey was done well. It has plenty of problems. But sample size is not one of them. It’s counterintuitive that 1200 people can accurately represent millions, but the math says otherwise. Of course, those 1200 have to be chosen well.
Sample size is a problem for the bus survey for questions related to levels of service as opposed to questions about the bus vehicle itself or customer information. See my articles Monday and Tuesday Oct 31 and Nov 1 in Sheepsheadbites.com.
Did the survey figure out why the IND smells pissier than the other divisions? For some reason which I can’t figure out, the IND always smells like pee, even in winter, while the IRT/BMT not so much. Maybe IND stations are farther apart and riders just can’t hold it in?
Right now, though, the money seems to be in place to improve only information technology. Is the authority then essentially tricking us all into thinking our commutes are better? I personally like the countdown clocks and find myself less anxious over my waits. But when all is said and done, I’d rather have more train service.
It’s not a trick, it’s just human nature.
Examples:
http://davidmaister.com/articles/5/52/
S = P – E.
In this formulation, ‘S’ stands for satisfaction, ‘P’ for perception and ‘E’ for expectation. If you expect a certain level of service, and perceive the service reviewed to be higher, you are a satisfied client. If you perceive the same level as before, but expected higher, you are disappointed and, consequently, a dissatisfied client.
The point, of course, is that both the perception and the expectation are psychological phenomena. They are not the reality. In a benevolent world, both the perception and the expectation will have some connection to reality, but they are not reality. Accordingly, all service managers must pay attention to three things: what was actually done to or for the client, what was perceived by the client, and what the client expected. Fortunately, all three can be managed.
http://www.hsor.org/what_is_or.....psychology
Many companies (Disney is one example) have become expert in understanding the psychology of waiting. Waiting in a line that is moving seems less boring than standing still in the same spot. TV monitors with engaging pictures help keep visitors’ minds off the clock. In addition, if they can see and hear some of the excitement of those who have completed their wait, anticipation increases and waiting seems worthwhile. Lastly, expectations are a major factor in determining customer satisfaction. If customers approach a line and are told the wait will be fifteen minutes, at least they have the information to make an informed judgement as to join the line or not. If it turns out to be less than the quoted fifteen minutes then they are pleasantly surprised.
Was this survey conceived and conducted internally or farmed out to a professional market research firm? Neither the press release or the survey results mention this.
As a former market researcher, I can vouch that the bi-polar attudinal satisfaction scale is a common practice (save for a neither satisfied or dissatified choice), but to assign these choices to numbers is not. Especially without clear, explicit instructions. For instance, in this survey, respondents who rate variables as a 6 or as an 8 are essentially collapsed to mean the same thing, when there might be substantial difference behind why one would choose a specific rating. I wonder just how much was communicated to respondents about the intentions and utilization of their responses.
Probably none.
As flawed as the methodology is for the subway survey, from what I remember from last year, the methodology for the bus survey was even more flawed because of the range of questions asked which primarily dealt with environmental factors like aspects of the bus itself (was it clean?) and few if any questions related to how long it took to get somewhere and if they were satisfied with that,or how many transfers were required or any questions regarding the directness of the route(s) being used. If you really want to ascertain if someone is satisfied, you can’t limit yourself only to questions where you know that you are doing a good job and ignore other factors.
Yes, these surveys are just for PR purposes, not to find areas that need improvement and to take action on them.
[…] of the subway survey, along with a discussion of the faulty methodology the MTA employed, in the article and comments at Second Avenue Sagas. Therefore, I will limit my analysis and remarks only to the bus […]
[…] the MTA released its subway satisfaction survey last week, it also published a similar one concerning the buses, and I didn’t pay it much […]