The MTA can get plenty of satisfaction, according to internal polling. (Source)
Let’s try this one for size: The MTA yesterday released the results of its first-ever agency-wide standardized customer satisfaction survey, and while New Yorkers seem to make a sport out of hating the MTA, most riders seem to be satisfied with the state of their subway and mass transit survey. The results seem to belie reality on the ground, but when faced with a choice, New Yorkers may recognize the limitations of a vast system and their intractable demands.
The MTA didn’t share too much about methodology, but the authority says this survey represents the first time such statistics have been compiled “across the MTA family.” Customers were asked to rate the quality of service; MTA employees; on-board conditions such as lighting and comfort; information and communication; convenience, safety and security; home stations; and overall satisfaction. The commuter rail roads were deemed exemplary as as 93 percent of respondents say they are satisfied or very satisfied with Metro-North and 89 percent say the same of Long Island Rail Road. Overall, 18,000 people responded to the survey, and the results have margins of errors ranging from 1 to 4 percent.
Let’s drill down on the Transit results. The various presentations are all available right here. Initially, we see that the MTA used a 10-point sliding scale but grouped responses into four categories. Those who ranked services at 1 or 2 are considered very dissatisfied; 3-5 are simply dissatisfied; 6-8 means satisfied; and 9 or 10 lead to very satisfied customers. The presentation of the results, in other words, simplifies the scale.
Immediately, we see that 95 percent of riders are either satisfied or very satisfied with the countdown clocks. It’s hard to imagine anyone being dissatisfied with the efforts to bring real-time information underground, but I guess five percent of people prefer the time-tested method of peering wistfully into an empty tunnel.
On an overall basis, 71 percent of riders were satisfied with subway service, and 77 percent were satisfied with the line they use most often. That result seems to bolster the theory I set forth yesterday: Straphangers grow attached to their favorite train lines. In terms of service quality, 83 percent of people were satisfied with subway travel times while only 65 percent were satisfied with wait times. The MTA expects that real-time train arrival information will boost that figure, but I believe that until riders never have to wait for trains, they won’t be fully satisfied with service.
Next, we examine the information and communication totals. Clearly, riders enjoy having more info at their fingertips, but we start to see some sample size and selection bias concerns. Only 103 subscribers responded to the survey over email alerts, and these were, by and large, the satisfied customers. The placement of maps in the system, always a hot topic, drew some criticism.
The authority didn’t score too highly when it came to non-automated sources of information though. Only 69 percent of riders rated notices of service changes at least a six on the MTA’s scale, and only 56 said on-board announcements were clear. Only 62 percent found the availability of pocket maps satisfactory. While the MTA’s system is more complex than, say, London’s, the lack of a small pocket map is noticeable.
By and large, customers are happy with the rolling stock, and since so much of it is new, they ought to be. Where the authority is lagging though is in the cleanliness department. Only 68 percent were satisfied with car cleanliness, and just 64 percent say the stations are clean enough. That latter total seems far too high to me.
Interestingly, after nearly a year of debate over the role of station agents, straphangers didn’t say the subways seemed unsafe. Concerns were most pronounced after dark though as 80 percent say they feel personally secure before 8 p.m. but only 60 percent feel the same after 8 p.m. I’ll withhold judgment until we have another year’s worth of data to assess, but I can’t imagine those figures moving too far in either direction.
Even as we might cast a skeptical eye toward it, MTA officials were quick to promote the good news in this data dump. “This has obviously been an extremely tough year for our transit system and for our customers, but the survey results show that our customers appreciate the improvements we have been able to put into place, like countdown clocks,” MTA Chairman and CEO Jay Walder said. “This survey demonstrates the importance of improving service where we can in cost-effective ways.”
I found the most amusing part to be the customer suggestions though. Riders want more predictable travel times, cleaner stations and more real-time information. They also want shorter wait times, fewer delays, less crowding during rush hour and the Easter Bunny to arrive. What they don’t want to do is pay for it, and therein lies the great contradiction. Whether these numbers accurately reflect a generally acceptable level of satisfaction with the subways or whether the MTA is simply patting itself on the bank in a time of bad news matters little without the funds.
47 comments
I’m like a broken record at this point, but boy I think the numbers would be dramatically different if you told people what each thing cost:
1. Countdown clocks are great, but do you love them as much when you know they cost a couple hundred million dollars to install?
2. Destination signs on cars/email alerts/system maps/signs all cost a small fraction of the clocks.
In short, any survey that measures satisfaction without taking cost into effect is somewhat silly, in my opinion. Ask a bunch of people in a focus group if they prefer a top-of-the-line 50″ LED TV to a 36″ LCD, and you’ll get huge numbers for the 50″ LED, but if you tell people the price of each, they will overwhelmingly vote for the LCD, despite its poorer quality. Simply put, I still contend that the countdown clocks were a poor use of resources in a time of fare hikes and service cuts.
As one straphanger put it in the Times: “Trains coming more often would be better, but knowing when they’re going to come is a good thing.”
As a refresher, the clocks and the PA system upgrade to make them user friendly cost $171 million (only for partial installation). That $171 does NOT include the cost of upgrading the signal system to allow the clocks to operate (that’s another $213 million– though that project, independent of the clocks, has its benefits).
http://www.nytimes.com/2010/03.....wanted=all
We can’t just say, as Gene Russianoff did, that, “This is what riders want.” We have to weigh available capital against existing needs and determine what riders need most. As I’ve contended all along, increased frequency (or a laundry list of other improvements) would rank higher on the list than clocks, imho.
Good to see high ratings for the system in general, though!
Honestly $171 million seems like a good investment to me, even in a recession.
The results may be skewed because those paltry few respondents are those who are most likely to score favorably.
I simply cannot believe that stations got a 71% satisfactory rating. Most stations are old, dirty, dilapidated, dark and dingy. And renovated stations like Union Sq. which should be in sparkling shape are not much better off.
This is an MTA of old survey, not a Jay Walder style survey.
Now that they’ve released the methodology, I see they used simple random sampling techniques and developed a sample size that is legitimate enough. I wouldn’t discount the number of respondents, but I would look at how satisfactory ranges from a 6 to a 10.
They should have displayed the “very dissatisfied” and “dissatisfied” data series on the bar graphs. As the results are presented now the reader has to remember that the white space on the top of each bar represents customers who are unsatisfied.
One has to wonder why the sample size for New York City Transit Subways was so small (1,048 customers surveyed) compared to Long Island Railroad (7,548 customers surveyed) and Metro North Railroad (5,564 customers surveyed). Ideally they should have sampled a statistically significant number of customers on each subway line – as they apparently did for MNR and the LIRR.
But all in all its a good start.
Wow, ok guys, where to start (both with the article and the posts above):
1.) The scales used by the MTA were, in no uncertain terms, designed to create a falsely positive sensation about the MTA. Basic survey methodology would show that many people treat a 10 pt scale as though they are grades. Therefore, anything <6 is extremely dissatisfied, while 6-7 are dissatisfied, and only 8-10 show any level of satisfaction with the system. I am curious which MR firm in the city went along with such a ruse.
2.) People's interaction with the system is their fare. If countdown clocks are not linked directly to fare, they cost the people using the subway nothing. This proves the value of linking costs to outcomes. On the other hand, the idea that "if someone knew what it cost, they wouldn't like it as much" is a complete red herring. People have a "set" utility for a given item. That utility doesn't change with the item's price. The utility might exceed the price, though. In this particular case, that of countdown clocks, the utility is heavily weighted towards the MTA. While people will tell you it's "nice" to have them, behavioral scientists can tell you (thanks WMATA!) that people will wait three times as long for a train so long as they know when it is actually going to show up. Installing countdown clocks is far cheaper than running three times the train frequency.
3.) Jay Walder's surveys are no different than old MTA surveys, thank you very much. In fact, his might not be as good based on the speed with which he would like results (this favors samples of convenience over more scientific samples). That said, if the sample was random, 1000 people is MORE than enough to get a positive read on subway attitudes. There are certainly places where sampling caused bias (for example, if the study was landline only), but the data was likely weighted, and point 1 above goes a long way towards explaining the issues with the results.
4.) The reason the samples were likely different is probably due to the fact that each of the 7 MTA agencies has their own MR department, each with their own budget. That means NYCT could only afford 1000 completes, while LIRR and MNR had much meatier budgets. It also depends on the other studies the groups are doing. NYCT always has Title 4 studies to complete, and so their discretionary survey budget tends to be a little smaller than the commuter railroads.
That's my $.02 on the whole thing for now. Once I have time to review the presentations, maybe I'll have more.
I disagree with your premise in #1. If I give something a 7 out of 10 rating, I definitely would not consider that a “dissatisfied” rating. But I agree a 6 probably shouldn’t be considered “satisfied” either. If anything I’d put it something like: 1-2 extremely dissatisfied, 3-4 dissatisfied, 5-6 neither satisfied nor dissatisfied, 7-8 satisfied, 9-10 very satisfied.
I appreciate your argument, but I think it’s too personally framed. If you can believe it, this sort of thing (survey research) is my day job. Based on best practices, you will find that most people will only give a 6-9 rating on a 10 point scale (with really angry people giving a 1). It’s also important to note that Likert scales like these are not continuous, but in fact ordinal (7 is better than 6, but by how much, we do not know).
Here’s some reading on the topic:
http://blog.vovici.com/blog/bi.....-Practices
http://www.marketdirectionsmr......Scales.pdf (PDF)
One of the best Meta-Analyses on the topic was performed by the particularly famous methodologist John Krosnick at Stanford. The blog post above is based on that analysis. I can’t find the full paper at the moment, but if and when I do, I’ll post a link to that as well.
Sorry to do this to you, but please see my disassociated post below (since I gave up trying to respond here due to my own inability to use a blog comment system. Ugh.
Matt, as to #2, I see your point, but believe it is a matter of semantics rather than reasoning. My point is, writ broadly, that countdown clocks, while nice, are not anywhere near worth the price and that given the very real choice between having clocks and increasing frequency (as far as I know, the clocks were not paid for out of a limited purpose grant), the vast majority of straphangers would prefer additional frequency.
Or, put another way, the fact that people like the clocks a lot is completely irrelevant information to me. Of course they like the clocks. The question is do they loathe service cuts more than they like clocks, or do they dislike dirty stations more than they like the clocks (as you put it– relative utility is the most important metric).
I apologize if I was being unclear, as it seems I was 🙂
But here’s the thing – with the signs, the MTA can cut service by 2/3rds before people complain. That gives them an awful lot of headroom later should the need arise.
I see your point about behavioral psychology. What I am saying is that from a very practical perspective (in terms of man-hours/efficiency), making sure Person A can get from Brooklyn Bridge to GCT in 12 min. on average as opposed to 20 min. on average, has real effects even if Person A is “happier” to take 20 min. with a clock than 12 min. without one. Clocks may be cognitively beneficial. They do next to nothing for efficiency/productivity EXCEPT when placed in locations that give straphangers a choice about what platform to run to (it is these situations where I think clocks could potentially serve the greatest good). No doubt you are right about the cognitive point though…if only our brains worked right 🙂
Do you have a citation for the 2/3s figure?
I’m not doubting you, but what you say doesn’t accord with the ridership screens I’ve heard of. The MTA’s own model considers a minute spent waiting (without clocks) to be equivalent to 1.75 minutes spent in motion. I don’t know what the model says about waiting with clocks, but an unrealistic lower bound of 1:1 would imply the clocks only allow cutting frequency by a factor of 1.75, i.e. by 43%.
Ok, so, I’m not finding the WMATA study. What I am finding is this: Link. Perceptions of wait time decreased 20%. Doing the math on that suggests that if a train runs every 5 minutes, the MTA could run one every 6.5 minutes with signs and have no perceived change in level of service. That translates from 12 trains an hour to about 9, which is a big deal. That’s actually a change in service of 25% (not the 1/3rd I stated above). Still not chump change in savings.
Argh. I cannot get a post with a link in it to stick. What is wrong with me?
Ok, so, I’m not finding the WMATA study. What I am finding is this: nctr.usf.edu/jpt/pdf/jpt%209-1%20dziekan.pdf. Perceptions of wait time decreased 20%. Doing the math on that suggests that if a train runs every 5 minutes, the MTA could run one every 6.5 minutes with signs and have no perceived change in level of service. That translates from 12 trains an hour to about 9, which is a big deal. That’s actually a change in service of 25% (not the 1/3rd I stated above). Still not chump change in savings.
Add www to the PDF linked above to get it to work.
It still doesn’t work.
Anyway, if the perceptions decrease 20%, then it means that 5-minute service without signs is equivalent to 6-minute service with signs, for a 20% reduction in frequency.
I don’t think we’re approaching what they said the same way. It’s a perceived 20% reduction in wait times. That means with a 5 minute interval, the trains will seem like the come every 4 minutes. You’re right, it’s not 6.5 minutes, it’s actually 6.25 minutes (5/.8). That translates to 9.6 trains per hour instead of 12, or a 20% reduction in service. Sorry for the rough back of the envelope calculation before, but the fact of the matter is that the signs are cheap(ish), and a one time cost, while service reductions provide ongoing savings.
I disagree. Unless headways are extremely short – reliably 2 minutes or so – countdown clocks are useful for route planning. The cost of running everything on a 2-minute headway, around the clock, would be astronomical.
Countdown clocks are important to me personally. Not everybody has an iPhone to check when the next train is coming.
They are coming eventually, and the costs only seem to be increasing. I don’t know what the life span is on them, but its a good investment in modernization for the subway.
Remember there are lies, dam lies & statistics. The first question i always ask is what was the sample size? 18,000 is a small sample compared to the actual number of daily riders.
18,000 is a huge huge sample. You only need to randomly sample 384 people out of any sufficiently large population to estimate a yes/no question with 95% confidence and a confidence interval of 5% (that is to say, 95% of the time your measurement is within +/-5% of the true value.
Where this is complicated is when you start to either split the sample up into groups (you would need 384 males and 384 females to do the same thing if you wanted to know what males or females though). Additionally, the less random your sample is, the more likely there are to be design effects (clustering, for example), that will reduce the quality of your estimate. Even then, 18k is a really REALLY large sample for something this non-scientific.
For reference, I’m not questioning the quality of the data collected. I am more concerned with how it is being analyzed. A better analysis would have indexed each of the satisfaction scores so that it would be clear which scores people were more satisfied with and which they were less so. Unfortunately, that analysis would not have allowed the MTA to trumpet that 71% of all riders are somewhat or very satisfied with the subway!
Argh! All of my replies to replies to me are being eaten right now. So this post gets shorter every time. Here’s links to likert best practices. Working on getting the Krosnick meta-analysis. People only score 1 and 6-9 on a 10 pt scale. Maybe it’s because of the links? Attempting to post without them…
Argh, sorry guys, for whatever reason my posts with links are being eaten. I’ll try to get you links later.
Definitely I think people are more likely to rate 1 or 6-9 on a 10 point scale. But I don’t think that’s hard and fast either. Personal example. On IMDB you rate movies from 1-10. The majority of my ratings are from 6-9, but I do have three 1’s, one 2, one 4, two 5’s, and four 10’s. But that’s movies. By and large I don’t think there are a lot of truly bad movies, so most legitimately fall into that 6-8 range.
When it comes to something like the MTA, I think the “6-9 rule” wouldn’t really apply. If you thought the stations were disgusting and filthy, would you really give them a 6? I don’t think so.
It doesn’t really matter what the “something” is. It’s a built in bias of the scale. What I’m really suggesting is that the MTA picked a 10 point scale and converted to a 5 point scale specifically to game the satisfaction ratings. They could just as easily gone with a 5 point scale in the first place.
Additionally, by converting from a 10 to a 5 point scale you are ASSUMING (incorrectly), that your 10 point scale is an interval scale. It is not. It is ordinal. There is no way to ascertain the difference between 7 and 8 from 8 to 9. All we know is that 8is higher than 7 and 9 is higher than 8. This, compounded with the natural bias inherent to a 10pt scale makes the final data almost unusable.
Even if only some people have slightly less inclination to rate 2-5 and 10, and a slight propensity to score 6,7,8,9 this creates a large difference across a population of 18,000 and this is doubled by the rescaling (because the top 2 box becomes 7,8,9,10).
If I give someone a 6 out of 10 I am basically giving them a ‘D’ so I would not consider that ‘satisfied’. I would be curious to see what the breakdown would be omitting all the 6s from the ‘satisfied column, especially for areas like cleanliness.
I agree that these results seem like they came from people who are afraid of more services being cut and fares repeatedly raised. We all complain continuously about trash, cleanliness, station disrepair, vermin, and the egregious waste of money caused by using awful vendors that make our transit system a huge version of the movie “the money pit”.
Everyone seems to be interpreting the countdown clocks thing as customer satisfaction with the concept of having them. I interpret it as satisfaction with the operation of the clocks we have in place today. Count me among the “slightly dissatisfied.” While I understand that the grass is always greener on the other side of the fence (and that this is especially true for transit, because you get on the NYC subway hundreds of times a year, and are bound to experience its flaws, compared to a week-long jaunt in London where, statistically, you are less likely to experience disruptions), I feel that I have been treated far too often to this little monologue:
“Ladies and gentlemen, the next L Train is now arriving on the-”
“Ladies and gentlemen, the next L Train is now arriving on the Brooklyn-Bound Track. Please stand away from the platform edge.”
No train arrives. People look confused.
“Ladies and gentlemen, the next L Train is now arriving on the Brooklyn-Bound Track. Please stand away from the platform edge.”
A train arrives on the 8th Ave-bound track. Brooklyn-bound customers continue staring down the tunnel in despair.
Have you tried the countdown clocks along the IRT lines? Those work much better than the L service (which has improved greatly lately). Most people in the survey probably aren’t L riders and most countdown clocks in the system now aren’t along the L.
I guess I must be in the 5% of people dissatisfied with the countdown clocks. I’ve never seen the ones on the L train, but I did see it once at Nevins Street, the time I told you about. I found them to be totally useless giving me confusing and inaccurate information. And you say the L clocks are even worse?
By the way, you never got back to me regarding why the clocks behaved the way they did?
As far as I can tell, that was a one-time event brought about because the clocks are still officially in test phase. I use the IRT lines all the time, and since the summer, the clocks have been deadly accurate. Are you basing your assessment of them on one experience?
Yes I am. That was the first time I used Nevins Street in about ten years and it has been the only countdown clock I’ve seen. I will revise my opinion about them the next time I encounter one which may not be for a while.
I’ll echo Ben and say that I’m satisfied with the clocks’ accuracy, and I use the 1 often. However, I’m not completely satisfied with the choice of which stations they’re placed in. 34th is the most glaring omission, since people have to decide ahead of time which train’s platform to wait on.
I think you’re missing Jeff’s point. The survey wasn’t asking if people are happy that countdown clocks exist – it was asking if people are satisfied with the way they work.
While I’m a major proponent of countdown clocks, and I’m very pleased that I use several stations where they’re installed, I’d probably give them a rating of 6 or 7. Why?
First: While the information is usually pretty accurate, it’s occasionally way off (and not only on the L), with missing trains and trains coming several minutes earlier than predicted. Teething problems, most likely, but if the MTA wants my opinion of a product that’s still in the teething phase…
Second: In most cases, there’s no need to show anything past the third train. When the sign is showing the seventh train, it isn’t showing the second or third, which I’m probably more interested in knowing about.
Third: Having a single sign showing trains on two tracks is just plain stupid, since trains on those two tracks are either going in opposite directions or making different stops. If I’m on a local pulling into an express stop, I may need to know right away when the next express is due in order to make an informed decision of my next step. I don’t care when the next local is coming. I want a sign that will, at a glance, inform me when the next express will arrive.
Fourth: Why can’t we pull up the countdown clocks from our computers and smartphones? If there’s a delay on the line I was planning to ride, I want to know before I arrive at the station – perhaps I could have walked in the other direction or remained on the bus to reach a different line.
Second: In most cases, there’s no need to show anything past the third train. When the sign is showing the seventh train, it isn’t showing the second or third, which I’m probably more interested in knowing about.
This really needs to be repeated over and over again. There’s no need to show anything more than the next 2 trains. I’m not waiting around for the 3rd train.
Well, if I don’t get my way with my “Third” point (a separate sign for each track), then showing only two trains might not show any expresses or any locals. And even if I do get my way, if I’m waiting for a 5 train but the first two trains are 4’s, it would be nice to see what the third is.
Maybe that’s a better approach: show the first two trains, whatever they are, and then show the first train of any designation that hasn’t appeared yet, even if it won’t be coming for a while.
I don’t have a problem with all the trains appearing on one sign if that keeps the cost down. In no case, however, would I need to see two number 5 trains if they are both going to the same destinations. You need to see the first train of each letter designation in each direction. You don’t even need the second or third, because it is too confusing.
I disagree. If the first train is overcrowded, some people will let it go if they see the next train is close behind it.
I notice the the NYCT survey was only for subway riders. After, Howard Roberts fiasco of surveying bus riders a few years ago, where the average grade was a C, they decided not to repeat it again.
The whole purpose of surveys should be to highlight where your strengths and weaknesses are, so you know where to make improvements. By not surveying bus riders again, it sounds like no improvements were made as a result of the survey results and the MTA has given up on its local buses. Guess they felt the service cuts would result in even more negative comments than last time and riders would regard them as a waste of money.
Their purpose in these surveys is obviously to structure it somehow so that it comes out with positive results so they only surveyed subways.
When they do a bus survey again, it will only be for SBS routes, and they will concentrate on questions regarding time savings, while minimizing the problems to give the impression it has been a success.
They will continue to ignore local buses and only focus on more cuts. I hope I am wrong, but I think I’m just being realistic.
They did a bus survey. It’s available here as a PDF, and it focuses on local buses. I didn’t have a chance to write it up yet.
Thanks for the link. I glanced at it briefly. I already see a few problems but will comment further after I have a chance to look at their methodology and after you have had a chance to review it.
This has to be one of the most biased surveys I’ve ever seen. A survey has to be objective which this one most certainly was not. It definitely had an agenda: to show that most people are happy with the MTA and that the MTA is actively addressing problems.
You don’t ask a question regarding important issues prefacing it with “Other than service cutbacks and fares, what else is important to you? Why is the MTA afraid of publishing a result showing the percent of passengers who are concerned about the two issues most people are concerned about?
Why are there so few service related questions, only two or three? While there are numerous questions relating to every aspect of appearance, announcements, etc. There is nothing wrong with those questions, but if they are going to go into such detail about those issues, they also need to go into details regarding service by asking, for example: are you satisfied with the directness of the routes you take? They asked about convenience of routes, but that is totally different. Most people would interpret that as the distance to walk to a bus route.
The rating system is also all screwed up. I also don’t think someone on the phone can comprehend a ten point system. An A, B, C, D, or F or 1-5 rating is much easier to understand. By using a ten point system and declaring 6 and above satisfactory, you are really saying that if something works more than half the time, you are doing a satisfactory job. That in essence is what the MTA has done here.
Noted that you are talking mostly about the countdown clocks, but someone mentioned the disparity between the number of “respondents” on Long Island vs the city. We take the LIRR. No one on the LIRR that we have been speaking to has seen or heard of ANY survey and then all of a sudden they released the results of this alleged survey. Yes, we know they are alleging that they surveyed over 7000 LIRR riders, but no one seems to know who the heck they were. Perhaps they are related to Jay and Helena?? Maybe the bigger issue should be the “validity” of the survey on the whole. Just wondering.
Who’s the “we” you’re talking about here?
They needed to release more details regarding the methodology used, for example, during what time of day were the calls made?
[…] MTA last year revealed its customer satisfaction survey, and astute SAS readers questioned the methodology. The survey is now back for a second year, and […]