Nov
30

Whither those B Division countdown clocks?

By

A few weeks ago, I first linked to The Atlantic’s lengthy piece on New York City Transit’s technological woes. At the time, I examined the trial and tribulations of bringing communications-based train control online and highlighted how the MTA’s current approach is both impossible to sustain and inefficient in its execution. It is the classic story of a large and conservative bureaucracy unable to adapt to technological change, let alone a fast pace of adaption.

Let’s dive back into the piece and explore the countdown clock conundrum. As you may recall, James Somers initially set out to write about why only the A Division subway lines — the numbered routes — have countdown clocks while the B Division trains — the lettered lines — do not and will not for the foreseeable future. He also wants to understand why the A Division countdown clocks arrived years late. It is, he writes, “the story of a large organization’s first encounter with a large software project.” As you can imagine, it hasn’t gone particularly well for the MTA.

First, Somers notes that Automatic Train Supervision, the project that allowed the MTA to introduce countdown clocks on the A Division, is a subset of CBTC, and had the agency better coordinated and understood technology, they wouldn’t have spent 14 years installing an interim solution. The story goes south from there:

A post-mortem by the Federal Highway Administration details how from the start, an agency which had had little experience with large “systems” projects tried to wing it. For instance, the consulting firm tasked with developing the project plan never made a list of requirements, didn’t talk to the workers who would be maintaining the system until after it was designed, and left vague instructions for large chunks of work—specifying, for instance, “similar functionality to what is currently available”—that later became the focus of drawn-out contract disputes.

The MTA thought that they could buy a software solution more or less off the shelf, when in fact the city’s vast signaling system demanded careful dissection and reams of custom code. But the two sides didn’t work together. The MTA thought the contractor should have the technical expertise to figure it out on their own. They didn’t. The contractor’s signal engineer gave their software developers a one-size-fits-all description of New York’s interlockings, and the software they wrote on the basis of that description—lacking, as it did, essential details about each interlocking—didn’t work.

Gaffes like this weren’t caught early in part because the MTA “remained unconvinced of the usefulness of what seemed to them an endless review process in the early requirements and design stages. They had the perception that this activity was holding up their job.” They avoided visiting the contractor’s office, which, to make things worse, was overseas. In all, they made one trip. “MTA did not feel it was necessary to closely monitor and audit the contractor’s software-development progress.”

The list goes on: Software prototypes were reviewed exclusively in PowerPoint, leading to interfaces that were hard to use. Instead of bringing on outside experts to oversee construction, the MTA tried to use its own people, who didn’t know how to work with the new equipment. Testing schedules kept falling apart, causing delays. The training documentation provided by the contractor was so vague as to be unusable.

The MTA’s attempts at bringing the ATS system to the larger B Division faltered three times during the first decade of the 21st Century, and instead of trying to speed up the pace of installation of the CBTC system which would, as an ancillary benefit, introduce countdown clocks systemwide, Transit is again looking at a piecemeal solution. Again, it’s not working.

As we now know, the MTA doesn’t anticipate completed the installation of the Integrated Service Information and Management system on the B Division before 2020. An original deadline of 2017 was deemed unrealistic, and the delay in capital funding pushed this project back to the next five-year plan. And here’s the rub:

The problem is that the project has slowly taken on a bigger and bigger scope. The minutes of a 2012 Capital Program Oversight Committee meeting reveal that initially, the project’s focus “was to provide Train Arrival Information in stations.” Several service incidents, including a winter storm, drove the MTA to “re-focus project priority to provide centralized service-monitoring and information… followed closely by customer information.”

It is growing to look more and more like ATS. A request for proposal as recent as six months ago—back when funding looked more secure—called for a 77-month software contract to build out a sophisticated Rail Traffic Management System as part of ISIM-B. That piece of the project is envisioned as a complex centralized “expert system” that would allow operators to quickly diagnose service problems and would intelligently suggest ways to work around the disruption. It is, in a word, ambitious. And ambition is the death knell for big software projects. It’s what made ATS such a quagmire in the first place. It is, one suspects, why funding for countdown clocks has been cut from the latest capital plan: The rest of ISIM-B costs too much. It costs too much because it is trying to do too much. The consequence being that for five or six years, customers will hardly see anything get done at all.

At this point, Somers comes up short on a solution. He properly cites to BusTime, the MTA’s greatest software success story, as an example that the agency has had a few people at various times with the ability and trust to do something in-house. But those involved in the original creation and implementation of BusTime have long left the MTA, and Jay Walder, the CEO who was willing to give BusTime a shot, was forced out over his apparent lack of political savviness in dealing with both Albany and the TWU.

So what comes next? Certainly not countdown clocks on the B Division trains any time soon, and certainly not much faith that the MTA can execute complex technology upgrades in a timely or efficient manner. The MetroCard replace is on tap and could suffer from the same fate. Meanwhile, everything is years late and millions of budget. When it’s going to take the better part of a century to bring CBTC to the entire subway line, in the end, do we have any real hope that, without a top-to-bottom organizational overhaul, the MTA can execute on projects that are standard throughout the world? I’m not sure anyone really likes the answer to that question.



Categories : MTA Technology

27 Responses to “Whither those B Division countdown clocks?”

  1. John R says:

    This is so true. We can’t afford for the MTA to not whip itself into shape quickly.

    (Side note: I thought Jay Walder resigned, not forced out?)

  2. Eric says:

    Off topic – our peer city London is building a subway extension with *entirely private funding*. Any way we could do that?

    http://www.railwaygazette.com/.....round.html

    • webster says:

      Sounds like a convenient turn-of-phrase, to me…

      They’re using public money, but the investment is to be recovered from executions on developments in the specific area benefiting from the extension.

      Essentially, it sounds like a Tax Increment Financing scheme, which is pretty much how the 7 Line extension works out – once the planned developments actually occur, I suppose. The funding to actually build the line comes from the public sector, but the value is captured from current/future development schemes.

      http://content.tfl.gov.uk/nl-factsheet-i-web.pdf

    • AG says:

      Yeah Webster explained it well… On another note I usually laugh when they say privately funded. It’s kind of like those guys building the “All Aboard Florida” train system. They may be paying for it with private money – but they are asking to borrow billions from government sources. Not much different than many stadium deals.

  3. Larry Littlefield says:

    “Jay Walder, the CEO who was willing to give BusTime a shot, was forced out over his apparent lack of political savviness in dealing with both Albany and the TWU.”

    Jay Walder, who was the sacrificial lamb when the bills from a 30 year party came due and he was no longer able to hand out debt-funded candy to everyone with power. And was instead forced to do all kinds of unpopular things to prevent (delay?) a system collapse.

    Tell it like it is. You think the politicians and unions are about to step up and say “our debts and 2000 pension increase created this disaster?”

    Once the dirty deeds were done, Cuomo wanted someone to hand out a few more goodies the way he wanted it. So he put in a former head of (of all places) the LIRR.

    People don’t adjust what someone at the top does for the circumstances in which he does it. Paterson did nothing good, just handed out pain. He appointed Walder. But Paterson was in reality the best Governor we’ve had since M. Cuomo.

  4. Jedman67 says:

    Until we get a mayor and governor who are fiscally responsible, and realistic about the role the MTA development plays in the city’s economy; all we will ever get is more delays, skyrocketing costs (why does it take $9 BILLION and nearly a decade to renovate an elevated subway station??) and declining value.

    • AG says:

      Well we did… That’s how we got the 7 train extension… Even though the MTA still couldn’t get that one done on time. Make no mistake – even though it is missing one station (because of cost overruns) – without Bloomberg and his team – there would be no 7 train extension at all. Don’t forget the City Council and the state Assembly play huge roles. Convicted Sheldon Silver used his power up there to kill congestion pricing that would have funded transit. He gave Bloomberg his biggest defeat with that one.

  5. Alistair says:

    Sounds like somebody over there needs to understand the concept of a Minimum Viable Product. All you desperately need is a data feed — just capturing the data from the interlockings and saying ‘oh yes, this one is a B train running through to Bedford Park Blvd’, then feeding it into the same online channels that the A Division stuff already has.

    Once you have that, you can follow it up (as a separate project) with actual physical countdown clocks at the stations, so people don’t need to check their phones for countdown information, and if need be with some internal train-planning software (which, by the way, should at that point be common between the A and B divisions).

    But here’s the key point: this should be a read-only feed from the interlocking, and therefore it should not be safety-critical. So it should need minimal ‘commissioning’ or disruption to service. You just solder some wires into the lights on the panel at the interlocking, or the equivalent, and feed them into a couple of networked workstations — through relays if you want to maintain full electrical isolation. Run that back through the corporate network to headquarters and just do some simple software analysis. I hesitate to say that you can do it all with a thousand lines of Python, but there’s a good chance it isn’t a whole lot more than that.

    • Ed Unneland says:

      What about building upon those signals on the Sea Beach line that let passengers keeping warm in the station house that the train was coming?

    • Anonymous says:

      You are absolutely correct, that is all that’s needed. Perhaps also some dedicated servers, so that the data can be consumed in real time.

      The problem with this approach is that it is too fast and too cheap – you would be undermining entire departments and project groups sitting for years on millions of dollars and doing nothing. No one wants to be “that guy” to call their bluff and potentially get many people fired (or at least angry) because everyone will realize they are useless.

      It is the central problem of an organization that promotes secrecy and fear instead of openness and communication.

  6. Seth Rosenblum says:

    NYCT needs to realize this is the new normal and hire a staff (not appointed) CTO. If their going to do big Information Systems projects going on and actually be able to maintain what they’re building, they need at least a small core of software architects/engineers who can know when a contractor is screwing up, and tell the organization when they’re flushing money down the toilet.

    With CBTC, this software is going to become as much a core competency of the MTA as rail-welding and train sweeping; they can’t just outsource their whole business.

    • Larry Littlefield says:

      “They need at least a small core of software architects/engineers who can know when a contractor is screwing up.”

      I just remembered something, and I’m not sure anyone will read this with a new post up, but I’ll write it anyway. The MTA tried this, and it was a complete failure.

      It created a new department around the year 2000 to oversee all telecom and IT projects. How to fund it? With green money? No, with debt. All of the Department’s hours were billed to capital projects.

      The organization went out and hired a whole bunch of people, but they were people who it turned out couldn’t do the work. So now you had TIS engineers, Dept. of Subways engineers, consultants, contractor engineers, contractor consultants, environmental engineers, environmental consultants all billing these jobs, with multiple levels of oversight and approvals. And to top it off the put 2 Broadway on the same bill. Overhead costs soared.

      Eventually there was a battle and the new Telecom/IT group was blown up and shut down.

      • AG says:

        How about this as well… The last mayor started an information/media/tech company. His team rightly saw NYC needed more tech talent to revamp the local economy for the 21st century. This resulted in the city putting up funds for the “Applied Sciences Initiative”. Out of that has been spawned a new dedicated tech campus for Cornell Univ. and the Technion (of Israel). Secondarily were formed the Columbia Univ. institute of Data Sciences and the NYU center for urban studies and progress (CUSP). All three of these schools have noted their focus on the built environment. CUSP is actually being built in an old MTA building in downtown Brooklyn. CUSP is also heavily involved in the development in Hudson Yards to monitor everything from air quality to noise to trash volume to energy usage. So I should think that for these 4 major schools – and their budding tech entrepreneurs – tackling the technological deficiencies of the MTA would be a nice feather in the cap. Sadly though – this current mayoralty seems to have zero understanding of such things. Hopefully someone initiates it in spite of him.

        • Larry Littlefield says:

          “The last mayor started an information/media/tech company.”

          He still got completely screwed on CityTime, although the city was able to sue and get its money back.

          The government always seems to get screwed on these sorts of things. It’s amazing EZ Pass works.

          • AG says:

            well sure… Had it been at his company it would have never gotten so bad and heads would have rolled… Government is still government yes – but that’s my point. Those schools wouldn’t be in it for the money… For the prestige yes (which then brings in more donor money) – but not the cash outright.

  7. summer says:

    I’m not going to argue against the necessity of CBTC but does anyone know why countdown clocks can’t be installed without it? Walder mentioned at a talk he gave over the summer that it could be done through RFID.

  8. Joe says:

    It’s so embarrassing NYC won’t have these by 2020. I can’t think of another major city in the world that doesn’t have 100% coverage with this technology yet.

    And no, it’s not that hard, I’m a software developer and I can think of a few simple ways to capture and encode this information. You could easily, for example, read the car numbers off the side of the train as it passes through stations that follow interlockings to identify which trainset has just platformed. This would be an incredibly easy job for OCR. Someone in dispatch just needs to set the trainset to the run (which indicates its route and terminus). Then you can deduce which train is activating a block using track circuits. From there, just use an simple “learning” algorithm that compares the recent average times between each monitored block to estimate how long until it arrives at each station, and feed that data to the displays.

    This is pretty much how BusTime works (the track circuits make it marginally more complicated though) and that was developed cheaply and quickly in house. It’s insanity that the MTA cannot figure this out.

  9. Herb Lehman says:

    Again, be careful what you wish for. Do you really want the “1,413 tons of trash” announcement, announcements admonishing you to not cough or sneeze on the subway, etc. playing nonstop in the B division stations as well as all the IRT stations? While I’m definitely not against CBTC, the countdown clocks themselves don’t make the trains come any faster or get you to your destination any quicker. I will admit that the ability to check online when the train is coming before leaving my workplace is nice.

    • Larry Littlefield says:

      “While I’m definitely not against CBTC, the countdown clocks themselves don’t make the trains come any faster or get you to your destination any quicker.”

      They can when you have a choice of express vs. local, or more than one way to get home.

      • Herb Lehman says:

        But they don’t.

        Case in point: This morning I arrived at Atlantic Av on a 3 train. There was an announcement on my train that the train would be running express to Franklin Av. I was hoping to use the info on the clock displays to help me determine whether it would make more sense to stay on the train and just walk from Franklin, or wait for the next local to make my regular stop. But there was an announcement blaring in the station, and the text of the announcement was scrolling on the clock, blocking the info on when the next train would come. Finally, the answer appeared — after the doors of the 3 train closed and I lost that option.

    • AG says:

      Considering I see people throw trash on the tracks most days – and witness disgusting and a lack of hygiene every day – those announcements should be played all day every day.

  10. AG says:

    One more – the MTA is large but they are far from “conservative”. They do the exact opposite of “conserve”. They waste loads of time and money because of obstinance and NY political folly.
    To answer the question – no – without an overhaul of the system nothing will change. While Democrat machine politics continue to dominate the city and state of NY – what reason do they have to change. If the government of Singapore was imported into NY we would see changes – but that won’t happen anytime soon I fear. If the MTA suddenly became the Hong Kong or Tokyo system then we could have hope… Again – that goes back to the politics.

  11. TimK says:

    Gaffes like this weren’t caught early in part because the MTA “remained unconvinced of the usefulness of what seemed to them an endless review process in the early requirements and design stages. They had the perception that this activity was holding up their job.” They avoided visiting the contractor’s office, which, to make things worse, was overseas. In all, they made one trip. “MTA did not feel it was necessary to closely monitor and audit the contractor’s software-development progress.”

    I posted a link to the original Atlantic piece on Facebook with some ruminations. To my mind, what this (the paragraph I quoted above) describes is a fatal error.

    I was involved in a large software project in Sweden in the mid-1990s that suffered from the same problem. The client wanted the contractor, my American employer, to accept a big book o’ specifications and go off and develop the software without any client input. So we did. Meanwhile, the client’s business environment (mobile telephony) was rapidly changing. So when the software was delivered, it met the specified requirements, but did not meet the client’s needs.

    In any large software development project (and many small ones), regular contact with the client is absolutely essential.

Leave a Reply

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>