Every day, Bloomberg handles a staggering amount of financial data: about 100 billion separate market data updates. Consider that that data, which forms the backbone of the Bloomberg Professional Service (or Bloomberg Terminal), arrives from thousands of different feeds, each tied to a different exchange or another source around the globe. A persistence and aggregation layer, or Ticker Plant, processes this data, stores it, adds business value (such as arranging it in a time series), and sends it on so it’s available to Bloomberg customers. And this all happens in milliseconds.
Garry Ryan is Bloomberg’s Global Feeds SRE Manager, and his team is responsible for the availability and stability of every feed, and every bit of data. “Without us, Bloomberg doesn’t exist,” he says. “It’s as simple as that.”
Ryan arrived at Bloomberg a little over three years ago, as a feeds development manager on the Engineering team in London. There, he saw the need to better automate the work that the Feeds team was doing. When Bloomberg ramped up its SRE initiative about 18 months ago, Ryan was asked to head up SRE globally for this mission-critical team, which has members in London, New York, Tokyo and Hong Kong.
Ryan says SRE work is uniquely high-profile at Bloomberg, as the SRE initiative is being championed by the company’s global head of engineering, Vlad Kliatchko. “The Feeds SRE team is adding real value and the SREs get that sense,” Ryan says. “It feels like you’re making a difference almost every day.”
“People often think of SRE as an operational function,” he says. “At Bloomberg, that’s absolutely not the case.”
Of course, his team does ‘production support.’ Each person on his team does that for one day every two weeks. Because the team is split across time zones, they’re rarely ever woken up in the middle of the night to fix something, because a colleague in another region is already working on it.
Ryan’s team spends the vast majority of their time improving Bloomberg’s systems. The goal is for the systems to be able to fail and recover on their own, rather than relying on human intervention. This will improve Bloomberg’s customer experience, in addition to giving Ryan’s team more time to devote to automation, thereby creating a virtuous cycle.
Today, Ryan’s team has three main priorities. First, he says, is to “automate the hell out of everything.” The team has historically relied on manual procedures to handle support tickets, but with tens of thousands of tickets a year and a relatively small team, that doesn’t make sense. Ryan expects to halve the number of manual tickets his team handles this year, while driving that even lower going forward. The team will do that by preventing as many tickets as possible and then automating as much of the remainder as they can.
Next is to automate the deployment of Bloomberg’s parsers. Bloomberg’s Feeds Development Team can easily have 25 releases a week; next year that could go to 75. With an aggressive schedule like that, no parser can be a diva. They all have to fail the same way, recover the same way, and be maintained and monitored the same way. As Ryan puts it, right now, each piece of software is like a ‘pet’, with its own proclivities and tendencies. Ryan would prefer ‘cattle’, so his team can monitor and maintain them en masse. Automation has another advantage: it allows the team to make sure best practices are always followed.
Finally, there’s the development and deployment of new monitoring systems. These will piggyback on the standardized software binaries that Ryan and his team are deploying, making it even easier and faster to diagnose and remediate problems. These telemetry systems, designed in partnership with the Feeds team, will form the basis for company-wide standards for the consistent implementation of code.
Ryan’s team is diverse across gender, race, and culture. In addition to representing 12 different nationalities, they also represent many different career paths. Some started as software engineers like Ryan, some had their start as systems administrators, and others joined after completing their Ph.D. There’s also an army veteran who served in a bomb disposal unit, an ex-football events manager, and an ex-rugby coach. And that’s in addition to others with more traditional backgrounds in software development, testing, and technical operations.
“The inclusive environment we work in at Bloomberg attracts a unique blend of talent,” Ryan notes. “This helps us approach problems from different angles and address them in novel ways.”
One of the key characteristics of the team, Ryan says, is laziness. “I mean that genuinely,” he says. Ryan doesn’t want someone who’s willing to do the same work over and over. He wants someone who is going to automate that work instead. He says his team members are also generally tech enthusiasts who enjoy playing with open source software and testing the newest advances in the field. The team mostly does its development work using Python, but he’s willing to transition people whose experience is in other languages.
An SRE, he says, “is a developer at heart, but one who’s focused on avoiding and solving production systems problems.” Really, he says, “We’re just looking for people who get a kick out of improving things.”