Meet the Team: Data Services Engineering

The rise of machine learning and automation, coupled with an increased availability of data, has led to a renaissance in data analytics. Bloomberg’s rapidly growing Data Services Engineering team is responsible for core data and analytics services that run on over a thousand machines that serve over 300 billion requests every day. This team of engineers is responsible for putting data at the fingertips of Bloomberg’s clients, enabling them to perform world-class financial analysis and make smarter decisions.

Let’s meet Manish Nair, who is leading the charge to maintain and scale Bloomberg’s data infrastructure, an important backbone for the Bloomberg application stack.

Tell us about the Data Services team.
We are responsible for a set of core services and standard APIs that engineers across Bloomberg’s Engineering department use to access financial data to develop various Bloomberg applications. We serve current and historical time series for reference data, market data, and standard analytics for all financial instruments. We deal with unique challenges related to large volume requests, low latency data access, high availability and scalability, as well as overall system reliability for a large-scale distributed systems environment that is in use across our entire enterprise.

Our key offering is the API Gateway Service, which is used by all applications across the firm, including the Bloomberg Terminal, mobile applications, Excel API, and enterprise data. We also maintain and manage Metadata, Symbology, ID Master, Entitlements, and the Data Dictionary for Bloomberg financial content.

What are some of the factors driving the rapid expansion of your team?
We are upgrading our system to better support a variety of non-traditional datasets. We are enhancing our metadata systems to make our data more self-describing and machine-readable. We have also invested in building an anomaly detection platform to further improve the quality of our data and to catch errors earlier in the development lifecycle. 

We are upgrading the APIs and tools that are offered to both clients and providers to our Gateway. We are in the process of breaking up parts of our system that are built as a monolith into more manageable microservices that can be independently rolled out and scaled horizontally. 

What skills do you look for when hiring engineers for your team?
Apart from coding fluency in at least one programming language, we look for an engineering mindset and a solid grasp of computer science fundamentals. We like to see an ability to break down a real-life problem and model it in software. They need to be able to reason logically and communicate technical design effectively. Also important is the ability to think big and strategically, while not dropping any of the important details or edge cases.  

We try to assess what kind of opportunities candidates have had in the past and how they’ve taken advantage of these opportunities to learn and improve themselves. We look for instances where they demonstrated determination and grit. Ownership is a very important aspect of the engineering culture at Bloomberg and we consider persistence to be a good indicator of how well they perform with this responsibility. 

How do you foster culture on your team?
We have a very collaborative culture in our teams. We are organized into eight scrums and follow Agile software development methodologies. Each scrum has regular syncs, including daily stand-ups, grooming, sprint planning, and retrospectives. We also do a multi-scrum sprint review every two weeks. All of these Agile ceremonies, including our meetups and quarterly planning processes, have been adapted so that we have been able to continue doing them virtually during the pandemic.

We prioritize learning and we have multiple avenues for engineers to connect and learn from each other. We have kept these alive even during the pandemic. We continue to do our weekly tech talks, book clubs, and lunch & learn sessions, all of which have been adapted for work from home.

“We deal with unique challenges related to large volume requests, low latency data access, high availability and scalability, as well as overall system reliability for a large-scale distributed systems environment that is in use across our entire enterprise.”

– Manish Nair

Software Engineer Rupali Saboo joined Bloomberg after studying computer science at Johns Hopkins University. She has twice presented at the Grace Hopper Celebration together with other engineers from Data Services — in 2018, she co-presented a poster on deep learning-based text anomaly detection, and in 2019, she helped lead a workshop about interpreting machine learning models.

What is your background?
I had the opportunity to work on the Anomaly Detection team within Data Services right after completing Bloomberg’s new hire training program. This was a challenging and exciting project where we implemented anomaly detection algorithms using distributed analytics and machine learning. Throughout my time here, I received a lot of help and mentorship from many people across the team, as well as from members of Bloomberg Women in Technology (BWIT), which helped me every step of the way. With everyone’s encouragement, I was also able to begin working towards my goal of committing to open source projects.

What project are you working on now and how was the transition?
After approximately three years of working in Anomaly Detection, I wanted to explore other areas in Data Services. I was excited about the Data Gateway project, which was part of a big effort in Data Services and across Bloomberg to dramatically change how calculated data is implemented and accessed. My transition to the new team was seamless, and I was working there within a couple of weeks of discussing this move with my original manager.

Beyond work, what are some initiatives you are involved in?
I strongly believe in working at the grassroots level to help bring diverse views to our team. Alongside work, I was always interested in helping with recruiting. Since last year, I have taken a leading role in recruiting at Johns Hopkins University. In this role, I feel I am able to bring change to our recruiting strategy at the campus hiring level and can make an effort to ensure our hiring process is fair for people of all backgrounds. Within the larger umbrella organization of Data Services (Data and Analytics (DNA) Platform), I also helped start the Women in DNA community, which aims to build a space for women and allies in our area to network, learn from each other, and find mentors. One of our successful initiatives was working directly with engineers in DNA to encourage more of them to submit our work to be presented during the Grace Hopper Celebration.

“Throughout my time here, I received a lot of help and mentorship from many people across the team, as well as from members of Bloomberg Women in Technology (BWIT), which helped me every step of the way.”

– Rupali Saboo

Yogesh Arora joined Bloomberg in 2010 and now leads a group of engineers who are working on rebuilding some of the core parts of Bloomberg’s data infrastructure.

What motivated you to choose Bloomberg? And what has kept you here?
One of my previous colleagues recommended Bloomberg, and he referred me for a role here. I am really glad he did. Bloomberg has a very strong engineering culture, which focuses on hiring talented people and giving them ample opportunities to build some of the most innovative solutions for global financial markets.

One of the most important things has been internal mobility. When you feel ready for a fresh challenge or a new team, you are encouraged to explore opportunities internally. Managers are more than happy to take the time to discuss what their teams are working on and to see if it is a good match for you.

How have you been able to manage your career over the past decade?
Over the years, I have tried to follow a simple mantra to manage and grow my career: I evaluate if my impact within Bloomberg is more than before. If that is not the case, then I look for opportunities where I can have more impact. Moving to the Data Services team gave me those kinds of opportunities. 

Tell us about what you’re working on now.
Our team works on a Data Gateway which is used by a large number of Bloomberg applications for their data needs. In a way, it is one of the central pieces of the Bloomberg application stack.

Our biggest technical challenges are related to the scale at which our infrastructure operates. Ours is a massively distributed system that serves hundreds of billions of data queries running across thousands of machines.

After earning her degree in Mechanical Engineering and getting some industry experience, Connie Yee studied Operations Research in grad school. She then joined Bloomberg as a data scientist and is now working on data-driven ways to improve our infrastructure.

What’s it like to be a data scientist at Bloomberg?
Having worked in the software field for nine years, I have come to appreciate and value the highly collaborative engineering culture at Bloomberg. As a data scientist, I have ample tools and internal systems at my disposal, such as the Bloomberg Data Science Platform and the Hadoop ecosystem, both of which make training models and running experiments easy. In addition, through internal meetups, Guilds and external conferences, both data scientists and engineers are encouraged to share knowledge with others who might be working on similar problems.

What are some of the unique opportunities you’ve seen to apply data science to our business?
In Data Services, we are applying data science in a number of areas while tackling challenges with scaling our algorithms and optimizing their performance.

First, we build models to detect anomalies in our financial data and infrastructure metrics. Machine learning enables us to automatically identify and then correct or make decisions based on the anomalies found in the vast amount of heterogeneous data.

Second, we are augmenting our financial datasets by inferring metadata information, such as attributes that are correlated or commonly used together. This information can reveal relationships in the data that help us improve data management and quality.

A third area we are exploring is the use of graph theory to gain actionable insights on call graphs, which are representations of code dependencies within a binary application. 

You’ve worked in both Global Data and Engineering. How are they similar and different?
I am fortunate to work at Bloomberg, which offers different areas for a data scientist to make an impact. In both teams, I’ve had first-hand experience collaborating on cross-departmental projects. In Global Data, I worked closely with the data teams that directly communicate with our clients and stock exchanges, as well as the engineering teams to productionize solutions. The data science projects there are centered on data quality, workflow automation, and business intelligence.

Now that I’m in Engineering, I am closer to the applications that provide data to our clients and the infrastructure on which the applications run. This has opened up even more diverse data science projects, which run the gamut from extracting information out of unstructured data to optimizing job scheduling algorithms.

“Through internal meetups, Guilds and external conferences, both data scientists and engineers are encouraged to share knowledge with others who might be working on similar problems.”

– Connie Y

Jordan De Gea joined Bloomberg after finishing his master’s degree in Computer Science. Since then, Jordan has contributed to expanding the Data Services team in London and now leads efforts in Data Services’ permissions systems.

You interned at Bloomberg, got a job elsewhere, and then decided to come work at Bloomberg full time. What made you ultimately choose to work at Bloomberg?
Working as an intern for Bloomberg in London was an adventure for me. I wanted to discover a different country and challenge myself. But I liked my home country of France, and started my career there.

However, my experience with Bloomberg was so nice that I decided to ultimately come back. I was surrounded by smart, interesting, and interested people; the work was challenging and the network was gigantic. 

What’s your best advice for inspiring new college grads who have recently joined Bloomberg?
Take care of your career and improve your network. I had the chance to meet different people in diverse areas and it was enriching. When you work on projects involving different teams — especially when it’s a team outside your domain — you have to learn to understand their needs and problems, which can vary greatly from yours.

At Bloomberg, it is easy to reach out to your mentors, as well as team leaders and managers, and not only your own. Early on, I remember asking to meet different senior people in our management team. I was happily surprised that they accepted and I was able to talk about my growth with them.

Senior Software Engineer Rakesh Guttikonda joined Bloomberg’s Data Services team in early 2016, when it was just a team of five engineers. He has worked on a number of key projects that were presented to the Global Head of Engineering.

Can you tell us about your journey at Bloomberg? What kind of problems have you worked on?
My first project at Bloomberg was to add a major feature to one of the core services that handles hundreds of billions of requests per day, and also one of the core infrastructure systems at Bloomberg that is highly distributed, scalable, and reliable. I thoroughly enjoyed working on a project with such high impact, and learned many things during the process. Stability and reliability are some of the most important aspects for such critical systems. I had the opportunity to work on a wide range of problems in these areas as well.

What is it like to take up new opportunities or move to a different office at Bloomberg?
I was excited when I found an opportunity within our team that was expanding in the London office. After a few discussions with my manager, I was encouraged to pursue the opportunity, and Bloomberg fully supported me in transitioning to the role, including helping me with my relocation, including immigration and accommodation.

Our team in the London office has grown to more than 10 members in a short period and has a wide range of responsibilities, from working on core distributed services to having SRE responsibilities. Now I am starting a new team here that will be responsible for building the infrastructure for exploration and distribution of metadata. Metadata is as important as the data itself; the complexity of capturing relationships within the data makes it even more challenging. I do see a lot of growth opportunities in this area and we are currently looking to hire for our team.