A Conversation about Developer Experience with Lei Zhang

Lei Zhang became the head of Bloomberg’s Developer Experience group in the company’s Software Infrastructure Engineering department in November 2017 after being with the company for 10 years. He started as a software engineer in the BVAL team, and then joined the Derivatives group, where he held a variety of roles. From 2015 to 2017, Lei led the Derivatives Data team, where he transformed the team from a traditional library provider with highly operational data processing into an area where large scale data platforms and data science techniques are heavily employed to provide market-leading data services for all Bloomberg pricing and risk platforms. Before that, Lei led the effort of Derivatives Library to provide structuring and valuation for cross-asset derivatives and structured notes. Lei has always looked for creative ways to automate processes and make software development simpler and more enjoyable. We spoke with him about his team and the role they play in helping Bloomberg’s engineers manage the company’s codebase.

Q: What is Developer Experience (DevX)? How do you define DevX?

Developer Experience means ensuring that our developers can work productively. The broader term encompasses all of the developer-facing products, libraries, tools, and APIs that exist to ensure developers have a good experience. DevX is not just tools. It is also about using them effectively. That means consistent and intuitive interfaces, as well as clear documentation, so developers can adopt new workflows and get the most value from them with the minimum effort.

My team at Bloomberg exists to level up our firm-wide engineering effectiveness. We build world-class tools to enable a streamlined developer experience, enhance operational visibility, and suggest best practices for Bloomberg’s more than 5,000 developers around the globe. We want the right thing to be easy to do.

Q: Your DevX team focuses on Build, Deployment, and Developer Workflow. Could you elaborate a bit more about this?

Bloomberg prides itself on our ability to deliver advanced functionality into the hands of our customers quickly. Our development agility is critical to our success, so we take a great deal of care to ensure we enable developers to be both productive and adhere to best practices.

Our codebase is huge, consisting of many libraries and applications. Teams often need to coordinate changes across departments, so being able to quickly build and deploy changes at scale is very important to us. We also take care to maintain the ability to roll out changes in response to customers’ needs – sometimes even the same day – without compromising the stability of our production environment.

Finally, the whole developer workflow drives our software development lifecycle. It allows our developers to move quickly to meet customer needs, while also having the confidence that their workflow is reinforcing best practices and helping them detect and eliminate problems in the code early. An efficient code-build-test cycle coupled with a transparent deployment process is the best way we can help our developers be productive.

With effective solutions around Build, Deployment, and Workflow, DevX is a force multiplier on the creativity of our development community.

Q: Your DevX team supports many developers with different workflows. Why does effective communication and documentation matter?

Effective communication is essential to our success. A clear understanding of our developers’ challenges helps us to shape our product direction and prioritize our efforts. Sharing our initiatives and roadmaps ensures our users are equipped to implement the best decisions and workflows for their needs. Communication and information gathering are primary activities and not just an afterthought: we communicate our direction and important releases across a range of channels, chat groups, and blog posts to reach as much of our community as we can. We are ramping up efforts to collect both user feedback and metrics about the different tools our developers are using, so we can track how successful we have been.

The quality of documentation can strongly influence a product’s success. As a tool and system provider, we believe our documents are our voice to our users – the value added by comprehensive documentation for users is tremendous. Our training material is a good place to start when looking to learn about one of our offerings. Once a developer becomes familiar with a tool and wants to go deeper to understand the tradeoffs or how it works, we provide FAQs and reference content to point at solutions for common problems. We also make all of our documentation available for editing and collaboration, so our users can point out gaps and provide insight into how our tools are used.

When developers can solve their problems by reading documentation, our team has more time to build tools and communicate about unique aspects of our developer workflow.

Q: What are some of the principles that you follow with regards to simplicity, usability, and innovation?

We want everything we offer to be easy to adopt. The simpler a tool or workflow is to adopt and use, the easier it is for a busy development team to get value from it. This isn’t just a nice-to-have property, it’s essential. We talk internally about getting out of the engineer’s way. During the implementation stage, we eliminate unnecessary steps to have intuitive, self-explanatory use-cases backed by documentation, so that our users have a great out-of-box experience.

Being able to scale horizontally is an important part of usability too. Once a team starts using one of our offerings, we want their ongoing experience to be great too. If everything works great with 100 users and then things become unreliable with 5,000+ users, this drags down our engineers’ productivity. We want to ensure that whenever load increases, we can add resources without having to redesign the system.

We innovate by listening. Often, we encounter development teams with unique ideas about their workflow. Often, those ideas won’t translate well to the company as a whole, but there is very often an innovative element that is valuable and we don’t want to overlook it. It is easy to dismiss an idea as unworkable, but true innovation comes from understanding our developers, however diverse their practices are. Some of our greatest ideas have come from ‘outlier’ teams.

Q: What do you do as Head of Developer Experience?

Our area has grown rapidly over the last year. I focus on building a strong team with a shared vision. We want to understand our developers’ and senior stakeholders’ needs, and we want to create effective tools and have a deep partnership with our developer community.

I also stay current on industry trends by attending both internal and external events and conferences, and we talk to teams from other technology firms with similar challenges. We are increasingly presenting at conferences too. We are aware of different solutions and want to adopt best practices from others, in addition to contributing our best practices to the industry.

Q: What types of systems and products do you work on?

The Developer Experience team provides developer tools and workflows at every step of the software development lifecycle – from inception to deployment. Our efforts fall into two main areas: building out powerful and easy to use tooling for specific parts of the development workflow, such as peer review, static analysis, or testing, and integrating those steps together into an easy to use end-to-end workflow.

Bloomberg has a diverse product stack with many unique requirements. As a result, there are many different types of build solutions with unique language support and dependency management that we need to cater for. We try to simplify the tools a developer needs to know by offering a hybrid solution so that very large scale builds can happen correctly and efficiently. Underneath, we are investing significantly in providing a scalable build infrastructure that supports our need to continuously build and test at scale.

We want our developers to be encouraged to improve the quality of code wherever possible, and not feel they have to choose between delivering a product and good engineering. Providing best-of-breed code quality and automated refactoring tools means our developers don’t have to compromise.

In DevX, we are also offering a platform-as-a-service solution that offers scalable clusters as a deployment endpoint, along with the monitoring and automation to operate them effectively.

Now, why is integration of all these parts important? Our top-level solution aims to provide a cohesive experience across the entire workflow and toolset to accelerate development and improve quality. With an off-the-shelf solution, our developers can create or import a project with a simple click. Development, analysis, publishing, and release are all fully integrated and automated. The Continuous Integration experience is also mirrored in the local development process for pre-commit checks and debugging. Developers spend less time deciding how to build and deploy their products, and more time implementing features.

Because they have a turn-key solution they can depend on without having to extensively customize, developers can deploy more frequently and with more confidence, meaning they can implement features faster and respond to customer feedback quickly.

The ability for a developer to trust that ‘your workflow has your back’ is a major boost to productivity that is hard to overstate.

Q: What types of tools do you use? 

Aside from the Bloomberg Terminal and a number of homemade solutions, we strive to use open source tools for both our own and our developers’ use wherever it makes sense to do so. This includes Git repository hosting (in the form of GitHub Enterprise) for source control; build systems like CMake; Kubernetes to automate the deployment and scaling of containerized applications; Docker to run applications in containers; Google Test and Google Mock for testing; Clang Tools for static analysis; Jenkins for Continuous Integration/Continuous Delivery; and Grafana for system-wide telemetry.

We also make extensive use of Kafka for data/event streaming, which allows us to integrate all of our tools and offerings inexpensively.

Q: How does DevX collaborate with the open source community?

While we focus on some unique challenges within Bloomberg Engineering, many of the problems we attempt to solve would benefit a wider community. We successfully hosted a Bazel Hack Day in New York late last year and a Build Meetup in London where engineers from different technology companies collaborated on Bazel enhancements and bug fixes, as well as remote execution client and server implementations, such as BuildFarm, BuildGrid and ‘recc’. Ideas were exchanged, bugs were squashed and documentation was clarified.

We also make a lot of contributions to the open source tools we use, such as the clangmetatool we recently released to the LLVM community.

Q: How have you improved the development environment through DevX?

Our fundamental idea is to enable high-velocity development by adopting better solutions while also reducing the cognitive load on teams to do so.

One example is that we are now offering a managed Continuous Integration solution that provides pre-set pipelines across a range of project types. This reduces the complexity of setup and enables standardization across groups with large numbers of repositories. In many cases, the CI system can auto-sense the nature of a project, so the adoption effort is reduced to close to zero. Similarly, we released a Local Development Bootstrapper to reduce the time that engineers spend setting up their workstations and laptops.

We recently released the “Run-Static-Analysis-Tools (RSAT)” framework. This enables our developers to run many types of static analysis in a standardized way against codebases in different languages. So instead of having to learn every tool, our developers can simply run RSAT and generate analysis reports in a normalized format. This also lets us introduce new static analysis tools and code quality initiatives across our codebase, without requiring any additional integration effort required by participating teams.

Q: What unique challenges do you face scaling developer tools for 5,000+ engineers around the globe? 

We have a heterogeneous environment at Bloomberg, and our ecosystem supports thousands of different applications, including finance applications and communication tools. Our very creative engineers use different technologies for their solutions, some of which are domain-specific, and supporting this diversity can be very challenging. Talking with the different teams across the company to understand their unique needs and challenges is important to help us prioritize appropriately. This also helps us identify common challenges to address that will benefit the most teams.

This kind of computing environment and wide variety of applications also make having a scalable development environment challenging. Rather than choose between a prescriptive solution that might work for 50 percent of our developers and a solution with very easy customization and added functionality, we offer both. That choice brings its own challenges in terms of balancing the needs of different teams and investing in common workflows, but it is one we make consciously.