Open Source At Bloomberg: Solr And Our Work To Enhance Enterprise Search

About the author: Steven Bower has worked for 15 years in the web/enterprise search industry – First as part of the R&D and Services teams at FAST Search and Transfer, Inc. and then as a principal engineer at Attivio, Inc. He has participated/led the delivery of hundreds of search applications and now leads the search infrastructure team at Bloomberg LP, providing a search as a service platform for 100+ applications. 

Improving and enhancing search capabilities for our customers has become an area of focus for Bloomberg from both a product and technological perspective. We are constantly looking at ways to enhance our user experience by increasing search capabilities for subscribers of the Bloomberg Professional service (aka the terminal).

As a result, Bloomberg is an active contributor to Apache Solr – an open source project for enterprise search technologies.  In fact, we recently submitted an entire new analytics framework that has been accepted by the Solr community, which will be part of the 5.0 release.

For those of you who attended the Solr/Lucene Revolution 2014 conference in Washington DC in November you know that we shared – for the first time ever —  our approach to open source search and the work we are doing for this effort. Harry Hight,  Anirudha Jadhav and I each gave presentations on the different  pieces we are working on here at Bloomberg.

Harry presented on the complexities and approach used in designing a large-scale, multi-tenant architecture to support one of our enterprise products, Bloomberg Vault.  Since the scale at which this system operates is well beyond most common applications of Solr, there was tremendous interest from the attendees into our approach. People asked a number of questions about the capability we created that allows us to have huge numbers of shards offline and bring them online only when needed.

My colleague Anirudha presented on the work he has done to support high volume, low latency indexing in Solr as well as some ideas he’s been working on to create a system to automatically tune ingestion performance.

My own presentation was on the Solr Analytics module that I co-developed here at Bloomberg. Performing analytics within Solr is an area of great development within the community and generated a number of interesting ideas and potential collaborations.

The conference itself was an excellent opportunity to connect further with the Solr community and both share our efforts to date and learn more about what is going on with other projects. The conference provided a great mix of in-depth technical presentations and very interesting use-case presentations. This led to many creative brainstorming conversations and discussions around planning future collaborations with members of the Solr community.

For those of you who missed the conference, we will continue to share more broadly the role search plays at Bloomberg and our interaction with Solr. We will be hosting ongoing Solr/Lucene Meetups in San Fransisco and New York City where we will discuss some of our  interesting use-cases and development as well as hosting others in the Solr/Lucene community.

We hope you can attend.