Generative adversarial networks (GANs) are a relatively new class of machine learning systems. First introduced in 2014, these deep neural nets have mostly been used to create lifelike images and videos. It is only since about 2017 that GANs have been used for text generation.
In a paper entitled “Keyphrase Generation for Scientific Articles Using GANs,” which was presented at the Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20) in New York City (February 7-12, 2020), a team of research scientists from Bloomberg’s AI Group have shown for the first time that GANs can be used to generate keyphrases from scientific abstracts.
The Bloomberg team completed this research in collaboration with students and researchers at the Multimodal Digital Media Analysis (MIDAS) Lab at Indraprastha Institute of Information Technology, Delhi (IIIT-Delhi), where Debanjan Mahata, one of the Bloomberg researchers, is an adjunct faculty member.
“We are the first ones exploring the use of GANs in keyphrase generation,” says Haimin (Raymond) Zhang, one of the Bloomberg research scientists who co-authored the paper.
The GAN structure is unique in that it pits two neural networks against each other. One network, called the generator, comes up with possible answers to a problem. The second network, called a discriminator, tries to distinguish those answers from ones created by a human. Over time, the generator learns from the discriminator, and comes up with candidates that are ever closer to those proposed by a person. Intuitively, GANs make sense: They use a machine to teach a machine.
“People still have some doubts about the use of GANs to generate text,” notes Rakesh Gosangi, another Bloomberg researcher who co-authored the paper. “But, in problems specifically related to the summarization of texts, GANs have had some promising results.”
“There was a clear opportunity to try this out on text, and specifically in keyphrase generation,” says Mahata. Keyphrase generation, or the process of coming up with natural language phrases that accurately represent a larger body of text, is a foundational task in natural language processing.
The Bloomberg team developed the architecture for the GANs, and advised students Avinash Swaminathan and Raj Kuwar Gupta who were interning at MIDAS Lab, which is led by Dr. Rajiv Ratn Shah, Assistant Professor in the Department of Computer Science and Engineering at IIIT-Delhi.
The GANs were put to work on four different data sets, containing thousands of abstracts of scientific papers, their titles, and their human-curated keyphrases. The model obtains state-of-the-art results in terms of accuracy, but it does better in terms of keyphrase diversity in three out of four datasets.
One of the main challenges in using GANs for this task lies in so-called backpropagation, a technique used for training neural networks. While used widely in training GANs, backpropagation cannot be easily adapted to text because the outputs from the generator are discrete, in that they are not differentiable. To address this problem, they use reinforcement learning to train the discriminator.
Another challenge, says Zhang, is to use the right discriminator. For this research, the Bloomberg team designed the discriminator from scratch. Zhang sees a lot of opportunity to innovate on this discriminator, possibly by changing the way it provides rewards to the generator. Another option would be to use multiple discriminators to teach the generator, or to combine them in different ways. The research team is already at work trying to make the deep neural nets more effective.
“This is an important problem that needed some attention and had to be tried,” says Mahata.
Added Zhang, “By doing this initial analysis and generating these preliminary results, we can see the potential and are working on additional research to further establish whether GANs are the right approach for this problem.”
Read some of the other Bloomberg co-authored research papers presented at AAAI 2020 and additional peer-reviewed academic conferences at journals here.