Generating Referring Expressions

Author : Dan Schulman
Publisher :
ISBN 13 :
Total Pages : 72 pages
Book Rating : 4.:/5 (129 download)

DOWNLOAD NOW!

Download or read book Recognizing and Generating Natural Language Referring Expressions in Images written by Dan Schulman and published by . This book was released on 2019 with total page 72 pages. Available in PDF, EPUB and Kindle. Book excerpt: In this work we tackle the tasks of recognizing and generating natural language referring expressions in images. In the first task, given an image with several objects and an expression referring to a specific object within the image, we would like to recognize this object by retrieving a bounding box enclosing it. In the second task, given an image and a bounding box enclosing a specific object, we would like to generate an unambiguous expression describing the object. The datasets for these tasks are built using an interactive game called ReferIt (Kazemzadeh et al., 2014). In this work, we focus on two of the datasets in this family, RefCOCO and RefClef. While RefClef contains a variety of object types including background such as sky and clouds, which are hard to detect, RefCOCO contains mainly day-to-day objects, such as persons, cars and cats, which are easier to detect. Recent works focus mainly on RefCOCO variants and tend to ignore RefClef. The general pipeline for these tasks adopted by most approaches consists of the following steps: (1) Retrieve candidate object bounding boxes, either by using the gold-truth from the dataset, or by generating candidates using methods such as EdgeBox (Zitnick and Dollar, 2014) and MaskRCNN (He et al., 2017); (2) Encode both the entire image and each of the candidates using a Convolutional-Neural-Network (CNN); (3) Encode the referring expression using word embeddings and a Recurrent-Neural-Network (RNN); (4) Compute a score for each candidate based on the previous encodings and choose the highest scoring bounding-box. We reproduce state of the art models on the two tasks of natural language referring expressions recognition and generation in images and introduce two specific improvements: We show an automated way to augment an existing dataset, multiplying its size by two. We also show that by replacing the existing visual encoder component with a better one, we achieve better results. We compare our results with the current state-of-the-art in both tasks. On the basis of our experiments and dataset augmentation method, we investigate the differences between the existing datasets. We identify that segmentation methods that are aware of the semantic labels for the objects, such as MaskRCNN, provide better results on RefCOCO, while a more generalist method such as EdgeBox is more robust on the more challenging RefClef dataset. We also identify subsets of the dataset which are a priori more challenging for the family of neural architectures we have identified - because they lack a form of symmetry which simplifies the task. -- abstract.