Publications

2022

1. NeurIPS

   Iterative Scene Graph Generation

   Siddhesh Khandelwal, and Leonid Sigal

   In Proceedings of Neural Information Processing Systems, 2022

   Abs Bib PDF Supp Code

   The task of scene graph generation entails identifying object entities and their corresponding interaction predicates in a given image (or video). Due to the combinatorially large solution space, existing approaches to scene graph generation assume certain factorization of the joint distribution to make the estimation feasible (e.g., assuming that objects are conditionally independent of predicate predictions). However, this fixed factorization is not ideal under all scenarios (e.g., for images where an object entailed in interaction is small and not discernible on its own). In this work, we propose a novel framework for scene graph generation that addresses this limitation, as well as introduces dynamic conditioning on the image, using message passing in a Markov Random Field. This is implemented as an iterative refinement procedure wherein each modification is conditioned on the graph generated in the previous iteration. This conditioning across refinement steps allows joint reasoning over entities and relations. This framework is realized via a novel and end-to-end trainable transformer-based architecture. In addition, the proposed framework can improve existing approach performance. Through extensive experiments on Visual Genome and Action Genome benchmark datasets we show improved performance on the scene graph generation.

   @inproceedings{khandelwal2022iterative,
     title = {Iterative Scene Graph Generation},
     author = {Khandelwal, Siddhesh and Sigal, Leonid},
     booktitle = {Proceedings of Neural Information Processing Systems},
     year = {2022},
   }

2021

1. NeurIPS

   Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting

   Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khandelwal, and 5 more authors

   In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021

   Abs Bib HTML PDF Code

   We introduce Argoverse 2 (AV2) — a collection of three datasets for perception and forecasting research in the self-driving domain. The annotated Sensor Dataset contains 1,000 sequences of multimodal data, encompassing high-resolution imagery from seven ring cameras, and two stereo cameras in addition to lidar point clouds, and 6-DOF map-aligned pose. Sequences contain 3D cuboid annotations for 26 object categories, all of which are sufficiently-sampled to support training and evaluation of 3D perception models. The Lidar Dataset contains 20,000 sequences of unlabeled lidar point clouds and map-aligned pose. This dataset is the largest ever collection of lidar sensor data and supports self-supervised learning and the emerging task of point cloud forecasting. Finally, the Motion Forecasting Dataset contains 250,000 scenarios mined for interesting and challenging interactions be- tween the autonomous vehicle and other actors in each local scene. Models are tasked with the prediction of future motion for “scored actors" in each scenario and are provided with track histories that capture object location, heading, velocity, and category. In all three datasets, each scenario contains its own HD Map with 3D lane and crosswalk geometry — sourced from data captured in six distinct cities. We believe these datasets will support new and existing machine learning research problems in ways that existing datasets do not. All datasets are released under the CC BY-NC-SA 4.0 license.

   @inproceedings{wilson2021argoverse,
     title = {Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting},
     author = {Wilson, Benjamin and Qi, William and Agarwal, Tanmay and Lambert, John and Singh, Jagjeet and Khandelwal, Siddhesh and Pan, Bowen and Kumar, Ratnesh and Hartnett, Andrew and Pontes, Jhony Kaesemodel and others},
     booktitle = {Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)},
     year = {2021},
   }

2. ICCV

   Segmentation-grounded scene graph generation

   Siddhesh Khandelwal, Mohammed Suhail, and Leonid Sigal

   In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021

   Abs Bib PDF Supp Code

   Scene graph generation has emerged as an important problem in computer vision. While scene graphs provide a grounded representation of objects, their locations and relations in an image, they do so only at the granularity of proposal bounding boxes. In this work, we propose the first, to our knowledge, framework for pixel-level segmentation-grounded scene graph generation. Our framework is agnostic to the underlying scene graph generation method and address the lack of segmentation annotations in target scene graph datasets (e.g., Visual Genome) through transfer and multi-task learning from, and with, an auxiliary dataset (e.g., MS COCO). Specifically, each target object being detected is endowed with a segmentation mask, which is expressed as a lingual-similarity weighted linear combination over categories that have annotations present in an auxiliary dataset. These inferred masks, along with a novel Gaussian attention mechanism which grounds the relations at a pixel-level within the image, allow for improved relation prediction. The entire framework is end-to-end trainable and is learned in a multi-task manner with both target and auxiliary datasets.

   @inproceedings{khandelwal2021segmentation,
     title = {Segmentation-grounded scene graph generation},
     author = {Khandelwal, Siddhesh and Suhail, Mohammed and Sigal, Leonid},
     booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision},
     pages = {15879--15889},
     year = {2021},
   }

3. CVPR

   Unit: Unified knowledge transfer for any-shot object detection and segmentation

   Siddhesh Khandelwal, Raghav Goyal, and Leonid Sigal

   In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021

   Abs Bib PDF Supp Code

   Methods for object detection and segmentation rely on large scale instance-level annotations for training, which are difficult and time-consuming to collect. Efforts to alleviate this look at varying degrees and quality of supervision. Weakly-supervised approaches draw on image-level labels to build detectors/segmentors, while zero/few-shot methods assume abundant instance-level data for a set of base classes, and none to a few examples for novel classes. This taxonomy has largely siloed algorithmic designs. In this work, we aim to bridge this divide by proposing an intuitive and unified semi-supervised model that is applicable to a range of supervision: from zero to a few instance-level samples per novel class. For base classes, our model learns a mapping from weakly-supervised to fully-supervised detectors/segmentors. By learning and leveraging visual and lingual similarities between the novel and base classes, we transfer those mappings to obtain detectors/segmentors for novel classes; refining them with a few novel class instance-level annotated samples, if available. The overall model is end-to-end trainable and highly flexible. Through extensive experiments on MS-COCO and Pascal VOC benchmark datasets we show improved performance in a variety of settings.

   @inproceedings{khandelwal2021unit,
     title = {Unit: Unified knowledge transfer for any-shot object detection and segmentation},
     author = {Khandelwal, Siddhesh and Goyal, Raghav and Sigal, Leonid},
     booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
     pages = {5951--5961},
     year = {2021},
   }

2020

1. ArXiv

   What-if motion prediction for autonomous driving

   Siddhesh Khandelwal, William Qi, Jagjeet Singh, Andrew Hartnett, and Deva Ramanan

   arXiv preprint arXiv:2008.10587, 2020

   Abs Bib PDF Code

   Forecasting the long-term future motion of road actors is a core challenge to the deployment of safe autonomous vehicles (AVs). Viable solutions must account for both the static geometric context, such as road lanes, and dynamic social interactions arising from multiple actors. While recent deep architectures have achieved state-of-the-art performance on distance-based forecasting metrics, these approaches produce forecasts that are predicted without regard to the AV’s intended motion plan. In contrast, we propose a recurrent graph-based attentional approach with interpretable geometric (actor-lane) and social (actor-actor) relationships that supports the injection of counterfactual geometric goals and social contexts. Our model can produce diverse predictions conditioned on hypothetical or "what-if" road lanes and multi-actor interactions. We show that such an approach could be used in the planning loop to reason about unobserved causes or unlikely futures that are directly relevant to the AV’s intended route.

   @article{khandelwal2020if,
     title = {What-if motion prediction for autonomous driving},
     author = {Khandelwal, Siddhesh and Qi, William and Singh, Jagjeet and Hartnett, Andrew and Ramanan, Deva},
     journal = {arXiv preprint arXiv:2008.10587},
     year = {2020},
   }

2019

1. ICCV

   AttentionRNN: a structured spatial attention mechanism

   Siddhesh Khandelwal, and Leonid Sigal

   In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019

   Abs Bib PDF Supp

   Visual attention mechanisms have proven to be integrally important constituent components of many modern deep neural architectures. They provide an efficient and effective way to utilize visual information selectively, which has shown to be especially valuable in multi-modal learning tasks. However, all prior attention frameworks lack the ability to explicitly model structural dependencies among attention variables, making it difficult to predict consistent attention masks. In this paper we develop a novel structured spatial attention mechanism which is end-to-end trainable and can be integrated with any feed-forward convolutional neural network. This proposed AttentionRNN layer explicitly enforces structure over the spatial attention variables by sequentially predicting attention values in the spatial mask in a bi-directional raster-scan and inverse raster-scan order. As a result, each attention value depends not only on local image or contextual information, but also on the previously predicted attention values. Our experiments show consistent quantitative and qualitative improvements on a variety of recognition tasks and datasets; including image categorization, question answering and image generation.

   @inproceedings{khandelwal2019attentionrnn,
     title = {AttentionRNN: a structured spatial attention mechanism},
     author = {Khandelwal, Siddhesh and Sigal, Leonid},
     booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision},
     pages = {3425--3434},
     year = {2019},
   }

2017

1. NeurIPS

   Improving distantly supervised relation extraction using word and entity based attention

   Sharmistha Jat, Siddhesh Khandelwal, and Partha Talukdar

   Automated Knowledge Base Construction Workshop, Neural Information Processing Systems, 2017

   Abs PDF Code

   Relation extraction is the problem of classifying the relationship between two entities in a given sentence. Distant Supervision (DS) is a popular technique for developing relation extractors starting with limited supervision. We note that most of the sentences in the distant supervision relation extraction setting are very long and may benefit from word attention for better sentence representation. Our contributions in this paper are threefold. Firstly, we propose two novel word attention models for distantly- supervised relation extraction: (1) a Bi-directional Gated Recurrent Unit (Bi-GRU) based word attention model (BGWA), (2) an entity-centric attention model (EA), and (3) a combination model which combines multiple complementary models using weighted voting method for improved relation extraction. Secondly, we introduce GDS, a new distant supervision dataset for relation extraction. GDS removes test data noise present in all previous distant- supervision benchmark datasets, making credible automatic evaluation possible. Thirdly, through extensive experiments on multiple real-world datasets, we demonstrate the effectiveness of the proposed methods.

2. ECIR

   Faster k-means cluster estimation

   Siddhesh Khandelwal, and Amit Awekar

   In European Conference on Information Retrieval, 2017

   Abs Bib PDF Code

   There has been considerable work on improving popular clustering algorithm ‘K-means’ in terms of mean squared error (MSE) and speed, both. However, most of the k-means variants tend to compute distance of each data point to each cluster centroid for every iteration. We propose a fast heuristic to overcome this bottleneck with only marginal increase in MSE. We observe that across all iterations of K-means, a data point changes its membership only among a small subset of clusters. Our heuristic predicts such clusters for each data point by looking at nearby clusters after the first iteration of k-means. We augment well known variants of k-means with our heuristic to demonstrate effectiveness of our heuristic. For various synthetic and real-world datasets, our heuristic achieves speed-up of up-to 3 times when compared to efficient variants of k-means.

   @inproceedings{khandelwal2017faster,
     title = {Faster k-means cluster estimation},
     author = {Khandelwal, Siddhesh and Awekar, Amit},
     booktitle = {European Conference on Information Retrieval},
     pages = {520--526},
     year = {2017},
     organization = {Springer},
   }