Multi-modal Learning

Associate director of Bayesian learning team

Gary G. Lee (POSTECH)

Research areas: 

(1) Open Information Extraction

(2) Spoken Language Understanding

Participant professors of Bayesian learning team

Hyeran Byun (Yonsei Univ.)

Research areas: 

(1) Computer vision applications

(2) Food recognition

Seungwon Hwang (POSTECH)

Research areas: 

(1) Cost-based Optimization of Spatial-Keyword Queries

(2) Harvesting Information of Emerging Spatial Entities

In this research center, we are trying to develop a novel Open Information Extraction system (Open IE), which extracts relationships between arguments in an input sentence. The major difference between traditional IE and Open IE is domain independence. Traditional IE requires pre-defined set of relationships whereas Open IE does not. This characteristic enables web scale IE. In this work, we present seed based semi supervised learning to automatically construct training examples from the web corpus without human annotation efforts. We extract seed triples from the web corpus and use the triples to find sentences with high probability of containing relationships the triples have. We also present neural network extraction model. It iterate over all words in an input sentence and classify each words as relation, arguments or nothing. The model uses pre-trained word embedding with position vector indicating relative positions in an input sentence.
One of the main issues on natural language processing is to understand intention from user utterance. We have been developing Spoken Language Understanding (SLU) techniques for speech dialogue system which assures sufficient robustness and flexibility. We present a new language model adaptation method to recover in-complete inputs from speech. Several machine learning methods have been applied for our supervised/unsupervised SLU systems to extract semantic concept from users’ utterances. We also have developed several techniques consists of linguistic feature process, information extraction, relational data learning to construct SLU techniques.
In this research center, we design a cost-based optimizer to process spatial-keyword queries. While there are several dedicated indices for spatial-keyword queries, each index is designed for a specific circumstance, and thus it may not perform well on some queries. Instead of proposing entirely new index, we enumerate possible query plans that consist of using one or more indices, and choose the best one. For this task, we design a cost model using static features to choose the best plan. In particular, we leverage selectivity of keyword and spatial predicates, and their correlations. Based on the cost model, we devised five optimization techniques to produce a plan theoretically close to optimal. Figure 1 shows an example of optimization, and the optimized plan performs about 79 times faster than a base plan. Our quantitative evaluation shows that our optimization techniques achieve 9- to 11-fold speedup for 99th percentile query response time over diverse datasets.
In Foursquare or Google+ Local, emerging spatial entities, such as new business or venue, are reported to grow by 1% every day. As the information on such spatial entities is initially limited, we need to quickly harvest related information from social media, such as Flickr photos. In this research center, we develop a framework to automatically harvest information of spatial entities from social media such as Flickr photos, by aggregating location, image, and text evidences. The framework consists of two components, CTM of deriving entity linking through photos and STM of finding synonymous entity names, which can reinforce each other in a hybrid scheme. Compared with state-of-the-art baselines, our proposed framework improves up to 24% and 18%, respectively, in recall and F1 score. We also leverage this research for travel route recommendation by matching harvested information of spatial entities with user’s preference query. To provide diverse query results, we develop a travel route recommendation framework, KSTR, which adopts skyline concepts to rank routes.