FlowchartQA: The First Large-Scale Benchmark for Reasoning over Flowcharts

Simon Tannert; Marcelo Feighelstein; Jasmina Bogojeska; Joseph Shtok; Assaf Arbelle; Peter W.J. Staar; Anika Schumann; Jonas Kuhn; Leonid Karlinsky

FlowchartQA: The First Large-Scale Benchmark for Reasoning over Flowcharts

Simon Tannert, Marcelo Feighelstein, Jasmina Bogojeska, Joseph Shtok, Assaf Arbelle, Peter W.J. Staar, Anika Schumann, Jonas Kuhn, Leonid Karlinsky

Research output: Contribution to conference › Paper › peer-review

Abstract

In this paper, we present FlowchartQA, a new and unique large-scale benchmark for visual question answering (VQA) over flowcharts. FlowchartQA comprises close to 1M flowchart images and 6M question-answer pairs, covering various aspects of geometric and topological information contained in the charts. The questions have been carefully balanced to minimize biases. To accompany the proposed benchmark, we present a baseline model and perform comprehensive ablation studies and qualitative analyses to provide a solid foundation for future work. Our experimental results reveal interesting findings and demonstrate the potential of FlowchartQA as a testbed for flowchart understanding, which has been previously absent in the community.

Original language	English
Pages	34-46
Number of pages	13
State	Published - 2023
Externally published	Yes
Event	1st Workshop on Linguistic Insights from and for Multimodal Language Processing, LIMO 2023 - Ingolstadt, Germany Duration: 22 Sep 2023 → …

Conference

Conference	1st Workshop on Linguistic Insights from and for Multimodal Language Processing, LIMO 2023
Country/Territory	Germany
City	Ingolstadt
Period	22/09/23 → …

Bibliographical note

Publisher Copyright:
©2023 Association for Computational Linguistics.

ASJC Scopus subject areas

Linguistics and Language
Language and Linguistics

Cite this

@conference{68c2a13ac03a4735bf3f562e04553eaf,

title = "FlowchartQA: The First Large-Scale Benchmark for Reasoning over Flowcharts",

abstract = "In this paper, we present FlowchartQA, a new and unique large-scale benchmark for visual question answering (VQA) over flowcharts. FlowchartQA comprises close to 1M flowchart images and 6M question-answer pairs, covering various aspects of geometric and topological information contained in the charts. The questions have been carefully balanced to minimize biases. To accompany the proposed benchmark, we present a baseline model and perform comprehensive ablation studies and qualitative analyses to provide a solid foundation for future work. Our experimental results reveal interesting findings and demonstrate the potential of FlowchartQA as a testbed for flowchart understanding, which has been previously absent in the community.",

author = "Simon Tannert and Marcelo Feighelstein and Jasmina Bogojeska and Joseph Shtok and Assaf Arbelle and Staar, {Peter W.J.} and Anika Schumann and Jonas Kuhn and Leonid Karlinsky",

note = "Publisher Copyright: {\textcopyright}2023 Association for Computational Linguistics.; 1st Workshop on Linguistic Insights from and for Multimodal Language Processing, LIMO 2023 ; Conference date: 22-09-2023",

year = "2023",

language = "English",

pages = "34--46",

}

TY - CONF

T1 - FlowchartQA

T2 - 1st Workshop on Linguistic Insights from and for Multimodal Language Processing, LIMO 2023

AU - Tannert, Simon

AU - Feighelstein, Marcelo

AU - Bogojeska, Jasmina

AU - Shtok, Joseph

AU - Arbelle, Assaf

AU - Staar, Peter W.J.

AU - Schumann, Anika

AU - Kuhn, Jonas

AU - Karlinsky, Leonid

PY - 2023

Y1 - 2023

N2 - In this paper, we present FlowchartQA, a new and unique large-scale benchmark for visual question answering (VQA) over flowcharts. FlowchartQA comprises close to 1M flowchart images and 6M question-answer pairs, covering various aspects of geometric and topological information contained in the charts. The questions have been carefully balanced to minimize biases. To accompany the proposed benchmark, we present a baseline model and perform comprehensive ablation studies and qualitative analyses to provide a solid foundation for future work. Our experimental results reveal interesting findings and demonstrate the potential of FlowchartQA as a testbed for flowchart understanding, which has been previously absent in the community.

AB - In this paper, we present FlowchartQA, a new and unique large-scale benchmark for visual question answering (VQA) over flowcharts. FlowchartQA comprises close to 1M flowchart images and 6M question-answer pairs, covering various aspects of geometric and topological information contained in the charts. The questions have been carefully balanced to minimize biases. To accompany the proposed benchmark, we present a baseline model and perform comprehensive ablation studies and qualitative analyses to provide a solid foundation for future work. Our experimental results reveal interesting findings and demonstrate the potential of FlowchartQA as a testbed for flowchart understanding, which has been previously absent in the community.

UR - http://www.scopus.com/inward/record.url?scp=85198217961&partnerID=8YFLogxK

M3 - Paper

AN - SCOPUS:85198217961

SP - 34

EP - 46

Y2 - 22 September 2023

ER -

FlowchartQA: The First Large-Scale Benchmark for Reasoning over Flowcharts

Abstract

Conference

Bibliographical note

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this