More Works by the Authors

xLSTM-Mixer: Multivariate Time Series Forecasting by Mixing via Scalar Memories

The time series backbone behind the xQA action encoder, mixing predictions across scalar memory states.

NeurIPS 2025
Finding the Zeitgeist in Time Series Foundation Models

An analysis of where today's time series foundation models stand.

ICLR 2026 Workshop on Time Series in the Age of Large Models (TSALM)
Right on Time: Revising Time Series Models by Constraining Their Explanations

Steering time series models away from confounders by constraining what their explanations may attend to.

ECML PKDD 2025

QuAnTS: Question Answering on Time Series

Felix Divo1,*, Maurice Kraus1,*, Anh Q. Nguyen2, Hao Xue3, Imran Razzak4, Flora D. Salim2, Kristian Kersting1,5,6,7, Devendra Singh Dhami8
1AIML Lab, TU Darmstadt
2University of New South Wales
3Hong Kong University of Science and Technology (Guangzhou)
4MBZUAI, Abu Dhabi
5Hessian Center for AI (hessian.AI)
6German Research Center for AI (DFKI), Darmstadt
7Centre for Cognitive Science, TU Darmstadt
8TU Eindhoven
*Authors contributed equally
Published in the Journal of Data-centric Machine Learning Research (DMLR)
OpenReview arXiv Dataset Generation Code Evaluation Code
QuAnTS generation pipeline: an action sequence is sampled, question and answer types are drawn per sequence, each is instantiated from a template, and the human motion diffusion output is combined with the QA pair

QuAnTS is generated in several steps: An action sequence is sampled ①, where for each we sample five question and answer types ②. For diversity, each of them is then instantiated from a sampled template ③. The time series from the human motion diffusion ④ is then combined with the QA-pair and auxiliary data ⑤. Dice indicate randomized operations to enhance dataset diversity.

Abstract

Text offers intuitive access to information. This can, in particular, complement the density of numerical time series, thereby enabling improved interactions with time series models to enhance accessibility and decision-making. While the creation of question-answering datasets and models has recently seen remarkable growth, most research focuses on question answering (QA) on vision and text, with time series receiving minute attention. To bridge this gap, we propose a challenging novel time series QA (TSQA) dataset, QuAnTS, for Question Answering on Time Series data. Specifically, we pose a wide variety of questions and answers about human motion in the form of tracked skeleton trajectories. We verify that the large-scale QuAnTS dataset is well-formed and comprehensive through extensive experiments. Thoroughly evaluating existing and newly proposed baselines then lays the groundwork for a deeper exploration of TSQA using QuAnTS. Additionally, we provide human performances as a key reference for gauging the practical usability of such models. We hope to encourage future research on interacting with time series models through text, enabling better decision-making and more transparent systems.

Explore the Data

Every QuAnTS context is a 16-second human motion trajectory of four consecutive actions, paired with five question–answer pairs. Models see the raw 320 × 24 × 3 numbers; the videos below are renderings of exactly that signal, as shown to the participants of our human study. Pick a sample from the test split:

Sample #27000 · test split · 16 s · 320 × 24 × 3 values

Action sequence

  1. 0–4 s bowing
  2. 4–8 s waving
  3. 8–12 s picking something up with both hands
  4. 12–16 s picking something up with both hands
Ground-truth scene description (for debugging only)

The sequence starts with bowing from 0:0.0 to 0:4.0 (for 4.0 seconds). The person is waving. This action is starting at 0:4.0 and continues until 0:8.0, resulting in a total time of 4.0 seconds. Someone is picking something up with both hands. This is happening from 0:8.0 to 0:12.0, which means for a total time of 4.0 seconds. As a last action, the person is picking something up with both hands. This action is starting at 0:12.0 and happens until 0:16.0, meaning the engagement in this is for 4.0 seconds.

The five question–answer pairs

  • Open

    After engaging in bowing, what followed immediately?

    Someone is waving.

    right_after_open

  • Open

    At 2.803, what action is the person performing?

    The person is bowing.

    descriptive_identification_open

  • Multiple-choice

    What is the person doing at 4.354? A: waving, B: bowing, or C: picking something up with both hands?

    The correct answer is: waving.

    descriptive_identification_multi

  • Binary

    Is it accurate to state that the individual repeats waving 0 times?

    False

    count_binary

  • Open

    Following the performance of bowing by the person, what actions were done later?

    The subject is waving and picking something up with both hands.

    after_open

Sample #27001 · test split · 16 s · 320 × 24 × 3 values

Action sequence

  1. 0–4 s kicking a ball
  2. 4–8 s playing guitar
  3. 8–12 s eating with the right hand
  4. 12–16 s kicking a ball
Ground-truth scene description (for debugging only)

The person is starting off the sequence with kicking a ball. This action is starting at 0:0.0 and continues until 0:4.0, resulting in a total time of 4.0 seconds. The person is playing guitar from 0:4.0 to 0:8.0 (4.0 seconds total). The activity eating with the right hand can be observed. This action is starting at 0:8.0 and continues until 0:12.0, resulting in a total time of 4.0 seconds. The sequence finishes with kicking a ball from 0:12.0 to 0:16.0 (for 4.0 seconds).

The five question–answer pairs

  • Open

    What was the last action performed by the person?

    We see someone who is kicking a ball.

    last_open

  • Multiple-choice

    How often does the person engage in skipping rope? Do they do it 1 times (Option A), 2 times (Option B), or 0 times (Option C)?

    The correct answer is 0.

    count_multi

  • Multiple-choice

    Immediately prior to eating with the right hand, what action was performed by the person? Did they perform A: catching a ball, B: playing guitar, or C: throwing a ball?

    The correct answer is playing guitar.

    right_before_multi

  • Binary

    Do eating with the right hand and kicking a ball happen with the same frequency for the person?

    No

    comparison_counting_binary

  • Multiple-choice

    Among A: sitting down, B: playing guitar and C: kicking a ball, which action has an equal number of occurrences to eating with the right hand?

    The correct answer is: playing guitar.

    comparison_counting_multi

Sample #27002 · test split · 16 s · 320 × 24 × 3 values

Action sequence

  1. 0–4 s eating with the right hand
  2. 4–8 s bowing
  3. 8–12 s running
  4. 12–16 s bowing
Ground-truth scene description (for debugging only)

The person is starting off the sequence with eating with the right hand. This is happening from 0:0.0 to 0:4.0, which means for a total time of 4.0 seconds. The person is engaging in bowing from 0:4.0 to 0:8.0 (for 4.0 seconds). The person is running. It is starting at second 0:8.0 and continuing until 0:12.0, so the person is doing this for 4.0 seconds. Finally, the person is bowing. This action is starting at 0:12.0 and happens until 0:16.0, meaning the engagement in this is for 4.0 seconds.

The five question–answer pairs

  • Open

    At 11.943, what action is the person performing?

    Someone is running.

    descriptive_identification_open

  • Binary

    Does the person take part in running 2 times?

    Not true

    count_binary

  • Binary

    Was bowing the final activity done by the person?

    Yes

    last_binary

  • Binary

    Can we confirm that the individual participated in exactly 3 activities throughout the entire sequence?

    Yes

    interval_whole_sequence_binary

  • Binary

    Is there a dissimilarity between the actions at the beginning and the end?

    This is correct!

    comparison_first_last_different_binary

Sample #27003 · test split · 16 s · 320 × 24 × 3 values

Action sequence

  1. 0–4 s picking something up with both hands
  2. 4–8 s holding a baby
  3. 8–12 s sitting down
  4. 12–16 s sitting down
Ground-truth scene description (for debugging only)

The person is starting the sequence with picking something up with both hands at timestamp 0:0.0 and ending at 0:4.0, resulting in a total time of 4.0 seconds. The activity is holding a baby from 0:4.0 to 0:8.0 (for 4.0 seconds). The subject is sitting down from 0:8.0 to 0:12.0 (4.0 seconds total). The sequence finishes with sitting down. It is starting at second 0:12.0 and continuing until 0:16.0, so the person is doing this for 4.0 seconds.

The five question–answer pairs

  • Binary

    Was jumping once the most commonly undertaken action by the person throughout the sequence?

    False

    extremum_most_binary

  • Open

    After the person performed holding a baby, what ensued directly?

    We see someone who is sitting down.

    right_after_open

  • Multiple-choice

    Some time before a person was sitting down, was it A: eating with the right hand, B: picking something up with both hands, or C: running?

    The correct answer is: picking something up with both hands.

    before_multi

  • Binary

    Are the first and last actions alike?

    False

    comparison_first_last_same_binary

  • Binary

    Was holding a baby observed as a subsequent action for the person some time after picking something up with both hands?

    Yes

    after_binary

Sample #27004 · test split · 16 s · 320 × 24 × 3 values

Action sequence

  1. 0–4 s bowing
  2. 4–8 s running
  3. 8–12 s sitting down
  4. 12–16 s holding a baby
Ground-truth scene description (for debugging only)

The activity bowing is the first action done by the person from 0:0.0 to 0:4.0 (4.0 seconds total). The person is engaging in running. This action is starting at 0:4.0 and happens until 0:8.0, meaning the engagement in this is for 4.0 seconds. The subject is sitting down. It is starting at second 0:8.0 and continuing until 0:12.0, so the person is doing this for 4.0 seconds. The last movement the person is doing is holding a baby from 0:12.0 to 0:16.0 (4.0 seconds total).

The five question–answer pairs

  • Multiple-choice

    Throughout the duration from 5.579 to 10.231, how many distinct behaviors is the person engaged in? A: 3, B: 2, C: 4

    The correct answer is 2.

    interval_part_sequence_multi

  • Binary

    Can we state that the initial and final actions are an exact match?

    No

    comparison_first_last_same_binary

  • Multiple-choice

    What action can be observed right before the person was holding a baby? Did they perform A: T-posing, B: punching, or C: sitting down?

    The correct answer is sitting down.

    right_before_multi

  • Binary

    After being involved in running, did the person transition to sitting down at some later point?

    This is true!

    after_binary

  • Binary

    Did the person end with bowing as the final action?

    This is not true.

    last_binary

Sample #27005 · test split · 16 s · 320 × 24 × 3 values

Action sequence

  1. 0–4 s skipping rope
  2. 4–8 s skipping rope
  3. 8–12 s waving
  4. 12–16 s holding a baby
Ground-truth scene description (for debugging only)

The person is starting off the sequence with skipping rope from 0:0.0 to 0:4.0 (for 4.0 seconds). The person is performing skipping rope from 0:4.0 to 0:8.0 (4.0 seconds total). The subject is waving. It is starting at second 0:8.0 and continuing until 0:12.0, so the person is doing this for 4.0 seconds. As the last action, the person is performing holding a baby at timestamp 0:12.0 and ending at 0:16.0, resulting in a total time of 4.0 seconds.

The five question–answer pairs

  • Multiple-choice

    What did the person do immediately after waving? Was it A: holding a baby, B: throwing a ball, or C: running?

    The correct answer is: holding a baby.

    right_after_multi

  • Multiple-choice

    How often does the person engage in throwing a ball? Do they do it 0 times (Option A), 1 times (Option B), or 2 times (Option C)?

    The correct answer is 0.

    count_multi

  • Binary

    Was skipping rope the action that happened most often for the person in the given sequence?

    This is correct!

    extremum_most_binary

  • Binary

    Was holding a baby the final action undertaken by the person?

    True

    last_binary

  • Open

    How many different actions does the person perform throughout the complete sequence?

    3

    interval_whole_sequence_open

The Dataset at a Glance

150,000 question–answer pairs
30,000 motion trajectories (~133.3 h)
45 balanced question types
3 answer types: binary, multiple-choice, open
19 distinct actions, four per sequence
320 × 24 × 3 time steps × joints × coordinates

The predefined splits hold 120,000 (80 %) training, 15,000 (10 %) validation, and 15,000 (10 %) test samples. All five QA pairs of a context trajectory land in the same split, so no motion leaks across them. By answer type, QuAnTS holds 61,804 binary, 43,599 multiple-choice, and 44,597 open-ended pairs, of which 125,726 QA pairs and 100,568 questions are unique. Beyond the question, answer, and trajectory, each sample carries its question and answer type, the ground-truth action sequence, and a textual scene description for debugging and diagnostics.

Hierarchy of question types (descriptive, temporal, comparison, reasoning, overarching) and answer types (binary, multiple-choice, open); those covered by QuAnTS are highlighted in blue

The hierarchy of question and answer types we identified. The blue types are the most fundamental to time series QA and are therefore included in QuAnTS. The gray ones are left for future extensions — the blue ones alone already make for a very challenging dataset.

Bar chart of the number of samples per question type, grouped by binary, multiple-choice, and open answer types, mostly similar in size, with a few rarer categories

A balanced mix, by construction

Question types are sampled explicitly rather than harvested, so all 45 of them occur at comparable frequency. Only a few categories, such as comparison_counting_open, are rarer because they can only be posed when the action sequence permits a non-trivial answer.

Diversity is enforced on every axis: about 10–20 hand-reviewed templates per QA type, LLM-assisted paraphrasing, randomized answer positions, and organic variation from the motion diffusion. Not a single trajectory is duplicated, and under a soft equality on the signal level the effective sample size stays above 99 %.

Nineteen clearly discernible actions

Rendered figure waving
waving
Rendered figure dancing
dancing
Rendered figure golfing
golfing
Rendered figure kicking a ball
kicking a ball
Rendered figure catching a ball
catching a ball
Rendered figure sitting down
sitting down

playing guitar · punching · running · jumping once · drinking with the left hand · bowing · throwing a ball · shaking hands · skipping rope · picking something up with both hands · dancing · eating with the right hand · waving · catching a ball · kicking a ball · sitting down · golfing (swinging a club) · T-posing · holding a baby

How QuAnTS Is Built

Questions are generated, not annotated. We first sample a sequence of four actions, then draw five question and answer types for it, and finally instantiate each from one of many manually reviewed templates. Defining the question types up front keeps the distribution balanced and makes it possible to inspect a model's performance per subtask.

The matching time series comes from a human motion diffusion model, Spatio-Temporal Motion Collage (STMC), conditioned on the very action sequence the questions are about. Four prompts of four seconds each are blended into one fluent 16-second motion at 20 fps, yielding three spatial coordinates for each of the 24 SMPL joints — a 320 × 24 × 3 multivariate time series. Generating rather than reusing recordings lets us ask precisely the questions we want, and gives multiple organic variations of the same action.

For Q2, Q3, and Naive, binary and multiple-choice answers are scored with accuracy and macro-averaged precision, recall, and F1; unparseable responses count as wrong. Open answers use stemmed ROUGE-L F1 and METEOR plus an LLM judge on a 1–3 scale normalized to [0, 1]. We validated the judge against human ratings on 200 samples and settled on Qwen3 8B, which reaches a Pearson correlation of 0.912 and deviates by 7.45 % on average.

Example chat: a user asks what the person is doing first, the assistant answers that they are waving, and the conversation continues about how often they jump

Where this is going. Text offers a highly intuitive medium of communication to interact with otherwise opaque multivariate time series.

Key Results

Given ground-truth scene descriptions, a finetuned Llama 3.1 8B answers QuAnTS almost perfectly (99.12 % F1 binary, 99.98 % multi, 0.9434 LLM judge) — the dataset is logically consistent and answerable. Input ablations show answer-format-dependent performance: question-only inputs (Q2) support above-chance binary accuracy, while time-series-only inputs (Q3) remain near chance on closed answers. Naive, which receives both the serialized trajectory and the question, performs weakly on closed answers but strongly on open answers.

Q2, Q3, and Naive each use Llama 3.1 8B with an independent QLoRA adapter per answer format, trained for five epochs. Numerical inputs retain the full 320 × 24 × 3 trajectory as compact nested JSON after scaling to [−999, +999] and rounding to integers. Finetuning and evaluating the nine adapters required approximately 5,492 NVIDIA H100 GPU-hours.

Binary and multiple-choice test results, in percentages. For Q2, Q3, and Naive, precision, recall, and F1 are macro-averaged.
Answer format System Accuracy ↑ Precision ↑ Recall ↑ F1 ↑
Binary Ablation: Only Question (Q2)64.53 %65.32 %64.70 %64.22 %
Ablation: Only Time Series (Q3)49.21 %50.15 %50.00 %33.62 %
Humans (n = 840)77.50 %81.08 %74.66 %77.74 %
Naive: TS + Question62.94 %64.54 %63.19 %62.16 %
ChatTS51.69 %55.38 %25.36 %34.79 %
xQA-Llama on GT79.44 %84.91 %72.42 %78.17 %
xQA-Qwen on GT80.92 %92.28 %68.14 %78.39 %
xQA-Llama on AE79.46 %84.60 %72.84 %78.28 %
xQA-Qwen on AE81.00 %92.52 %68.10 %78.46 %
Multi Ablation: Only Question (Q2)33.32 %30.38 %33.00 %26.25 %
Ablation: Only Time Series (Q3)33.29 %21.94 %33.08 %20.39 %
Humans (n = 820)86.59 %86.55 %86.61 %86.54 %
Naive: TS + Question34.01 %22.74 %33.68 %26.62 %
ChatTS30.40 %30.13 %30.58 %29.12 %
xQA-Llama on GT81.50 %81.64 %81.51 %81.50 %
xQA-Qwen on GT88.01 %88.04 %88.03 %88.01 %
xQA-Llama on AE81.18 %81.30 %81.19 %81.18 %
xQA-Qwen on AE87.97 %87.99 %87.98 %87.97 %
Open-ended test results on a 0–1 scale, not percentages. For Q2, Q3, and Naive, ROUGE denotes stemmed ROUGE-L F1; LLMJudge is normalized to [0, 1].
System ROUGE ↑ METEOR ↑ LLMJudge ↑
Ablation: Only Question (Q2)0.42330.28380.2092
Ablation: Only Time Series (Q3)0.09560.04910.1082
Humans (n = 440)0.09530.26350.6148
Naive: TS + Question0.81990.69470.8759
ChatTS0.04550.10600.0702
xQA-Llama on GT0.09130.36770.7729
xQA-Qwen on GT0.17890.36970.8279
xQA-Llama on AE0.08980.36790.7753
xQA-Qwen on AE0.17820.36920.8176

Human, ChatTS, and xQA rows provide reference scores under their respective evaluation setups; direct rankings require compatible scoring and evaluation conditions.

What this means. Question-only binary performance reveals predictive information in the textual inputs, so the ablations do not rule out shortcuts. Naive's strong open-answer scores show that a finetuned LLM on serialized trajectories can answer this format well, but do not establish general motion understanding or transfer. xQA separates perception from reasoning: an xLSTM-Mixer action encoder (AE) labels each segment, and an instruction-following LLM reasons over the resulting action sequence. Its AE and ground-truth-label (GT) variants achieve similar scores, supporting the feasibility of learning action recognition under the available supervision.

General-text retention after finetuning (perplexity)

We evaluate the base model and all nine final adapters on the same WikiText-2 raw test stream, without a task prompt or chat template: 289,075 scored tokens, a 2,048-token context, and a 1,024-token stride. BF16 is the primary comparison; NF4 uses BF16 computation. Perplexity (PPL) is the exponential of token-weighted mean negative log-likelihood. Lower is better, and PPL/base compares each adapter with the base at the same precision.

WikiText-2 test perplexity. PPL/base is a multiplicative ratio, not a percentage.
System Answer format BF16 PPL ↓ BF16 PPL/base NF4 PPL ↓ NF4 PPL/base
Base—5.611.005.971.00
Only Question (Q2)Binary3.17 × 1085.65 × 1073.71 × 1086.21 × 107
Only Question (Q2)Multi9870.981758.136.46 × 1041.08 × 104
Only Question (Q2)Open19.243.4325.654.30
Only Time Series (Q3)Binary2.10 × 1073.74 × 1061.74 × 1072.92 × 106
Only Time Series (Q3)Multi6.17 × 1081.10 × 1087.32 × 1081.23 × 108
Only Time Series (Q3)Open10.391.8512.602.11
Naive: TS + QuestionBinary1.14 × 1082.03 × 1071.48 × 1082.48 × 107
Naive: TS + QuestionMulti1.35 × 1072.40 × 1062.90 × 1074.86 × 106
Naive: TS + QuestionOpen10.611.8912.992.18

All active adapters increase general-text perplexity relative to their precision-matched base. Naive's strong open-answer performance coexists with a 1.89-fold BF16 perplexity increase; the closed-answer adapters show much larger increases. These are diagnostics of poorer next-token prediction on one general-text corpus, not QuAnTS answer accuracy or a comprehensive assessment of reasoning and conversational ability. The base weights remain frozen.

Get Started

Load the published dataset straight from the Hugging Face Hub:

from datasets import load_dataset

quants = load_dataset("dasyd/quants", "binary")  # or "multi", "open"
print(quants["test"][0]["question"])
# Is it accurate to state that the individual repeats waving 0 times?

Or generate your own variant — the pipeline is configurable in the number of samples, the question type distribution, and the answer formats: 

git clone https://github.com/mauricekraus/quants-generate.git
cd quants-generate
pip install -e quants[dev]
cd quants
python -m generate prompts --num-samples 1000

BibTeX

@article{divo2026quants,
  title={QuAnTS: Question Answering on Time Series},
  author={Divo, Felix and Kraus, Maurice and Nguyen, Anh Q. and Xue, Hao and Razzak, Imran and Salim, Flora D. and Kersting, Kristian and Dhami, Devendra Singh},
  journal={Journal of Data-centric Machine Learning Research},
  year={2026},
  url={https://openreview.net/forum?id=bNCDElSOXB}
}