TSFLab: A Self-Improving Agent-Agnostic Workbench for Time Series Forecasting Research

Overview

A workbench that lives outside the agent

As publications and products on time series forecasting (TSF) proliferate, researchers face a growing burden of analyzing existing work. First, not all publications release code: only about 20% of arXiv TSF papers have a code link in the surveyed period. Second, released code also requires manual maintenance. Thus, TSF researchers must constantly adapt or implement existing works. However, independent implementations can differ in experimental settings and are difficult to maintain.

We therefore propose to use AI agents for benchmarking TSF research and even supporting autoresearch, e.g., developing new TSF models. Yet directly applying existing coding agents, such as Claude Code and Codex, is suboptimal: prohibitive costs, limited scalability to large research projects (e.g., benchmarking), and lack of self-maintenance. Unlike a harness, which is part of each agent, a “workbench” lies outside agents: it is reusable and shareable across agents.

To address these challenges, we propose TSFLab, a self-improving agent-agnostic “workbench” that equips compatible coding agents with shared resources and tools for low-cost TSF research and continuous maintenance of research resources. Our experiments cover reproduction, benchmarking, and new method development. In reproduction and benchmarking, TSFLab improves task completion by 20–163% and cuts fresh tokens per completed task by 11–66% across baselines. A pilot study of agent-driven method development also shows improved forecasting accuracy via TSFLab, with 32.8–73.6% fewer fresh tokens and 9.2–68.8% less development time than the other settings.

Method

Four modules, for any compatible agent

The workbench provides reusable domain knowledge, research procedures, and experimental tools without changing the agent's model or harness. Our goal is to improve valid task completion (i.e., completing a research task exactly as required) and reduce cost per completed task. TSFLab supports research through four modules.

TSF-Domain Guide enables TSF-awareness through common research workflows, experimental settings, evaluation protocols, and known pitfalls. Entry instructions introduce conventions and module interactions, while 17 task-specific skills provide steps, checks, and pitfalls as needed. When benchmarking methods on a new dataset, for example, a skill guides the agent to identify time fields, forecasting variables, sampling frequency, and evaluation settings, then register data through the workbench's interface and inspect partitioning and scaling.

Knowledge Structurer decomposes existing TSF models into reusable units and organizes these units, methods, and diverse benchmark datasets into a unified structure. This structure is realized by a “knowledge card” for each method, unit, and dataset, read level by level (summary, card, source) as a decision requires, so that AI agents can reason efficiently without token-costly reading of code and data details. It also extracts common blocks, such as RevIN, series decomposition, and attention variants, into shared units that methods import; 263 methods reuse at least one unit.

Experiment Workflower executes agent-declared experiments through a unified runner that validates and runs experiment configurations and reports results under consistent protocols. Agents use Compose to combine presets, Override to change settings, and Sweep to expand methods or values. The runner expands configurations into a run matrix, then checks configuration validity, data, model compatibility, environments, and budgets, returning problems before training. Shared routines split data as configured, fit scaling on training data only, select checkpoints on validation data, and compute metrics consistently.

Recursive Improver routinely checks and verifies up-to-date TSF research outcomes and integrates verified artifacts (e.g., models, datasets) into the workbench for self-maintenance. It brings external resources and agent-produced outcomes back into the shared workbench through checks for usability, complexity, and consistency: continuous integration checks basic operations and code complexity, and an independent read-only review agent checks whether implementations preserve the defining mechanisms in papers and pinned official code. Failed checks prompt revision and rechecking before integration. The workbench is updated independently of agent releases.

Level	When to consult	Information provided
Summary	Screening candidates	Category and use cases
Card	Resource selection, configuration, unit reuse	Capabilities and limits, parameters and interfaces, verification records
Source	Checking details, resolving doubts, modifying methods	Original papers, reference code, detailed evidence

Three levels of research knowledge. This progressive disclosure lets agents retrieve the detail needed for each decision without rereading sources.

An agent works with the four TSFLab modules: TSF-Domain Guide, Knowledge Structurer, Experiment Workflower and Recursive Improver. — Overview of how an agent works with the TSFLab workbench.

Results

More tasks completed, at lower cost

We compare TSFLab with two TSF codebases and with no workbench: Time-Series-Library (TSLib), a model library, and TFB, a benchmarking framework. Without a workbench, the agent starts from an empty directory with PyTorch. All main settings use Codex CLI with GPT-6 Luna and identical prompts, data, network access, and budgets. The tasks are (i) paper reproduction, where the agent reproduces the reported results of a recent forecasting paper that has no official code and is absent from all compared codebases; (ii) benchmarking on new data, where, given two new datasets, ERCOT and Openmeteo, the agent benchmarks eight methods under a common protocol; and (iii) a method development pilot, where on ETTh1 and Weather the agent searches for a forecasting method using validation data, then evaluates its final selection once on test data.

TSFLab completes the most tasks with the most accurate results. It completes all 12 reproduction attempts and 42 of 48 benchmark results, the most of any setting. Its reproductions are also the closest to the papers: TSFLab reproduces 73% of the 96 scheduled results within 10% of the paper's MSE, against 21–46% for the other settings. TSFLab also costs less: each completed outcome takes fewer tokens and less time in reproduction, and fewer tokens and tool calls in benchmarking. Its added code is simpler. Benchmarking needs no code changes, since all eight methods are already integrated and Experiment Workflower runs them from configuration; in reproduction, agents add a similar number of lines but 15–29% fewer functions and 18–28% less added cyclomatic complexity than the other settings.

An existing codebase alone is not enough. TFB completes fewer outcomes than no workbench in both tasks, mostly because agents replace requested methods with substitutes. A framework helps an agent only when the agent can extend it faithfully. The method-development pilot shows similar benefits: on ETTh1 and Weather, TSFLab reaches the lowest test MSE with less time and fewer fresh tokens than every other setting.

Main results using Codex CLI with GPT-6 Luna. Costs are normalized by completed outcomes. Time, Tokens (fresh tokens, excluding cache reads), and Calls (agent tool calls) sum all attempts, including failures, and divide by the completed count; ΔLOC, ΔFuncs, and ΔCC are the net code lines, added functions, and added cyclomatic complexity per session.

Reproduction MSE error per setting, and development test MSE against development time on ETTh1 and Weather. — Research-task quality. (a) Reproduction MSE error (log scale; lower is better); labels show shares within 10% (96 scheduled). (b,c) Development test MSE versus time; bubble areas and labels show fresh tokens.

Ablation

Every module carries its weight

We remove one module at a time on four reproduction tasks with three attempts each: the TSF-Domain Guide (entry instructions and skills), the Knowledge Structurer (cards and sources), or the Experiment Workflower (runner and experiment commands); and we replace the agent with Claude Code with Sonnet 5.5 and Codex CLI with GPT-6 Sol. Removing any module lowers completion, and each fails differently. Without the guide, completion drops the most: agents write their own data loaders, and every rejected attempt uses wrong ETTh1 split boundaries. Without the Knowledge Structurer, cost stays close to the full workbench, but agents reuse fewer units and add more code and complexity. Without the Experiment Workflower, agents write their own training and evaluation code; they add the most code, and their errors move into evaluation, such as tuning on test scores and averaging batch errors.

Stronger agent configurations reduce interaction, but not monetary cost. Compared with GPT-6 Luna, GPT-6 Sol and Sonnet 5.5 reduce tool calls by 48.5% and 51.1%, and fresh tokens by 9.2% and 55.1%, respectively; under a common fresh-token mix, list prices imply estimated costs of 18.2× and 9.0× Luna's. The workbench lets a lightweight agent match the stronger configurations in completion: Luna and Sol complete all 12 attempts; Sonnet completes 11.

Module and agent ablations. The full workbench uses Codex CLI with GPT-6 Luna. Recursive Improver is examined through the maintenance and refactoring outcomes below.

Maintenance

A catalog that maintains itself

Coverage Using TSFLab, we integrated 323 TSF models and compiled 93 datasets, along with 71 model units that can be reused for autoresearch. TSFLab contains 44 of the 50 most cited methods, compared with 29 across the three other codebases combined. The catalog combines established baselines with recent methods that have official code, reducing the integration needed to begin a comparison.

Compact implementations Broader coverage accompanies compact model implementations. The median implementation has 144 lines, close to TSLib's 145, while the largest has 505 lines, compared with 2,071 in TFB. Shared units consolidate common computations, and method specifications handle construction and parameter requirements outside model logic.

Maintenance through reuse As the catalog grows, extracting shared units helps control code per method: one extraction reduced the average function count from 11.9 to 8.6. A separate refactor reduced maximum function complexity from 20 to 9 while preserving bit-identical parameters, outputs, and gradients on reference cases for all 323 methods. Research sessions also expose reusable fixes. Missing guidance, manual configuration, repeated log polling, and premature test access led to instruction and runner updates; all 17 initial usability failures were repaired before integration.

Integration beyond model code Model implementation accounts for only 9% of method-intake tokens. Source reading, catalog search, and verification account for 28%, 27%, and 28%, respectively. Efficient integration therefore requires reusing knowledge and verification evidence as well as code. Knowledge cards retain these findings, helping later agents locate suitable methods and units without repeating the same investigation.

Top-cited method coverage, median and maximum lines per method, functions per method over development time, and tokens by stage when adding a method. — Workbench coverage and evolution. (a) Top-cited coverage. (b) Median/max model lines. (c) Functions per method, initial (0%) to latest (100%). (d) Intake tokens including cache reads.

Entry instructions and skills, reproduction task details, metric definitions, and API prices are in the paper's appendices.

Citation

Cite this work

BibTeX

@misc{shen2026tsflab,
      title={TSFLab: A Self-Improving Agent-Agnostic Workbench for Time Series Forecasting Research},
      author={ChengAo Shen and Yanjun Zhao and Haoxin Liu and Bonnie Chen and Miles Ma and Dongjin Song and Hanghang Tong and Qingsong Wen and Jingrui He and Jingchao Ni},
      year={2026},
      eprint={2610.40901},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2610.40901},
}