DMMV

Time series, typically represented as numerical sequences, can also be transformed into images and texts, offering multi-modal views (MMVs) of the same underlying signal. These MMVs can reveal complementary patterns and enable the use of powerful pre-trained large models, such as large vision models (LVMs), for long-term time series forecasting (LTSF).

However, as we identified in this work, the state-of-the-art LVM-based forecaster poses an inductive bias towards “forecasting periods”. To harness this bias, we propose DMMV, a novel decomposition-based multi-modal view framework that leverages trend-seasonal decomposition and a novel backcast-residual based adaptive decomposition to integrate MMVs for LTSF.

Comparative evaluations against 14 SOTA models across diverse datasets show that DMMV outperforms single-view and existing multi-modal baselines, achieving the best mean squared error (MSE) on 6 out of 8 benchmark datasets.

Leveraging MMVs offers two key advantages: augmenting time series with alternative views can reveal patterns not evident in the original numerical data, and pre-trained large models can extract complex patterns specific to certain views, such as visual representations. DMMV integrates numerical and visual views in a compact architecture. We exclude the textual view due to its marginal gains in Time-VLM and recent doubts about the effectiveness and cost-efficiency of LLMs for LTSF.

Masked autoencoder (MAE) is pre-trained self-supervisedly by reconstructing masked image patches using the ImageNet dataset. To adapt it to LTSF, VisionTS adopts a period-based imaging technique introduced by TimesNet: each length-T univariate series is segmented into ⌊T/P⌋ subsequences of length P, where P is set to be the period of the series, which can be obtained using the Fast Fourier Transform or from prior knowledge on sampling frequency. The subsequences are stacked to form a 2D image. After standard-deviation normalization, it is duplicated 3 times to form a gray image, followed by a bilinear interpolation to resize it to an image of size 224 × 224 × 3 to fit the input requirement of MAE. The forecast is achieved by reconstructing a right-appended masked area of that image, corresponding to the future horizon of the series.

Due to period-based imaging and the spatial consistency enforced during MAE's pixel inference, VisionTS exhibits a strong bias toward inter-period consistency, often overshadowing the global trend. The panels below illustrate VisionTS's forecasts on a synthetic sinusoidal time series with a period of 24. As the segment length P varies from 24 to 48, forecasts alternate between accurate and inaccurate as P shifts from 1×period to 2×period, highlighting a strong inductive bias toward periodicity. Notably, the forecasts aren't mere repetitions — the decreasing intra-period amplitude indicates that LVMs can still capture local trends within each period.

The function of the synthetic time series is x(t) = A(t) · sin(2πt / P), where the period P is set to 24 and the amplitude function A(t) decreases linearly over time. The forecasts are more accurate when the segment length is a multiple of the period (e.g., 24, 48) than other values. In the table below, the fluctuations in MSEs and MAEs across different segment lengths other than 24 and 48 support the findings of the inductive bias toward “forecasting periods”.

We adopt 8 widely used multivariate benchmarks: ETT — ETTh1, ETTh2, ETTm1, ETTm2 — together with Weather, Illness, Traffic and Electricity. Following standard protocols, we split the datasets chronologically into training/validation/test sets using a 60%/20%/20% ratio for ETT and 70%/10%/20% for the others. The prediction horizon is set to {24, 36, 48, 60} for Illness and {96, 192, 336, 720} for the remaining datasets. By default, the look-back window is 336. We use Mean Squared Error and Mean Absolute Error to evaluate the LTSF performance of the compared methods.

The table below summarizes the LTSF performance of 10 representative methods across four categories: MMV-based, visual-view-based, language-view-based, and numerical-view-based approaches. Time-VLM's results on the Illness dataset are not reported in its paper and its code is unavailable at the time of this experiment, thus are marked by “—”. For DMMV, the stronger variant, DMMV-A, is reported.

Several key insights emerge: MMV and visual-view methods generally outperform language-view methods, underscoring the effectiveness of LVMs, particularly when integrated within MMV frameworks; numerical-view models such as PatchTST and CycleNet remain competitive, especially on datasets where VisionTS underperforms — ETTm2 and Electricity — highlighting their potential to complement visual models; the strong results of CycleNet, a lightweight model with learnable decomposition, demonstrate the value of combining simplicity with structure in LTSF; notably, DMMV-A, which unifies visual and numerical views through a novel adaptive decomposition, outperforms the baselines in most cases, achieving 43 first-places and confirming its effectiveness; and lastly, while VisionTS performs well on highly periodic datasets such as ETTh1, ETTm1 and Traffic due to MAE's inductive bias toward periodicity, DMMV-A alleviates this bias, resulting in more generalizable forecasts.

We validate the design of DMMV-A through ablation studies on four datasets: (a) replaces the linear model in the numerical forecaster with a PatchTST-style Transformer; (b) swaps MAE with SimMIM as the visual forecaster; (c) replaces the gating fusion with a simple sum; (d) removes BCMask, performing backcasting and forecasting on the full, unmasked look-back window; (e) substitutes BCMask with random masking; (f) freezes the entire visual forecaster instead of fine-tuning norm layers; and (g) removes the backcast-residual mechanism, feeding both forecasters the same input and merging their outputs via gating.

In (a), replacing the linear numerical forecaster with a Transformer slightly degrades performance, likely due to the increased difficulty of jointly training a Transformer with LVMs. In (b), MAE outperforms SimMIM as the visual forecaster, likely due to its ViT-based reconstruction decoder being better suited for pixel-level tasks like LTSF than SimMIM's linear decoder, while both models share similar encoder architectures. In (c), gate-based fusion outperforms simple summation, highlighting its adaptability to the distinct outputs of the two forecasters. (d) and (e) underscore the importance of BCMask: removing it recovers the full look-back window as the backcasted seasonal component, diminishing the trend signal and weakening the numerical forecaster, while random masking performs slightly worse due to poorer periodic pattern extraction, which leads to many fluctuations. In (f), fine-tuning only the norm layers significantly improves performance over freezing, confirming the benefit of coordinated learning between forecasters. Finally, (g) shows that removing the backcast-residual mechanism causes a major performance drop, affirming its role in effective decomposition.

A key advantage of the gate-based fusion is its interpretability. The chart below presents average gate weights across datasets: DMMV-A consistently places more weight on the visual forecaster, while DMMV-S tends to balance both forecasters but leans toward the numerical one. In DMMV-A, these weights are learned based on forecasting performance, emphasizing the visual forecaster's importance. Notably, although the numerical forecaster receives less weight, it remains essential — as evidenced by DMMV-A outperforming the visual-only baseline VisionTS in the table above. In contrast, DMMV-S's weights are limited by its fixed moving-average decomposition, leading to a non-adaptive and suboptimal allocation of forecasting roles.

The example below provides decompositions by DMMV-S and DMMV-A. DMMV-A produces a smooth, clearly periodic component — consistent with expectations — and a trend component with some noises. In contrast, DMMV-S's moving-average yields a smoother trend by absorbing fluctuations, pushing noise into the seasonal component. This makes forecasting harder for the visual forecaster, which is more sensitive to fluctuations than the numerical one, resulting in its lower gate weights. Since periodic patterns are crucial for long-term forecasting, the clearer period separation in DMMV-A leads to forecasts that better match the ground truth.

The plots below compare DMMV-A with a visual forecaster, VisionTS, and two numerical forecasters, PatchTST and DLinear, which can serve as its single-view ablations. The Illness dataset is excluded due to its short time series, 966 time steps in total. Using the MAE metric, we observe that DMMV-A and VisionTS benefit from longer look-back windows, while PatchTST and DLinear degrade beyond a length of 336. Notably, DMMV-A outperforms VisionTS at length 1152, highlighting the advantage of explicitly modeling global trends.