From Images to Signals

Input alignment Line Plot draws a 2D image with the x-axis representing time steps and the y-axis representing time-wise values. UVH (Univariate Heatmap) divides a univariate series into segments of length L – where L is a period obtained using the Fast Fourier Transform – and stacks them into a 2D image. MVH (Multivariate Heatmap) visualizes the matrix of a multivariate series, with the x-axis representing time steps and the y-axis representing variates. STFT, Wavelet and Filterbank transform the series into a spectrogram; GAF and RP produce square matrices with both axes representing time. The imaged series is resized to the LVM’s pretraining input size via bilinear interpolation, duplicated three times to form a grayscale image, and standardized in the same manner as the LVMs standardize each pretraining image.

Task-specific augmentation For the TSC task, we probe each LVM’s encoder: the classification layers of ViT and Swin, and the reconstruction decoders of MAE and SimMIM, are replaced with a linear classification layer. For the TSF task, we employ two frameworks from the literature. One trains a linear forecaster on the concatenated CLS and patch embeddings and applies to ViT and Swin; the other uses LVM reconstruction decoders for forecasting and applies to MAE and SimMIM – it applies a mask after the look-back window portion of the image, then reconstructs the masked patches and recovers forecasts from them. This requires input images to preserve time series values at the pixel level, and among the eight imaging methods only MVH and UVH do.

Ablation models To assess whether the LVM architecture is overly complex, we introduce two ablation models. Both retain the projection layer in the LVM encoder but replace the Transformer with simpler layers: the first replaces it with a linear layer, referred to as w/o-LVM; the second uses a single randomly initialized multi-head attention layer, referred to as LVM2Attn.

What type of LVM best fits the TSC (TSF) task? We observe that (1) supervised (semantic-level) LVMs and self-supervised (numerical-level) LVMs achieve comparable classification accuracies, while (2) self-supervised LVMs substantially outperform supervised LVMs in forecasting. Observation (2) can be attributed to the continuous nature of both pixels and time series, which enables self-supervised LVMs to transfer their capability of reconstructing masked pixels to predicting masked time series. SimMIM and Swin underperform, likely because they employ a window-based local attention mechanism that implicitly assumes translation invariance – an assumption that does not hold for imaged time series, where different locations correspond to different time steps or frequencies.

Which imaging method best fits the TSC (TSF) task? GAF performs best for classification, followed by MVH and RP, indicating their ability to encode distinguishable temporal patterns. In contrast, Line Plot substantially underperforms and therefore may be less suitable for this task. For forecasting, when combined with the reconstruction framework, UVH achieves the best performance, followed by MVH. This suggests their suitability for numerical-level tasks by leveraging the knowledge acquired by LVMs during masked-pixel reconstruction in large-scale image-based pretraining.

Can LVMs achieve SOTA performance? Here ViT and MAE are selected as the best-performing supervised and self-supervised LVMs, each configured with its best-performing imaging method – GAF for TSC and UVH for TSF. In classification, both ViT and MAE outperform the baselines, suggesting both supervised and self-supervised LVMs are effective for the high-level (semantic/pattern level) TSC task. In forecasting, across the eight datasets and two metrics, MAE outperforms non-LVM baselines in 9/16 cases, while ViT does not show clear superiority over non-LVM baselines, which may be due to its classification-based pretraining. The results suggest distinct behaviors of LVMs in TSF, indicating that greater challenges may arise in low-level (numerical level) tasks.

Are the pretrained parameters in LVMs useful? For TSC, the “freezing all parameters” setting outperforms “training from scratch” in all cases, suggesting that LVMs indeed transfer useful knowledge, and fine-tuning only the MLP and normalization layers is the minimal effort required to achieve performance comparable to full fine-tuning. For TSF, surprisingly, neither the “freezing all parameters” setting nor the “fine-tuning all parameters” setting consistently outperforms “training from scratch”. In contrast, fine-tuning only the normalization layers is generally better than other training and fine-tuning strategies. This may be attributed to the low-level nature of forecasting: since the model must predict numerical values directly, performance is strongly influenced by normalization, whereas fine-tuning more parameters than necessary may lead to overfitting.

How useful are LVM architectures? Training LVMs from scratch does not consistently outperform the two simpler models, suggesting that the architecture of LVMs alone may be overly complex for the available training data; however, since training from scratch is generally no worse than the simpler models, the overfitting issue does not appear to be severe. Moreover, the “freezing all parameters” setting and all fine-tuning settings outperform the simpler models in TSC, while fine-tuning the normalization layers consistently outperforms them in TSF. These results indicate that LVM architectures are not overly complex when serving as containers of transferable knowledge acquired through pretraining.

Can LVMs capture temporal dependencies? We perturb the temporal order in four ways: Sf-All randomly shuffles all time points; Sf-Half randomly shuffles the first half; Ex-Half swaps the first and second halves; and Masking randomly masks 50% of the time points. LVMs consistently exhibit performance drops under temporal perturbations, and they are more vulnerable to these perturbations than the ablation models. This suggests that LVMs likely make effective use of temporal patterns in time series during inference.

What are the computational costs of LVMs? LVMs have a larger number of parameters than the baselines. On average, LVMs require 3x (16x) the training time of the best TSC (TSF) baseline, primarily due to the larger number of trainable parameters. For inference, LVMs are 4x faster than the best TSC baseline but 20x slower than the best TSF baseline, due to both the increased model size and the additional cost of transforming time series into images. Compared with the best baselines, LVMs trade a relatively high computational overhead during inference for improved performance on many datasets.

Can LVMs make effective use of look-back windows? Intuitively, longer look-back windows can facilitate forecasting. MAE’s performance improves as the look-back window increases up to certain steps (e.g., 500–1500), after which it plateaus or declines. This behavior may stem from (1) the informativeness of a long look-back window, and (2) the image transformation process: since pretrained LVMs operate on fixed-size images with a limited number of pixels, they may be unable to capture all relevant information from excessively long time series, and image pixels are not fully utilized by the UVH imaging method to encode time-step information. Fortunately, contemporary LVMs can effectively handle moderately long look-back windows, and a length of about a thousand time steps is sufficient for many applications.