🤗Migrating to transformers 5: SigLIP2 Moved Over, clip-japanese-base Was Deferred

This article was co-authored with generative AI. Facts have been checked against public documentation where feasible, but errors may remain. Please verify primary sources before relying on this for important decisions.

Conclusion (up front)

On the image-search backend of a digital archive, two Dependabot PRs arrived to bump Hugging Face transformers from 4.57.6 to 5.5.0 (one per service). They were the same major bump, but the outcome diverged:

• The service using SigLIP2 (a model implemented natively inside transformers) could be moved to 5.5.0. It did require one fix: in 5.x the return value of get_text_features / get_image_features changed from a tensor to an output object, so the code has to pull out .pooler_output. Because that attribute is the same tensor 4.x returned, compatibility with the vectors already stored in Elasticsearch is preserved.
• The service using clip-japanese-base (a custom model loaded with trust_remote_code=True) does not work on 5.5.0 due to a two-stage incompatibility, so it stays on 4.57.6. Both stages originate in the model repository's own code and could not be worked around from the consumer side.

I'm recording the checks and the reasoning as a case where "whether the model is implemented inside transformers itself or via trust_remote_code" decided whether the upgrade was viable.

The systems in scope

For similarity search over source images and text-to-image search, we run two CLIP (Contrastive Language-Image Pre-training) family image-embedding services as CPU-inference Docker containers.

Service	Model	How it loads	Embedding dim
clip-siglip2	google/siglip2-base-patch16-256	transformers-native implementation	768
clip-ja	line-corporation/clip-japanese-base	trust_remote_code=True (custom class CLYPModel)	512

Both store the resulting embedding vectors in an Elasticsearch dense_vector and search by cosine similarity. In this setup, "the same input must produce the same vector after a library upgrade" matters: if the embedding computation changes, the new vectors can no longer be compared against everything already indexed, forcing a re-index.

Checking the dependencies

First I checked whether the post-upgrade dependencies resolve, using uv pip compile. The relevant constraints from transformers 5.5.0's metadata (PyPI) were:

Requires-Python: >=3.10.0
huggingface-hub  >=1.5.0, <2.0
numpy            >=1.17
tokenizers       >=0.22.0, <=0.23.0
safetensors      >=0.4.3

Of these, the only requirements change actually needed was huggingface_hub. It had been pinned at >=0.30,<1.0, which is incompatible with 5.5.0's >=1.5, so this too becomes a major bump (the resolution at the time landed on 1.23.0). numpy, on the other hand, is >=1.17, so the existing numpy==1.26.4 pin could stay as-is.

-transformers==4.57.6
-huggingface_hub>=0.30,<1.0
+transformers==5.5.0
+huggingface_hub>=1.5,<2.0

Dependency resolution itself passed cleanly for both services. The problem lies beyond that, at runtime.

SigLIP2: migrated after absorbing a return-value change in one place

I built the clip-siglip2 container on 5.5.0 and tried loading the model and running inference. The very first text encode raised this:

AttributeError: 'BaseModelOutputWithPooling' object has no attribute 'shape'

In transformers 4.x, SiglipModel.get_text_features() returned the pooled tensor ([batch, 768]) directly, but in 5.x it now returns an output object called BaseModelOutputWithPooling. Inspecting it, the fields were these two:

type: BaseModelOutputWithPooling
  last_hidden_state  [1, 64, 768]
  pooler_output      [1, 768]

The key point is that .pooler_output is the same tensor 4.x returned. The 4.x implementation internally pulled out pooler_output and returned it, so extracting it yourself on 5.x yields the same value and keeps compatibility with the existing embedding vectors. If you mistakenly pool last_hidden_state yourself, you get a value that cannot be compared against the already-indexed vectors.

The fix was just adding one helper function. Accepting both the 4.x line (which returns a tensor directly) and the 5.x line (which returns an object) makes it robust to moving between library versions.

def _pooled(out: object) -> torch.Tensor:
    """4.x returns a tensor directly; 5.x returns a BaseModelOutputWithPooling.
    .pooler_output is the same tensor as the 4.x return value."""
    if isinstance(out, torch.Tensor):
        return out
    pooled = getattr(out, "pooler_output", None)
    if isinstance(pooled, torch.Tensor):
        return pooled
    raise RuntimeError(f"unexpected SigLIP feature output type: {type(out)!r}")

After this fix, I confirmed inside the Docker container that everything from model loading to both text and image inference works (transformers 5.5.0 / huggingface_hub 1.23.0, output 768 dimensions, L2 norm 1.0). Because SigLIP2 is a model implemented inside transformers itself, even a major bump only takes this much follow-up.

clip-japanese-base: deferred due to a two-stage incompatibility

The trouble is the other one, clip-ja. line-corporation/clip-japanese-base is a custom-architecture model that loads by executing the implementation bundled in the model repository (CLYPModel in modeling_clyp.py) via trust_remote_code=True. That implementation is written against the transformers 4.x API.

Stage 1: it can't survive initialization on the meta device

Loading it on 5.5.0 stops during model construction with this error:

File ".../timm/models/eva.py", line 475, in __init__
    dpr = [x.item() for x in torch.linspace(0, drop_path_rate, depth)]
RuntimeError: Tensor.item() cannot be called on meta tensors

transformers 5.x now runs model construction in from_pretrained (cls(config)) under a torch.device("meta") context. Meta-device tensors are placeholders that hold only shape and no actual values, so code that performs value-dependent operations like .item() or .tolist() inside __init__ fails right there. clip-japanese-base's image tower calls into timm's (PyTorch Image Models) EVA02 backbone at construction time, and the pinned timm 1.0.11's __init__ is exactly this pattern.

As far as I could tell from reading the installed transformers 5.5.0 modeling_utils.py, this meta-device context is wired unconditionally into the initialization path, and I found no flag to disable it from the consumer side. low_cpu_mem_usage=False, which controlled similar behavior in 4.x, is no longer accepted in 5.x (it is treated as a removed argument), and specifying it changed nothing in practice.

Stage 2: even after upgrading timm, the next failure is the post_init() contract

A newer timm (1.0.28, the latest at the time) makes the relevant EVA02 code work under the meta device too, so stage 1 could be cleared. But now a different error appears:

AttributeError: 'CLYPModel' object has no attribute 'all_tied_weights_keys'

In transformers 5.x, a class inheriting from PreTrainedModel is required to call self.post_init() at the end of __init__, and internal attributes such as all_tied_weights_keys are initialized there. This convention is spelled out in the official Model structure rules, but CLYPModel, written in the 4.x era, does not make that call, so the attribute is missing partway through loading and it fails.

Stage 1 could be worked around by upgrading timm, but stage 2 requires fixing the code bundled in the model repository itself, which the consumer cannot do (monkey-patching the model class is possible, but it would have to be repeated for every subsequent incompatibility, and the verification cost of guaranteeing embedding correctness didn't seem worth it). So clip-ja stays on transformers 4.57.6, with the reason documented in a comment in requirements.txt, and the Dependabot PR was closed.

I also checked the security side of staying on 4.x. 4.57.6 includes the fixes for the ReDoS (regular expression denial of service) advisories, and the two model-loading GHSAs (GitHub Security Advisory) that are fixed only in 5.x are, I judged, not applicable to a usage pattern that loads exactly one version-pinned, trusted model and never runs the Trainer.

How transformers 5 frames this

As far as I looked into it, these appear to be intentional changes in transformers 5.x, framed as requiring the model implementation side to follow along.

• Initializing on the meta device is a design choice to avoid allocating memory at construction time and loading the weights twice. The Model structure rules above explicitly state, as conventions for model implementers, that a class inheriting from PreTrainedModel must call self.post_init() (rule TRF013) and that weight sharing must be declared with _tied_weights_keys (TRF004 and others). The same document also indicates, for trust_remote_code, a policy of not using it in native implementations because "remote code cannot be reviewed or maintained within transformers" (rule TRF014) — which hints at the structural reason custom-implementation models struggle to keep up with the framework's convention changes. Note that avoiding value-dependent operations like .item() inside __init__ is not listed as a separate rule in that document; it is required as a consequence of the meta-device initialization behavior.
• Similar reports appear in transformers issues (#43957, #43646) and across several model repositories that use trust_remote_code, such as BiRefNet, InternVL, and RMBG-2.0. As far as I could confirm, there is no consumer-side option to disable the meta device at initialization.
• Fixes are proceeding on a per-repository basis. For example, the audio model dasheng published a one-line fix commit that passes device="cpu" to torch.linspace(...) so the tensor is created on a real device (this resolves the stage-1 .item() error). For clip-japanese-base as well, once a 5.x-compatible revision is published on the repository side, the upgrade can be reconsidered at that point.

A checklist for deciding whether to upgrade

Organizing this experience into a set of checks for when a transformers major-bump PR arrives:

1. Check whether the model is a transformers-native implementation or trust_remote_code. If native, the framework already handles 5.x, so the follow-up is likely to stay within your own consumer code. If trust_remote_code, it depends on whether the model repository's code conforms to 5.x's conventions (meta-device init, calling post_init()).
2. Check dependency resolution first. transformers 5.5.0 requires huggingface_hub 1.x (>=1.5), so a pin at <1.0 stops you there. numpy is >=1.17, so a 1.x pin still passes.
3. For embedding use cases, check return-value compatibility. If get_text_features / get_image_features now returns an output object, you need to pick the field that yields the same value as your existing vectors (for the SigLIP family, .pooler_output).
4. Leave a record of the decision. When you defer an upgrade, writing "why, and under what condition it can resume" into a requirements comment saves you from redoing the investigation the next time the same PR arrives.

References

• huggingface/transformers#43957 — model loading with torch.device("meta") breaks some models on transformers 5.+
• huggingface/transformers#43646 — Transformers 5.0.0 breaks custom model initialization
• Model structure rules (transformers official documentation)
• line-corporation/clip-japanese-base (Hugging Face)
• google/siglip2-base-patch16-256 (Hugging Face)
• mispeech/dasheng-1.2B — an example of a transformers-5 compatibility fix commit