Linespace does not support SymInt step. (but int)

@chauhang

🚀 The feature, motivation and pitch

I'm working on exporting an object detection model which casts a list of tensor shapes into a tensor, and I'm trying to hint the compiler that it should treat it as a constant in order to avoid unnecessary GuardOnDataDependentSymNode errors. The problem is that when the list is casted to a tensor, it's data "unspecializes" and throws GuardOnDataDependentSymNode errors (on the linspace line).

import torch
import torch.nn as nn
import typing
import torch._dynamo as dynamo
import logging

class NetMRE(nn.Module):
    def forward(self, x: typing.List[torch.Tensor]):
        spatial_shapes: typing.List[typing.Tuple[int, int]] = []
        for xi in x:
            spatial_shapes.append(xi.shape[2:])

        spatial_shapes = torch.as_tensor(spatial_shapes, dtype=torch.long)
        reference_points_list = []
        for H, W in spatial_shapes:
            lin = torch.linspace(0.5, H - 0.5, H, dtype=torch.float32) # --- breaks here
            reference_points_list.append(lin)
        return reference_points_list

example_kwargs = {
    "x": [
        torch.rand(1, 3, 64, 64),
        torch.rand(1, 3, 32, 32),
    ],
}
exported_program: torch.export.ExportedProgram = export(
    NetMRE(), (), kwargs=example_kwargs, strict=True,
)

Alternatives

The easiest fix is to never cast the list to tensor and refactor all subsequent code that uses the tensor operations on that object. Although this is not a terrible alternative, I feel like it could be easily avoidable with the suggested feature.
The hackier solution I've found so far is disable the faketensor creation, but it doesn't work with strict=True and it also seems like it would probably break other things.

class NetC2v2(nn.Module):
    def forward(self, x: typing.List[torch.Tensor]):
        spatial_shapes: typing.List[typing.Tuple[int, int]] = []
        for i, xi in enumerate(x):
            spatial_shapes.append(tuple(xi.shape[2:]))
        
        with unset_fake_temporarily():
            with disable_proxy_modes_tracing():
                spatial_shapes_tensor = torch.tensor(spatial_shapes, dtype=torch.long)
        reference_points = self.get_reference_points(spatial_shapes_tensor)
        return reference_points

    @staticmethod
    def get_reference_points(
        spatial_shapes: torch.Tensor,
    ): 
        reference_points_list = []
        for i in range(len(spatial_shapes)):
            with disable_proxy_modes_tracing():
                with unset_fake_temporarily():
                    h = int(spatial_shapes[i][0])
            lin = torch.linspace(0.5, h - 0.5, h, dtype=torch.float32)
            reference_points_list.append(lin)
        return reference_points_list

Additional context

As far as I could see in the codebase, there is support for constant scalar tensors, but couldn't find anything else.

cc @chauhang @penguinwu @eellison @aorenste @liangel-02 @ezyang @bobrenjc93 @laithsakka @avikchaudhuri @zhxchen17 @tugsbayasgalan @angelayi @ydwu4 @bdhirsh @gmagogsfm @suo @zou3519 @yf225

Yeah, fake tensor has constant prop, we just have to make sure it actually gets triggered here. Your tensor is small enough that it should be eligible, so long as the as_tensor call appropriately passes make_constant=True to from_real_tensor. Have you tried using torch.tensor instead? I think maybe that one is setup.

@ezyang

Yes, previously I had seen that constant scalar tensors (t.numel() <= 1) were supported:

pytorch/torch/_subclasses/fake_tensor.py

Lines 1756 to 1764 in 9656a60

    
           if type(out) is Tensor and self.may_turn_const(out): 
        
               # NB: not in_kernel_invocation_manager because we're doing real 
        
               # compute here 
        
               # NB: no_dispatch() here is VERY DANGEROUS (like, segfault 
        
               # dangerous) if this is actually a wrapper subclass tensor, 
        
               # therefore the exact type test above 
        
               with no_dispatch(): 
        
                   out = out.clone() 
        
               return converter.from_real_tensor(self, out, make_constant=True)

pytorch/torch/_subclasses/fake_tensor.py

Lines 2220 to 2226 in 9656a60

    
           def may_turn_const(self, t: Tensor) -> bool: 
        
               return ( 
        
                   t.numel() <= CONSTANT_NUMEL_LIMIT 
        
                   and not is_sparse_any(t) 
        
                   and not self.is_our_fake(t) 
        
                   and not t.device.type == "meta" 
        
               )

pytorch/torch/_subclasses/fake_tensor.py

Line 101 in 9656a60

CONSTANT_NUMEL_LIMIT = 1

Although, for some reason it never occurred to me to override that constant...

So, in conclusion: torch._subclasses.fake_tensor.CONSTANT_NUMEL_LIMIT = 8 before compilation works :^)

Thanks @ezyang !

PS: Do you know if this could have any unwanted side effects? Why is CONSTANT_NUMEL_LIMIT set to such a small value? (1)

We could consider moving it higher. It's low pessimistically because of memory use concerns. Currently, I don't think we ever clear the constants from run to run so recompilations would accrue memory. Although we could clear all the fake tensors constacks in the stack after inductor is done.

Not sure what is a good default.

This is also something the compute_at_compile_time could help with, I forget exactly what the api is

Let's bump it higher / make it user configurable.

The option is have a way to explicitly say "please keep constant propagating this no matter how large until we're done"

@ezyang

@ezyang I noted that not all operations on this constant tensor succeed. For example, calling .item() makes it fail with Unsupported: torch.* op returned non-Tensor int call_method item:

New script (only change is .item(), full logs in this gist):

import torch
import torch.nn as nn
import typing
import torch._dynamo as dynamo
import logging

class NetMRE(nn.Module):
    def forward(self, x: typing.List[torch.Tensor]):
        spatial_shapes: typing.List[typing.Tuple[int, int]] = []
        for xi in x:
            spatial_shapes.append(xi.shape[2:])

        spatial_shapes = torch.as_tensor(spatial_shapes, dtype=torch.long)
        reference_points_list = []
        for H, W in spatial_shapes:
            lin = torch.linspace(0.5, H.item() - 0.5, H.item(), dtype=torch.float32) # --- Adding .item() breaks
            reference_points_list.append(lin)
        return reference_points_list

example_kwargs = {
    "x": [
        torch.rand(1, 3, 64, 64),
        torch.rand(1, 3, 32, 32),
    ],
}
torch._subclasses.fake_tensor.CONSTANT_NUMEL_LIMIT = 8
exported_program: torch.export.ExportedProgram = torch.export.export(
    NetMRE(), (), kwargs=example_kwargs, strict=True,
)

The relevant torch snippet is this:

pytorch/torch/_dynamo/variables/builder.py

Lines 2390 to 2395 in 2980aed

    
           else: 
        
               unimplemented( 
        
                   "torch.* op returned non-Tensor " 
        
                   + f"{typestr(example_value)} {proxy.node.op} {proxy.node.target}", 
        
                   case_name="unsupported_operator", 
        
               )

Adding the following line just before the previous snippet works, but there's probably a more graceful way haha:

elif proxy.node.target == 'item' and proxy.node.args[0].meta["example_value"].constant:
        return ConstantVariable.create(example_value, **options)

That's an independent Dynamo bug, where you have to explicitly say that it's OK for functions to return integer constants. I've tried universally accepting integer outputs but this is apparently load bearing so some fixing needed. WAR is fine...

@ezyang

@ezyang Okay, should I report these bugs in separate issues?

Also, these tensors throw "Dynamic slicing on data-dependent value is not supported" when one attempts to index another tensor with them (minimum reproducibility code in this gist).

The relevant torch snippet is this:

pytorch/torch/_dynamo/variables/lists.py

Lines 811 to 814 in 2265c2d

    
           if isinstance(start, variables.TensorVariable) or isinstance( 
        
               stop, variables.TensorVariable 
        
           ): 
        
               unimplemented("Dynamic slicing on data-dependent value is not supported")

Which similarly can be amended by replacing it with something like this:

        if isinstance(start, variables.TensorVariable):
            try:
                start = ConstantVariable.create(start.proxy.node.meta["example_value"].constant.item())
            except Exception:
                unimplemented("Dynamic slicing on data-dependent value is not supported")
        if isinstance(stop, variables.TensorVariable):
            try:
                stop = ConstantVariable.create(stop.proxy.node.meta["example_value"].constant.item())
            except Exception:
                unimplemented("Dynamic slicing on data-dependent value is not supported")
        if isinstance(step, variables.TensorVariable):
            try:
                step = ConstantVariable.create(step.proxy.node.meta["example_value"].constant.item())
            except Exception:
                unimplemented("Dynamic slicing on data-dependent value is not supported")

Feel free to do them separately if they have different texture / error message. However, your latest example also will be fixed by more aggressive const propping, no need to edit each site individually.

@dgcnz

@dgcnz dynamic slicing shall be supported now

Increase small fake tensor constant limit

Navigation Menu

Repository navigation

Linespace does not support SymInt step. (but int) #136642

🚀 The feature, motivation and pitch

Alternatives

Additional context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	if type(out) is Tensor and self.may_turn_const(out):
	# NB: not in_kernel_invocation_manager because we're doing real
	# compute here
	# NB: no_dispatch() here is VERY DANGEROUS (like, segfault
	# dangerous) if this is actually a wrapper subclass tensor,
	# therefore the exact type test above
	with no_dispatch():
	out = out.clone()
	return converter.from_real_tensor(self, out, make_constant=True)

	def may_turn_const(self, t: Tensor) -> bool:
	return (
	t.numel() <= CONSTANT_NUMEL_LIMIT
	and not is_sparse_any(t)
	and not self.is_our_fake(t)
	and not t.device.type == "meta"
	)

	else:
	unimplemented(
	"torch.* op returned non-Tensor "
	+ f"{typestr(example_value)} {proxy.node.op} {proxy.node.target}",
	case_name="unsupported_operator",
	)

	if isinstance(start, variables.TensorVariable) or isinstance(
	stop, variables.TensorVariable
	):
	unimplemented("Dynamic slicing on data-dependent value is not supported")

Linespace does not support SymInt step. (but int) #136642

Description

🚀 The feature, motivation and pitch

Alternatives

Additional context

Activity

ezyang commented on Sep 26, 2024

dgcnz commented on Sep 26, 2024

eellison commented on Sep 26, 2024

eellison commented on Sep 26, 2024

ezyang commented on Sep 27, 2024

dgcnz commented on Oct 25, 2024

ezyang commented on Oct 27, 2024

dgcnz commented on Oct 28, 2024

ezyang commented on Oct 29, 2024

laithsakka commented on Nov 26, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions