Serialization for trained models/pipelines are not portable to hosts with different byteorder.

[andrewsi-z]

How to reproduce the behaviour

Load a pre-trained model from little endian platform on big endian platform and attempt to use.
This results in the exception "ValueError: Little-endian buffer not supported on big-endian compiler" as seen below in this post.

Your Environment

Info about spaCy

• spaCy version: 3.1.3
• Platform: Linux-4.18.0-305.12.1.el8_4.s390x-s390x-with-glibc2.29
• Python version: 3.8.10
• Pipelines: en_core_web_sm (3.1.0)
• Environment Information: spaCy was built from source on an s390x system, which is big endian.

Example: loading x86 saved models from fails:

nlp = spacy.load("en_core_web_sm")
doc = nlp("This is a sentence.")

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/tmp/ipykernel_75/3390045506.py in <module>
----> 1 doc = nlp("This is a sentence.")

/git/spaCy/spacy/language.py in __call__(self, text, disable, component_cfg)
   1003                 raise ValueError(Errors.E109.format(name=name)) from e
   1004             except Exception as e:
-> 1005                 error_handler(name, proc, [doc], e)
   1006             if doc is None:
   1007                 raise ValueError(Errors.E005.format(name=name))

/git/spaCy/spacy/util.py in raise_error(proc_name, proc, docs, e)
   1533 
   1534 def raise_error(proc_name, proc, docs, e):
-> 1535     raise e
   1536 
   1537 

/git/spaCy/spacy/language.py in __call__(self, text, disable, component_cfg)
    998                 error_handler = proc.get_error_handler()
    999             try:
-> 1000                 doc = proc(doc, **component_cfg.get(name, {}))
   1001             except KeyError as e:
   1002                 # This typically happens if a component is not initialized

/git/spaCy/spacy/pipeline/trainable_pipe.pyx in spacy.pipeline.trainable_pipe.TrainablePipe.__call__()

/git/spaCy/spacy/util.py in raise_error(proc_name, proc, docs, e)
   1533 
   1534 def raise_error(proc_name, proc, docs, e):
-> 1535     raise e
   1536 
   1537 

/git/spaCy/spacy/pipeline/trainable_pipe.pyx in spacy.pipeline.trainable_pipe.TrainablePipe.__call__()

/git/spaCy/spacy/pipeline/tok2vec.py in predict(self, docs)
    119         DOCS: https://spacy.io/api/tok2vec#predict
    120         """
--> 121         tokvecs = self.model.predict(docs)
    122         batch_id = Tok2VecListener.get_batch_id(docs)
    123         for listener in self.listeners:

/usr/local/lib/python3.8/dist-packages/thinc/model.py in predict(self, X)
    313         only the output, instead of the `(output, callback)` tuple.
    314         """
--> 315         return self._func(self, X, is_train=False)[0]
    316 
    317     def finish_update(self, optimizer: Optimizer) -> None:

/usr/local/lib/python3.8/dist-packages/thinc/layers/chain.py in forward(model, X, is_train)
     52     callbacks = []
     53     for layer in model.layers:
---> 54         Y, inc_layer_grad = layer(X, is_train=is_train)
     55         callbacks.append(inc_layer_grad)
     56         X = Y

/usr/local/lib/python3.8/dist-packages/thinc/model.py in __call__(self, X, is_train)
    289         """Call the model's `forward` function, returning the output and a
    290         callback to compute the gradients via backpropagation."""
--> 291         return self._func(self, X, is_train=is_train)
    292 
    293     def initialize(self, X: Optional[InT] = None, Y: Optional[OutT] = None) -> "Model":

/usr/local/lib/python3.8/dist-packages/thinc/layers/chain.py in forward(model, X, is_train)
     52     callbacks = []
     53     for layer in model.layers:
---> 54         Y, inc_layer_grad = layer(X, is_train=is_train)
     55         callbacks.append(inc_layer_grad)
     56         X = Y

/usr/local/lib/python3.8/dist-packages/thinc/model.py in __call__(self, X, is_train)
    289         """Call the model's `forward` function, returning the output and a
    290         callback to compute the gradients via backpropagation."""
--> 291         return self._func(self, X, is_train=is_train)
    292 
    293     def initialize(self, X: Optional[InT] = None, Y: Optional[OutT] = None) -> "Model":

/usr/local/lib/python3.8/dist-packages/thinc/layers/with_array.py in forward(model, Xseq, is_train)
     28 def forward(model: Model[SeqT, SeqT], Xseq: SeqT, is_train: bool):
     29     if isinstance(Xseq, Ragged):
---> 30         return _ragged_forward(
     31             cast(Model[Ragged, Ragged], model), cast(Ragged, Xseq), is_train
     32         )

/usr/local/lib/python3.8/dist-packages/thinc/layers/with_array.py in _ragged_forward(model, Xr, is_train)
     88 ) -> Tuple[Ragged, Callable]:
     89     layer: Model[ArrayXd, ArrayXd] = model.layers[0]
---> 90     Y, get_dX = layer(Xr.dataXd, is_train)
     91 
     92     def backprop(dYr: Ragged) -> Ragged:

/usr/local/lib/python3.8/dist-packages/thinc/model.py in __call__(self, X, is_train)
    289         """Call the model's `forward` function, returning the output and a
    290         callback to compute the gradients via backpropagation."""
--> 291         return self._func(self, X, is_train=is_train)
    292 
    293     def initialize(self, X: Optional[InT] = None, Y: Optional[OutT] = None) -> "Model":

/usr/local/lib/python3.8/dist-packages/thinc/layers/chain.py in forward(model, X, is_train)
     52     callbacks = []
     53     for layer in model.layers:
---> 54         Y, inc_layer_grad = layer(X, is_train=is_train)
     55         callbacks.append(inc_layer_grad)
     56         X = Y

/usr/local/lib/python3.8/dist-packages/thinc/model.py in __call__(self, X, is_train)
    289         """Call the model's `forward` function, returning the output and a
    290         callback to compute the gradients via backpropagation."""
--> 291         return self._func(self, X, is_train=is_train)
    292 
    293     def initialize(self, X: Optional[InT] = None, Y: Optional[OutT] = None) -> "Model":

/usr/local/lib/python3.8/dist-packages/thinc/layers/chain.py in forward(model, X, is_train)
     52     callbacks = []
     53     for layer in model.layers:
---> 54         Y, inc_layer_grad = layer(X, is_train=is_train)
     55         callbacks.append(inc_layer_grad)
     56         X = Y

/usr/local/lib/python3.8/dist-packages/thinc/model.py in __call__(self, X, is_train)
    289         """Call the model's `forward` function, returning the output and a
    290         callback to compute the gradients via backpropagation."""
--> 291         return self._func(self, X, is_train=is_train)
    292 
    293     def initialize(self, X: Optional[InT] = None, Y: Optional[OutT] = None) -> "Model":

/usr/local/lib/python3.8/dist-packages/thinc/layers/maxout.py in forward(model, X, is_train)
     47     W = model.get_param("W")
     48     W = model.ops.reshape2f(W, nO * nP, nI)
---> 49     Y = model.ops.gemm(X, W, trans2=True)
     50     Y += model.ops.reshape1f(b, nO * nP)
     51     Z = model.ops.reshape3f(Y, Y.shape[0], nO, nP)

/usr/local/lib/python3.8/dist-packages/thinc/backends/numpy_ops.pyx in thinc.backends.numpy_ops.NumpyOps.gemm()

/usr/local/lib/python3.8/dist-packages/blis/py.pyx in blis.py.gemm()

ValueError: Little-endian buffer not supported on big-endian compiler

[andrewsi-z]

I'm new to experimenting with spaCy and am trying to dive a bit deeper here to see if there is any possible workaround.
I'm working through the debugging guide, etc. to see how best to display the model/pipeline constants (weights, etc.) to see if I can possibly correct them after load. If anyone has a pointer or info, I'd really appreciate it.
I'm wondering if numpy arrays or objects are potentially backing the constants on load, and can potentially be adjusted (e.g. https://numpy.org/doc/stable/user/basics.byteswapping.html) for use as a potential work around.

[adrianeboyd]

We've never tested this and don't have any hardware to test with, but this is an interesting question!

Before making any changes, it would be useful to know if there are any failures in the test suite. You can install the test requirements with pip install -r requirements.txt from the github repo and then run:

python -m pytest --pyargs spacy

My guess is that it will happily run and save and load in native byte order throughout the tests, but since we haven't tested it, I'm not sure.

Then on to loading an existing model.

First, it looks like it would be faster to use OpenBLAS (through numpy) than blis because blis is not optimized. I'm not sure what your numpy package looks like, but you can check with numpy.show_config() to see which blas libraries are included. It looks like you would need numpy to be built against OpenBLAS v0.3.6 or higher to use the vector instructions on z14, or v0.3.8 for z15. See: http://www.openblas.net/Changelog.txt

The first thing I would try is switching from blis to numpy with:

import spacy
from thinc.api import use_ops

with use_ops("numpy", use_blis=False):
    nlp = spacy.load("en_core_web_sm")

This is a flag in the model ops (NumpyOps from thinc) for whether it should use gemm from blis or from numpy.

If you try to use this pipeline, you might get a similar error just from numpy instead, but my guess is that it will work but just be slow because of the byte order.

I think this is the main place you'd need to adjust how the data is loaded with non-native byte orders swapped to newbyteorder('=') or something along those lines?

https://github.com/explosion/thinc/blob/23376d9bbdaa822ef0c64a5eb4836487a0e5be4f/thinc/backends/ops.py#L29-L33

The same method is used in saving and loading, so I think it will always save in native byte order. It would probably not be too difficult to add a flag to to_numpy so it can always save as little endian but load in native byte order. I am not sure how complicated this would be?

And the other place you probably need to adjust is where the vectors data is loaded (which is not relevant for en_core_web_sm, just for md and lg or any other models you've trained with vectors):

spaCy/spacy/vectors.pyx

Lines 371 to 477 in f64e39f

	def to_disk(self, path, **kwargs):
	"""Save the current state to a directory.
	path (str / Path): A path to a directory, which will be created if
	it doesn't exists.
	DOCS: https://spacy.io/api/vectors#to_disk
	"""
	xp = get_array_module(self.data)
	if xp is numpy:
	save_array = lambda arr, file_: xp.save(file_, arr, allow_pickle=False)
	else:
	save_array = lambda arr, file_: xp.save(file_, arr)
	def save_vectors(path):
	# the source of numpy.save indicates that the file object is closed after use.
	# but it seems that somehow this does not happen, as ResourceWarnings are raised here.
	# in order to not rely on this, wrap in context manager.
	with path.open("wb") as _file:
	save_array(self.data, _file)
	serializers = {
	"vectors": lambda p: save_vectors(p),
	"key2row": lambda p: srsly.write_msgpack(p, self.key2row)
	}
	return util.to_disk(path, serializers, [])
	def from_disk(self, path, **kwargs):
	"""Loads state from a directory. Modifies the object in place and
	returns it.
	path (str / Path): Directory path, string or Path-like object.
	RETURNS (Vectors): The modified object.
	DOCS: https://spacy.io/api/vectors#from_disk
	"""
	def load_key2row(path):
	if path.exists():
	self.key2row = srsly.read_msgpack(path)
	for key, row in self.key2row.items():
	if self._unset.count(row):
	self._unset.erase(self._unset.find(row))
	def load_keys(path):
	if path.exists():
	keys = numpy.load(str(path))
	for i, key in enumerate(keys):
	self.add(key, row=i)
	def load_vectors(path):
	ops = get_current_ops()
	if path.exists():
	self.data = ops.xp.load(str(path))
	serializers = {
	"vectors": load_vectors,
	"keys": load_keys,
	"key2row": load_key2row,
	}
	util.from_disk(path, serializers, [])
	self._sync_unset()
	return self
	def to_bytes(self, **kwargs):
	"""Serialize the current state to a binary string.
	exclude (list): String names of serialization fields to exclude.
	RETURNS (bytes): The serialized form of the `Vectors` object.
	DOCS: https://spacy.io/api/vectors#to_bytes
	"""
	def serialize_weights():
	if hasattr(self.data, "to_bytes"):
	return self.data.to_bytes()
	else:
	return srsly.msgpack_dumps(self.data)
	serializers = {
	"key2row": lambda: srsly.msgpack_dumps(self.key2row),
	"vectors": serialize_weights
	}
	return util.to_bytes(serializers, [])
	def from_bytes(self, data, **kwargs):
	"""Load state from a binary string.
	data (bytes): The data to load from.
	exclude (list): String names of serialization fields to exclude.
	RETURNS (Vectors): The `Vectors` object.
	DOCS: https://spacy.io/api/vectors#from_bytes
	"""
	def deserialize_weights(b):
	if hasattr(self.data, "from_bytes"):
	self.data.from_bytes()
	else:
	xp = get_array_module(self.data)
	self.data = xp.asarray(srsly.msgpack_loads(b))
	deserializers = {
	"key2row": lambda b: self.key2row.update(srsly.msgpack_loads(b)),
	"vectors": deserialize_weights
	}
	util.from_bytes(data, deserializers, [])
	self._sync_unset()
	return self

Let us know how it goes?

[andrewsi-z]

thank you @adrianeboyd!

The switch from blis to numpy did the trick to start with. I am able to load and proceed... So now I can proceed with additional testing and then exploring the other updates mentioned here.

I did run the tests, and see 2 failures on my s390x system that are not present on my x86 platform:

x86: 2720 passed, 1098 skipped, 14 xfailed, 1 xpassed, 1 warning in 220.67s (0:03:40)
s390x: 2 failed, 2718 passed, 1098 skipped, 14 xfailed, 1 xpassed, 1 warning in 412.78s (0:06:52)

The 2 failures are:

• test_overfitting_IO

           doc = nlp("London")
>       assert len(doc.spans[spancat.key]) == 1
E       assert 0 == 1
E        +  where 0 = len([])

• test_issue8168

>       assert ruler._ent_ids == {8043148519967183733: ("GPE", "san-francisco")}
E       AssertionError: assert defaultdict(<...-francisco')}) == {804314851996...n-francisco')}
E         Left contains 1 more item:
E         {18072791780479830321: ('GPE', 'san-francisco')}
E         Right contains 1 more item:
E         {8043148519967183733: ('GPE', 'san-francisco')}
E         Use -v to get the full diff

I will explore these 2 failures further.

[andrewsi-z]

Another quick update. As I try some of the examples here, I see very different results on s390x when I attempt to use en_core_web_sm after load.

In both platform examples, I set use_blis to false to keep things consistent.

Code:

from spacy.lang.en.examples import sentences 
from thinc.api import use_ops
with use_ops("numpy", use_blis=False):
    nlp = spacy.load("en_core_web_sm")
doc = nlp(sentences[0])

print(doc.text)
for token in doc:
    print(token.text, token.pos_, token.dep_)

Output

x86:

Apple is looking at buying U.K. startup for $1 billion
Apple PROPN nsubj
is AUX aux
looking VERB ROOT
at ADP prep
buying VERB pcomp
U.K. PROPN compound
startup NOUN dobj
for ADP prep
$ SYM quantmod
1 NUM compound
billion NUM pobj

s390x:

Apple is looking at buying U.K. startup for $1 billion
Apple VERB ROOT
is AUX ROOT
looking PROPN ROOT
at PROPN ROOT
buying PROPN ROOT
U.K. X ROOT
startup NUM ROOT
for PROPN ROOT
$ NOUN ROOT
1 NUM ROOT
billion NOUN ROOT

[adrianeboyd]

Ah, I should have also listed slow + nonsense output as a possibility. I was hoping numpy might kinda sorta do the right thing, but there are various places where we use a memory view, and then the model weights are probably out-of-whack unless you fix the byte order on load in thinc. (Is my guess, anyway, I am not really sure. But the test suite results are promising that it will work once the data is loaded correctly.)

[andrewsi-z]

Thanks! I'm going to try fixing the byte order as you suggested and we'll see if that helps...
I'll also try running a training example and just confirm that path works ok too.

[andrewsi-z]

After tracing through the code a bit, I seeing that to_numpy (referenced above) is only called on model save and not on load (at least I am not detecting it getting control). I did try changing this to save in little endian instead of native, but the problem persists.

Looking at from_disk (https://github.com/explosion/thinc/blob/a9b220da045619d12ef3c859e5e947d56cf9064e/thinc/model.py#L626-L629) I was able to see on s390x some of the weights here being loaded in little endian buffers. Although I think numpy knows how to handle this for python logic, I am assuming that this is causing issues in the cython code.
I tried adding logic here to convert the loaded weights and dtype to big endian, but errors persist. I believe this is likely because little endian buffers are in use for other constants not handled in the snippet I referenced.
(Note that all thinc unit tests also checked out on s390x).

After more investigation (including function entry tracing, etc.) I am seeing that a number of different locations in both spacy and thinc appear to be participating in loading/processing various parts of the pipeline which is making it a bit difficult to get a firm grasp around which loaded values may still be in an incorrect format.

I'm very new to spacy and so I apologize if this is more appropriate for a discussion thread. I will start one if so... I have a couple of questions as I'd like to try to address this issue:

1. Is any example code to navigate a pipeline and view the ndarrays and other loaded numeric values? I want to display their dtype attributes, etc. and this will help me understand what other areas may need to be touched. It may also help me create a temporary workaround (in the form of some python logic that can "correct" loaded models).

2. For the small model, do you expect spaCy to directly load any weight values or would they all be processed through thinc calls? I generally saw most of this processing as spacy.util.py ->thinc.model.from_bytes and then a lot of convert_recursive hits.

[adrianeboyd]

Ah, you're right, I got mixed up when I looked up the serialization functions.

I think the closest debugging option we have is spacy debug model, but I'm not sure it shows enough detail, and I'm also not sure it's currently working correctly (just now I kept running into errors, at least).

The model weights are all loaded through thinc. There are a small number of parameters stored in local configs in JSON or msgpack, but not as numpy arrays. I think the vectors data is the only place where numpy arrays are loaded directly in spacy.

[andrewsi-z]

I'm working on getting en_core_web_sm going on s390x , so I'm using that for testing here.

I added logic to convert_recursive to issue ndarray.byteswap().newbyteorder() against all ndarray values I could see getting loaded through that path and then spot checked both the bytes view (numpy.tobytes()) and arrays and the result looks good on a big endian platform. Prior to this, I did confirm that these were getting loaded as little endian buffers on s390x.
Further, the blis error has gone away if I try to use blis (I am usually reverting to numpy ops for testing here).

I am still getting garbage predictions (but different garbage predictions then before!).

Using spacy debug model as inspiration, I wrote a script that uses model.walk() to traverse the model and dump out all the model weights, etc. and display byteorder and other attributes to a file. I ran this both on x86 and s390x, and compared the results.

Some observations:

• All weights appear to match on numpy view (i.e., print() ).
• All weights/numpy dtype on s390x are big endian.
• From what was dumped out, the only difference I cannot explain (and I'm not sure is significant) is that the layer node.id is different for quite a few of the layers between s390x and x86.

On that last point, the node.id values are consistent across runs on each platform. On z they are consistently different though. For example:

   s390x: Layer 10: model ID 17: 'maxout>>layernorm>>dropout'
   x86    : Layer 10: model ID 21: 'maxout>>layernorm>>dropout'

I am not sure what to make of this, and whether there is any real significance to it yet... I am basically doing the following:

    for i, node in enumerate(model.walk()):
        if not layers or i in layers:
            print(f"Layer {i}: model ID {node.id}: '{node.name}'")

This is based on loaded en_core_web_sm as nlp, and then looping over nlp.pipe_names.
Then pipe = nlp.get_pipe(pipe_name) and pipe.model before looping over the layers.

I don't have any thoughts yet on where the issue is. Two next possibilities I'll explore:

1. Review any spacy level configs just in case. (Need to see how to dump this).
2. See if I can trace all the inputs / weights coming into the first layer (to start with) and see if I can check the inputs and outputs and compare LE vs BE).

Will report back, but if anything here sets off bells please let me know!

[andrewsi-z]

I think I've just about got it narrowed down @adrianeboyd ... After my earlier changes, all of the keys, values, ndarrys, etc. match until things get to the forward pass of a hashembed layer. I see hashembed.py function forward issues the following:

keys = model.ops.hash(ids, seed) % nV

And the result is garbage, though all the inputs are the same.

The hash function is cython and resolves to hash in numpy_ops.pyx.

Looking at that code, it looks like it calls an x64 (platform dependent) version of murmurhash. I am pretty certain this is where things deviate. Need to confirm to 100%

[adrianeboyd]

That would do it, yes. Nearly all the statistical models use HashEmbed and if the hashes are off you'll just get nonsense. Doesn't look too hard to fix, e.g.:

https://github.com/kloetzl/libmurmurhash/blob/master/murmurhash.c

[andrewsi-z]

Hi @adrianeboyd , I was able to get results to match for the _sm model across platforms of different endianness. So far the changes I had to make are:

1. implement byte order sensitive implementation of murmurhash3 (used approach as shown here: https://github.com/hideo55/node-murmurhash3/)
2. numpy_ops.pyx hash has logic that maps a unsigned char array to the input ids array (uint64). The approach used maps the underlying storage and "views" as char array, and this implementation will result in different values depending on system byteorder. I modified this to use the same value regardless of system byte order by implementing a bit-shift based approach.
3. on load in convert_recursive, when a match is detected after calling convert_item, ndarray.byteswap().newbyteorder() is used to set by order
4. on save of numpy values, all values are saved as little endian.

Todo:

• murmurhash2.cpp update for byte order
• update spaCy/spacy/vectors.pyx for endian load.
• Instead of step 4 above, (or in addition to), it may be good to record save platform byte order in the model meta data.

The changes implemented so far are contained in thinc and murmurhash.

From a design and architecture perspective, how should I go about determining if the maintainers would be willing to support byte order awareness? I am happy to contribute this and will work to meet all validation requirements, community guidelines and style (etc), but I am curious if there are any general reservations before start that .

[honnibal]

From a design and architecture perspective, how should I go about determining if the maintainers would be willing to support byte order awareness? I am happy to contribute this and will work to meet all validation requirements, community guidelines and style (etc), but I am curious if there are any general reservations before start that .

This question has come up a few times, and overall our response hasn't been that positive, I'm sorry to say. We always regret to not support some configurations, but there are trade-offs to be made. Ultimately the case will come down to costs and benefits. Until now the only requests we've had for this have been from IBM, and if it's only relevant to you and nobody else, it might be a better subject for a paid support contract.

The main concern I have for costs are the maintenance, complexity and performance implications. I'm reluctant to commit to not breaking big endian architectures in future, even if we get it working now. I understand we can probably avoid having any performance implications for the little endian architectures, but I expect that might come at some implementation complexity.

One solution is to try to make the modifications in "user space" as much as possible. For instance, if we can mostly confine the changes to a BigEndianCPUOps class in Thinc that is then set as the "cpu ops" if the hardware is detected, that would give you a space to put in your alternate implementations, and you'd have control of maintaining that. I think we could make that work in Thinc quite well. If there are any places we use endian-sensitive operations outside of a call to the ops class, we could move that into an ops method.

Perhaps the ops subclass in combination with doing a conversion over the weights files would be sufficient to solve most problems. If so, we'd be able to support big endian architectures quite smoothly.

I'm trying to think of other places where the ordering might matter. Have you seen this small string optimisation here? https://github.com/explosion/spaCy/blob/master/spacy/strings.pyx#L66

Finally, do you need to use GPU? The change in byte ordering between CPU and GPU seems likely to cause problems in some places. The first one that comes to mind is that there's an implementation of the hashing algorithm in CUDA. This will not work if you pass it a big endian-aligned array.