Prevent type overflow

[pik94]

Some databases, for example, Teradata, have a short limit for string and requires to set a number of symbols in CHAR/VARCHAR, i.e. it is required to write it like this

CREATE TABLE MyTable (
    col_name VARCHAR(n)
)

where n must be set and has some max allowed value.

To calculate md5 hash and convert it to an integer value, values of columns for one row are cast to strings, and concatenated afterwards. Let me clarify on an example:

CREATE TABLE MyTable (
    id INTEGER,
    data VARCHAR(n)
)

To concatenate we use a construction like this:

CAST(id AS VARCHAR(n1)) || '|'  || CAST(data AS VARCHAR(n2)) 

The question is what should n1 and n2 be? I am sure that the maximum allowed value for a specific type, for example, N. It is needed to keep all customer information without losses. However, such a concatenation will lead to a type overflow, because we are trying to have VARCHAR(N + N) which is not allowed.

To avoid such an overflow, we should shorten string values but not to loss information. I see one possible solution: taking hash for each item of a concat op, i.e.

md5( CAST(id AS VARCHAR(n1)) ) || '|'  || md5( CAST(data AS VARCHAR(n2)) )

Benefits:

• It is possible to avoid data type overflow for most of cases

Drawbacks:

• Performance might be decreased because we need to calculate hashes quite intensively. However, the current implementation enables it if necessarily i.e. only if at least one of databases participating in cross-diffing has a problem with a fixed varchar. This behavior is controlled by a PREVENT_OVERFLOW_WHEN_CONCAT flag.
• Probability of collision increases but i do not think dramatically.
• We still may have data overflow if where are many columns and a max n of VARCHAR(n) is low. Do not think it is a big problem because typically a large N might be 32000 or more symbols so customer columns should have more than 1000 column in diffing.

dismissing "changes requested" per previous convo