x86_64: fix near jump check

[inkydragon]

Seems a bug around x86-64 near jump/far jump check.

Problem: In detect_near_jump_type,
the x86-64 boundary check uses +6 for the upper bound and +5 for the lower bound regardless of jump type:

sljit/sljit_src/sljitNativeX86_common.c

Lines 694 to 699 in acf9f8f

	#if (defined SLJIT_CONFIG_X86_64 && SLJIT_CONFIG_X86_64)
	if ((sljit_sw)(label_addr - (jump_addr + 6)) > HALFWORD_MAX || (sljit_sw)(label_addr - (jump_addr + 5)) < HALFWORD_MIN)
	return detect_far_jump_type(jump, code_ptr);
	#endif /* SLJIT_CONFIG_X86_64 */
	short_jump = (sljit_sw)(label_addr - (jump_addr + 2)) >= -0x80 && (sljit_sw)(label_addr - (jump_addr + 2)) <= 0x7f;

This mixes jcc and jmp lengths and only rejects cases where both forms are out of range.

Impact: Edge cases where only one near jump form is out of range are allowed to pass.
If the emitted instruction is jmp/call (5-byte rel32) or jcc (6-byte rel32) and
its displacement is out-of-range, later patching asserts or incorrect displacement can occur.

xref: range check in reduce_code_size

sljit/sljit_src/sljitNativeX86_common.c

Lines 858 to 873 in acf9f8f

	#if (defined SLJIT_CONFIG_X86_64 && SLJIT_CONFIG_X86_64)
	if (type == SLJIT_JUMP) {
	if (diff <= 0x7f + 2 && diff >= -0x80 + 2)
	size_reduce += JUMP_MAX_SIZE - 2;
	else if (diff <= HALFWORD_MAX + 5 && diff >= HALFWORD_MIN + 5)
	size_reduce += JUMP_MAX_SIZE - 5;
	} else if (type < SLJIT_JUMP) {
	if (diff <= 0x7f + 2 && diff >= -0x80 + 2)
	size_reduce += CJUMP_MAX_SIZE - 2;
	else if (diff <= HALFWORD_MAX + 6 && diff >= HALFWORD_MIN + 6)
	size_reduce += CJUMP_MAX_SIZE - 6;
	} else {
	if (diff <= HALFWORD_MAX + 5 && diff >= HALFWORD_MIN + 5)
	size_reduce += JUMP_MAX_SIZE - 5;
	}
	#else /* !SLJIT_CONFIG_X86_64 */

But I'm not sure how to add a test for this issue.

---

background

Found by downstream project: digitalgust/miniJVM
When build with win64, sljit will crash with "Assertion failed at ..\minijvm\c\utils\sljit\sljitNativeX86_common.c:777"

sljit/sljit_src/sljitNativeX86_common.c

Line 777 in acf9f8f

SLJIT_ASSERT((sljit_sw)addr <= HALFWORD_MAX && (sljit_sw)addr >= HALFWORD_MIN);

[zherczeg]

It looks like a valid issue. It is interesting if this is never caused any issues before. Testing: code space is usually allocated above the 4GB area, so if you jump to a static function, it is > 2GB jump.

[zherczeg]

I think the original code used the "worst" value for both directions. When you jump backwards, (instr_addr + 6)-MAX_NEGATIVE_DISTANCE limits the jump, and when you jump forward, (instr_addr + 5)+MAX_POSITIVE_DISTANCE limits the jump regardless of jcc/jmp types. Maybe the two numbers (+5 and +6) were accidentally swapped.

The proposed change is better anyway.

Another question is whether <= and >= could be used.

[inkydragon]

E9 cw JMP rel16 Jump near, relative, displacement relative to next instruction. Not supported in 64-bit mode.

A relative offset (rel8, rel16, or rel32) is generally specified as a label in assembly code, but at the machine code level, it is encoded as a signed 8-, 16-, or 32-bit immediate value. This value is added to the value in the EIP register. (Here, the EIP register contains the address of the instruction following the JMP instruction). When using relative offsets, the opcode (for short vs. near jumps) and the operand-size attribute (for near relative jumps) determines the size of the target operand (8, 16, or 32 bits).

JMP: Jump (x86 Instruction Set Reference)

or Vol. 2A 3-496~497, Manuals for Intel® 64 and IA-32 Architectures

So, for jmp we have

• rel_diff = jump_addr - (label_addr+instr_len)
• Short jump, instr_len = 1+1 = 2: -128 <= rel8_diff <= +127
• Near jump, instr_len = 1+4 = 5: -2^31 <= rel32_diff <= +2^31-1
  • jcc, instr_len = 2+4 = 6
• Far jump, if current code segment changed

We donot use >=, <= here, since we invert the if condition, and going to call detect_far_jump_type.

[zherczeg]

Probably you need to add #if (defined SLJIT_CONFIG_X86_64 && SLJIT_CONFIG_X86_64) arond this as well.

[zherczeg]

I understand the concept. I was just thinking how the old code worked. Since this is really just 1-2 bytes we loose from 4GB range, probably the effect was little. Range inverting can have side effects as well, since INT_MIN and -INT_MIN is the same.

[zherczeg]

LGTM