Hi there,
I found a bug related to multibyte characters in the terraform plan output.
Terraform Version
$ terraform -v
Terraform v0.11.8
+ provider.aws v1.34.0
Terraform Configuration Files
provider "aws" {}
resource "aws_cloudwatch_metric_alarm" "hoge" {
alarm_name = "あああああああああああああああああああああああああああ"
comparison_operator = "GreaterThanThreshold"
evaluation_periods = "1"
datapoints_to_alarm = "1"
metric_name = "UnHealthyHostCount"
namespace = "AWS/ELB"
period = "60"
statistic = "Sum"
threshold = "0"
}Note that あ (e3 81 82) is a Japanese character consisting of 3 bytes.
The aws_cloudwatch_metric_alarm uses alarm_name as an ID of the resource.
Debug Output
Crash Output
Expected Behavior
The terraform plan output should not contain an invalid byte sequence in UTF-8.
Actual Behavior
When the length of ID is longer than 80 bytes, the terraform plan truncates the extra bytes.
If the ID is multibyte characters, It is truncated without considering the multibyte boundary, and this will result in an invalid byte sequence in UTF-8.
Steps to Reproduce
$ terraform init
$ terraform plan
$ terraform apply
$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
aws_cloudwatch_metric_alarm.hoge: Refreshing state... (ID: あああああああああああああ...ああああああああああああ)
------------------------------------------------------------------------
No changes. Infrastructure is up-to-date.
This means that Terraform did not detect any differences between your
configuration and real physical resources that exist. As a result, no
actions need to be performed.
Dumping the byte sequence of terraform plan output with the od command is as follows.
You can see that the bytes at the address 0x0000129 - 0x000012A is 81 82 and it is an invalid byte sequence.
$ terraform plan | od -Ax -tcx1
0000000 033 [ 0 m 033 [ 1 m R e f r e s h i
1b 5b 30 6d 1b 5b 31 6d 52 65 66 72 65 73 68 69
0000010 n g T e r r a f o r m s t a
6e 67 20 54 65 72 72 61 66 6f 72 6d 20 73 74 61
0000020 t e i n - m e m o r y p r i
74 65 20 69 6e 2d 6d 65 6d 6f 72 79 20 70 72 69
0000030 o r t o p l a n . . . 033 [ 0
6f 72 20 74 6f 20 70 6c 61 6e 2e 2e 2e 1b 5b 30
0000040 m \n T h e r e f r e s h e d
6d 0a 54 68 65 20 72 65 66 72 65 73 68 65 64 20
0000050 s t a t e w i l l b e u s
73 74 61 74 65 20 77 69 6c 6c 20 62 65 20 75 73
0000060 e d t o c a l c u l a t e
65 64 20 74 6f 20 63 61 6c 63 75 6c 61 74 65 20
0000070 t h i s p l a n , b u t w
74 68 69 73 20 70 6c 61 6e 2c 20 62 75 74 20 77
0000080 i l l n o t b e \n p e r s i
69 6c 6c 20 6e 6f 74 20 62 65 0a 70 65 72 73 69
0000090 s t e d t o l o c a l o r
73 74 65 64 20 74 6f 20 6c 6f 63 61 6c 20 6f 72
00000a0 r e m o t e s t a t e s t
20 72 65 6d 6f 74 65 20 73 74 61 74 65 20 73 74
00000b0 o r a g e . \n 033 [ 0 m \n 033 [ 0 m
6f 72 61 67 65 2e 0a 1b 5b 30 6d 0a 1b 5b 30 6d
00000c0 033 [ 1 m a w s _ c l o u d w a t
1b 5b 31 6d 61 77 73 5f 63 6c 6f 75 64 77 61 74
00000d0 c h _ m e t r i c _ a l a r m .
63 68 5f 6d 65 74 72 69 63 5f 61 6c 61 72 6d 2e
00000e0 h o g e : R e f r e s h i n g
68 6f 67 65 3a 20 52 65 66 72 65 73 68 69 6e 67
00000f0 s t a t e . . . ( I D : あ
20 73 74 61 74 65 2e 2e 2e 20 28 49 44 3a 20 e3
0000100 ** ** あ ** ** あ ** ** あ ** ** あ ** ** あ **
81 82 e3 81 82 e3 81 82 e3 81 82 e3 81 82 e3 81
0000110 ** あ ** ** あ ** ** あ ** ** あ ** ** あ ** **
82 e3 81 82 e3 81 82 e3 81 82 e3 81 82 e3 81 82
0000120 あ ** ** あ ** ** . . . 201 202 あ ** ** あ **
e3 81 82 e3 81 82 2e 2e 2e 81 82 e3 81 82 e3 81
0000130 ** あ ** ** あ ** ** あ ** ** あ ** ** あ ** **
82 e3 81 82 e3 81 82 e3 81 82 e3 81 82 e3 81 82
0000140 あ ** ** あ ** ** あ ** ** あ ** ** あ ** ** )
e3 81 82 e3 81 82 e3 81 82 e3 81 82 e3 81 82 29
0000150 033 [ 0 m \n \n - - - - - - - - - -
1b 5b 30 6d 0a 0a 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d
0000160 - - - - - - - - - - - - - - - -
2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d
*
0000190 - - - - - - - - - - - - - - \n \n
2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 0a 0a
00001a0 033 [ 0 m 033 [ 1 m 033 [ 3 2 m N o
1b 5b 30 6d 1b 5b 31 6d 1b 5b 33 32 6d 4e 6f 20
00001b0 c h a n g e s . I n f r a s t
63 68 61 6e 67 65 73 2e 20 49 6e 66 72 61 73 74
00001c0 r u c t u r e i s u p - t o
72 75 63 74 75 72 65 20 69 73 20 75 70 2d 74 6f
00001d0 - d a t e . 033 [ 0 m 033 [ 3 2 m \n
2d 64 61 74 65 2e 1b 5b 30 6d 1b 5b 33 32 6d 0a
00001e0 \n T h i s m e a n s t h a t
0a 54 68 69 73 20 6d 65 61 6e 73 20 74 68 61 74
00001f0 T e r r a f o r m d i d n
20 54 65 72 72 61 66 6f 72 6d 20 64 69 64 20 6e
0000200 o t d e t e c t a n y d i
6f 74 20 64 65 74 65 63 74 20 61 6e 79 20 64 69
0000210 f f e r e n c e s b e t w e e
66 66 65 72 65 6e 63 65 73 20 62 65 74 77 65 65
0000220 n y o u r \n c o n f i g u r a
6e 20 79 6f 75 72 0a 63 6f 6e 66 69 67 75 72 61
0000230 t i o n a n d r e a l p h
74 69 6f 6e 20 61 6e 64 20 72 65 61 6c 20 70 68
0000240 y s i c a l r e s o u r c e s
79 73 69 63 61 6c 20 72 65 73 6f 75 72 63 65 73
0000250 t h a t e x i s t . A s
20 74 68 61 74 20 65 78 69 73 74 2e 20 41 73 20
0000260 a r e s u l t , n o \n a c t
61 20 72 65 73 75 6c 74 2c 20 6e 6f 0a 61 63 74
0000270 i o n s n e e d t o b e
69 6f 6e 73 20 6e 65 65 64 20 74 6f 20 62 65 20
0000280 p e r f o r m e d . 033 [ 0 m \n
70 65 72 66 6f 72 6d 65 64 2e 1b 5b 30 6d 0a
000028f
Additional Context
I had found this issue when parsing terraform plan output with a third party tool, terraform-landscape.
At first, I had thought it might be a landscape's bug, but after a while debugging, I found out that it was a terraform core's bug.
$ terraform plan | landscape --trace
/usr/local/Cellar/terraform_landscape/0.1.18/libexec/gems/terraform_landscape-0.1.18/lib/terraform_landscape/printer.rb:87:in `apply_prompt': invalid byte sequence in UTF-8 (ArgumentError)
from /usr/local/Cellar/terraform_landscape/0.1.18/libexec/gems/terraform_landscape-0.1.18/lib/terraform_landscape/printer.rb:32:in `process_stream'
from /usr/local/Cellar/terraform_landscape/0.1.18/libexec/gems/terraform_landscape-0.1.18/lib/terraform_landscape/cli.rb:48:in `print'
from /usr/local/Cellar/terraform_landscape/0.1.18/libexec/gems/terraform_landscape-0.1.18/lib/terraform_landscape/cli.rb:34:in `block (2 levels) in define_commands'
from /usr/local/Cellar/terraform_landscape/0.1.18/libexec/gems/commander-4.4.5/lib/commander/command.rb:182:in `call'
from /usr/local/Cellar/terraform_landscape/0.1.18/libexec/gems/commander-4.4.5/lib/commander/command.rb:153:in `run'
from /usr/local/Cellar/terraform_landscape/0.1.18/libexec/gems/commander-4.4.5/lib/commander/runner.rb:446:in `run_active_command'
from /usr/local/Cellar/terraform_landscape/0.1.18/libexec/gems/commander-4.4.5/lib/commander/runner.rb:68:in `run!'
from /usr/local/Cellar/terraform_landscape/0.1.18/libexec/gems/commander-4.4.5/lib/commander/delegates.rb:15:in `run!'
from /usr/local/Cellar/terraform_landscape/0.1.18/libexec/gems/terraform_landscape-0.1.18/lib/terraform_landscape/cli.rb:25:in `run'
from /usr/local/Cellar/terraform_landscape/0.1.18/libexec/gems/terraform_landscape-0.1.18/bin/landscape:7:in `<top (required)>'
from /usr/local/Cellar/terraform_landscape/0.1.18/libexec/bin/landscape:22:in `load'
from /usr/local/Cellar/terraform_landscape/0.1.18/libexec/bin/landscape:22:in `<main>'
References
#12261
#13292
#13317
Thanks!
Hi there,
I found a bug related to multibyte characters in the terraform plan output.
Terraform Version
Terraform Configuration Files
Note that
あ (e3 81 82)is a Japanese character consisting of 3 bytes.The
aws_cloudwatch_metric_alarmusesalarm_nameas an ID of the resource.Debug Output
Crash Output
Expected Behavior
The terraform plan output should not contain an invalid byte sequence in UTF-8.
Actual Behavior
When the length of ID is longer than 80 bytes, the terraform plan truncates the extra bytes.
If the ID is multibyte characters, It is truncated without considering the multibyte boundary, and this will result in an invalid byte sequence in UTF-8.
Steps to Reproduce
Dumping the byte sequence of terraform plan output with the
odcommand is as follows.You can see that the bytes at the address
0x0000129 - 0x000012Ais81 82and it is an invalid byte sequence.Additional Context
I had found this issue when parsing terraform plan output with a third party tool, terraform-landscape.
At first, I had thought it might be a landscape's bug, but after a while debugging, I found out that it was a terraform core's bug.
References
#12261
#13292
#13317
Thanks!