From 866ebbf6aae1b07f5355c560997c149f619c17f1 Mon Sep 17 00:00:00 2001 From: Shane Carragher Date: Mon, 11 Mar 2024 17:09:51 +0000 Subject: [PATCH] added pattern for self managed apache kafka with pipe --- .../README.md | 67 ++++ .../example-pattern.json | 61 +++ .../smk-pipe-log-group.png | Bin 0 -> 20738 bytes .../template.yaml | 353 ++++++++++++++++++ 4 files changed, 481 insertions(+) create mode 100644 eventbridge-pipes-self-managed-apache-kafka-to-log-group/README.md create mode 100644 eventbridge-pipes-self-managed-apache-kafka-to-log-group/example-pattern.json create mode 100644 eventbridge-pipes-self-managed-apache-kafka-to-log-group/smk-pipe-log-group.png create mode 100644 eventbridge-pipes-self-managed-apache-kafka-to-log-group/template.yaml diff --git a/eventbridge-pipes-self-managed-apache-kafka-to-log-group/README.md b/eventbridge-pipes-self-managed-apache-kafka-to-log-group/README.md new file mode 100644 index 000000000..584a02d17 --- /dev/null +++ b/eventbridge-pipes-self-managed-apache-kafka-to-log-group/README.md @@ -0,0 +1,67 @@ +# EventBridge Pipes with Self Managed Apache Kafka as source and CloudWatch Log Group as target + +This pattern shows how to use EventBridge Pipes which polls events from a Self Managed Apache Kafka which are then sent to and stored in a CloudWatch Log Group. + +![Pipes diagram](./smk-pipe-log-group.png) + +Important: This application uses various AWS services and there are costs associated with these services after the Free Tier usage - please see the [AWS Pricing page](https://aws.amazon.com/pricing/) for details. You are responsible for any AWS costs incurred. No warranty is implied in this example. + +## Requirements + +* [Create an AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) if you do not already have one and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources. +* [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed and configured +* [Git Installed](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) +* [AWS Serverless Application Model](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html) (AWS SAM) installed + +## Deployment Instructions + +1. Create a new directory, navigate to that directory in a terminal and clone the GitHub repository: + ``` + git clone https://github.com/aws-samples/serverless-patterns + ``` +1. Change directory to the pattern directory: + ``` + cd eventbridge-pipes-self-managed-apache-kafka-to-log-group + ``` +1. From the command line, use AWS SAM to deploy the AWS resources for the pattern as specified in the template.yml file: + ``` + sam deploy --guided --capabilities CAPABILITY_NAMED_IAM + ``` +1. During the prompts: + * Enter a stack name + * Enter the Primary and Secondary IP's of the Self Managed Kafka Cluster + * Enter SMK Topic Name + * Allow SAM CLI to create IAM roles with the required permissions. + + Once you have run `sam deploy --guided` mode once and saved arguments to a configuration file (samconfig.toml), you can use `sam deploy` in future to use these defaults. + +2. Note the outputs from the SAM deployment process. These contain the resource names and/or ARNs which are used for testing. + +## How it works + +Apache Kafka is a distributed event store and stream-processing platform. It is an open-source system developed by the Apache Software Foundation written in Java and Scala. EventBridge internally polls for new messages from the Self Managed Apache Kafka source and then invokes the target. EventBridge reads the messages in batches and provides these to your CloudWatch Log Group as an event payload. + +## Testing + +In order for EventBridge Pipe to receive events from the SMK Topic, If you do not already have a Apache Kafka Cluster provisioned, refer to the following quick start guide provided by Apache - https://kafka.apache.org/quickstart + +You can provision this on a EC2 Instance, providing the Instance is reachable from the EventBridge VPC provisioned by this Template. + +## Cleanup + +**Note**: EventBridge Pipes utilizes a AWS backed Lambda ENI to Poll events from the Self Managed Kafka. Due to the creation of this resource, it can take up to 20 minutes to delete the attached Hyperplane ENI. Upon deletion of the stack the following resources will be retained as to allow the deletion of the stack to succeed; + +* VPC +* Security Group +* Private Subnet + +In order to delete the retained resources, once the ENI is no longer visible on the "EC2 > Network Interfaces" Section of the EC2 Console in the applicable region you can then manually delete the VPC, Security Group and Private Subnet to avoid any additional costs. + +1. Delete the stack + ```bash + sam delete + ``` +---- +Copyright 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved. + +SPDX-License-Identifier: MIT-0 \ No newline at end of file diff --git a/eventbridge-pipes-self-managed-apache-kafka-to-log-group/example-pattern.json b/eventbridge-pipes-self-managed-apache-kafka-to-log-group/example-pattern.json new file mode 100644 index 000000000..d6da67149 --- /dev/null +++ b/eventbridge-pipes-self-managed-apache-kafka-to-log-group/example-pattern.json @@ -0,0 +1,61 @@ +{ + "title": "EventBridge Pipe with a Self Managed Apache Kafka as the Source", + "description": "Create a EventBridge Pipe to poll a Self Managed Apache Kafka and send the events to a CloudWatch Log Group for storage.", + "language": "YAML", + "level": "200", + "framework": "SAM", + "introBox": { + "headline": "How it works", + "text": [ + "This sample project demonstrates how to integrate a Self Managed Apache Kafka Cluster as the Source of an EventBridge Pipe.", + "The EventBridge Pipe polls the Kafka Cluster for events and once received sends these events to a CloudWatch Log Group for storage." + ] + }, + "gitHub": { + "template": { + "repoURL": "https://github.com/aws-samples/serverless-patterns/tree/main/eventbridge-pipes-self-managed-apache-kafka-to-log-group", + "templateURL": "serverless-patterns/eventbridge-pipes-self-managed-apache-kafka-to-log-group", + "projectFolder": "eventbridge-pipes-self-managed-apache-kafka-to-log-group", + "templateFile": "stemplate.yaml" + } + }, + "resources": { + "bullets": [ + { + "text": "Amazon EventBridge Pipes", + "link": "https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes.html" + }, + { + "text": "Self managed Apache Kafka stream as a source", + "link": "https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-kafka.html" + }, + { + "text": "Apache Kafka Quickstart - Thrid Party Reference", + "link": "https://kafka.apache.org/quickstart" + } + ] + }, + "deploy": { + "text": [ + "sam deploy" + ] + }, + "testing": { + "text": [ + "See the GitHub repo for detailed testing instructions." + ] + }, + "cleanup": { + "text": [ + "Delete the stack: cdk delete." + ] + }, + "authors": [ + { + "name": "Shane Carragher", + "image": "https://gravatar.com/userimage/247879270/a37712369945ab50dd4e8bf609988420.jpeg?size=256", + "bio": "Shane is a Snr. Cloud Support Engineer with Amazon Web Service (AWS) in Dublin, Ireland", + "linkedin": "www.linkedin.com/in/shane-carragher-aws" + } + ] +} diff --git a/eventbridge-pipes-self-managed-apache-kafka-to-log-group/smk-pipe-log-group.png b/eventbridge-pipes-self-managed-apache-kafka-to-log-group/smk-pipe-log-group.png new file mode 100644 index 0000000000000000000000000000000000000000..4cda3422c9b9468e10cbaa9a963137d688e4b46f GIT binary patch literal 20738 zcmeIY1yq#px;G5NP|_U&(%nN#4Bd?&AYDTvjpWdXNTY(lfJlgdN-HG@2!e!krwY>D ze9tKUw|k%c?sLBHJMVeVyVh~JhUdASJFok9<$cBJXer}eqrQfMf`X^2qM(O@0)he0 z_pva5U%IoJ72pr5m!7g5O8I9R;D!8?x8iMYSATm)7dsRdi2Sc77D0ZPyO%c$M1e(6 z(8|Mu*VfU>!PCmsi`UK08z2GSUEOUR?H%oGf3*?h7Zl>*hw$)=83>55Kx733fFFnu zuYkCK$*=ZS4t8!=18Tb8b98aBVi8ml=H&;5VmEShvvv3L0v@#tfFA*VfDDlXo`4S` zLcbpUMT9s2imZo+i=C03wYnp)4kclTC@(|=AhWBh7-(v+2+9M`E{@K2z#nBh8)tW9 z7kQYcyDLCZ5)c&R1&01Pr@ocFm8at$Lm>BL=WXTiYd;#A2G$Dl4mPrexAeqC6_xb8 zoNoQvhp(Ndm!rGe)wYFz;eejV&fWnYcE6~$cD{}_z=TMumyMOn6f>{9w z{MG*_>CZ#?#m3*v^mKOz#{1*U{`~yAk*mMFoePrJzxjAI4* zghk~5x|g@7yE8EKuYs-_+gf?Serg-!M1T-_I|4Y+u(Gyu(Q)^3^!_E1*6!Zk?yi3{ zkacl%K+3AOy9YqEx_Yw#)`Ap4IhePr3-A?CRzKH%g>V5tCV#zOZ5}zLm6yja2-`dQ zBiAA4;qHhWMbQ@+42fgpAOMVhDgjr2V9OpbUMoK@AznLV3u{1t9PBJS96gW%1L!uS zazJjw?n)T{$szwF{cEKC4R6YJfRgj}3;>3`8WAFVg_0}jh6oG)n%3`cvhrI4!2S@E z-!!W$>2&yOWaOIvBM1iCTnWIxUOm#>{DYkQQ*Z+F{$$txfD?e(e=ay}T-<$Z{j31N zws3KG_^*Pd2>(AoQ{?LX4`>STi}DIz>FYn?DR$NLzYjbi|2{nbsk;2%tY6FVKHF{($V@~XzKq8btQn*El4r{7wSq>g!f8^{fP+is{{D&g9xF29}#wL zwz5bU2(TPBE>>P%jy3@M|FB1%?mlj|e;K@ACh@nEgFLBUGuQz}1sDN%`A@VTe+c1k z6Y?j!f73keT&%nuegA&`e|84{+6eqIVSh7ZI!L?vH!CLi52`}w_xtaeyxe^}ZS1ak z{l_fSzcb~(F`VM!zhU=#Qg16y2RrZo50gTG2n8^Xe_88)&6}&%KW*f{*YZDRO0UG{ zU-^0egUp|94xo+x_>Xi|{sEanLVvHYg!l#i5hnln*cATlon2{3VA5Zj;QxAL0%80= zIC(!^Yq_6}GjJ91t3=l{ZoBI z04^gL{Tnd<=iU3OaQzRRB#-n;jgVd`uuu(m2jEv3V7bSC)ARp(>kI)9@P7gk!r~C# zD-8Tr4#NKg6G*LwJpaGL#1-wo4<@b_{s$)X?Og1EWoufw0cjmu;7Jw`Nff00G|?H>ItUHPZm^YiQP1nI92R~?Y)>tBgA z$ER~eC@AzOstU3O{^sl1m}#W#Bb}Q)pIsjJqCUV%M9^Qmb}dCggWeiM#2>~O3u5H7 zN5w8I(pF5e)gq8ZRb)9TDOTZUz(}E|$7+28?+rM-S$jC_;-|Uq{dfcWX6@(kh|dPo zFHU^451R)YAHIG2Rw6c~>acP&PjNJ4PDP4A?ViRoGpGa0V6!LsKA)eg+1;qu;dk5| z>={2C98uJ1Ia4Rma%zBMQHWvrmlvb+8pn`^4d4m zF*f8j8^TRgCx&s?OPJr{8%l9ZyK&1;%EwRz0qf>uf6_doClR+8D7%x^lrh0hpYodI!|YHT3x{^;Dxyv2a|pvsN%ONE-YiL;|J$oC4)` z!~A=qAK?#%ZCZjcT)7aF50Oi;jD$WTfvLyT>aI6uLU*rKS5}lzIH?*dQRgzZ)0;3eL|!!4`V|+|P&FfNzKxl7G;F zc&JPO-_fpVeVrUnN_#LhLaS$gAc~&1m8_k&x14bQ4b|o>?(0U=tT)P009X695uOpA zVFk^${$C%gySX7Ka~cIkIEi5){`C@Orp?N|scsKAmsvqX3iR$RpLCATEoR7XF5Kz& zK71B-nW^eSHk>r=#5#1#FfhK;>>hPNvsWsWmq9eJKQ!?D++*Uc3W0(L1n%g>Fd|=; z%Gw}o&AzufW2?{%v0?r?34(G8Cu~O|rTdxc;4_kzC%jL$Q;O2qFv1*$0#Ng~j9;PN zXRR<8Q^^XY{Xnorm}G{;LRct>xu3d^mRJ`8WACZq$;Deyd^bNN;@aXfGCb!rOF_fY z8Bh^R3re8!%Z3z$tU%yX7(zL@(7l6#1C?ZNe#F-n0Y0#ILUOfRIE#+w+rIGCY30V6 zsq8G-uP+2L##otvV~)j6f;vn3DJftHvG#Z#UhpQnXOj`QzFBBAV)$AmJtuLl7whOJ zFvii}Ah$l1w=c&dUB5krJ(OC^q$3oJm)2uk^xf|yf1c29pd{l}@$E+1otWm*r`dsL zo?8z#?Bjw3<$$g5p%TLqzLjG+!#i*w(b*KRHLpE~nw=OKgNlbE99qvF4G-JhZq$i) z-YT;n!dPYl<2`R*C4g}E@PdE@ek|5d!xD^R|M2`wFjjrpo0+lz&M6EGOnVyrI+_-koqtow@AYvK>>1CPsT8R9%CQpDTursA156 zJ19`71bkHsTxBZ#DZWZj`U22;e<9f`pK)$>$p(x2$#LpQkDvB_j(+?#+j4YPnaMg% ze_h(c?>g#+`7_izcN69e^LT|YK+h|JF{21VUVc~WPDN|+a<+66t{O|@okEF9!CI`Ho7@trp1?~DOjft5RTL0 z8yljdeZ}o{)o%t56+I`b`=ZO>GVE_HnE}ywOiLXnwu0 zc=A38>|vU}hrNG+%%L%x?9_AD=e-^Fw3`^{__qLDp%BesMJgcTGI+07LT!4cr`?lj zX2QQbjq!aO7zd>XchAg%&=;k39={qai#D9x5@O3)O=G*T3?~C=4&gf(G9u{r9;_+_ z&Q|e~1_MV7y-kR9AIinUgPvG4A8Au{O{hZW$6e?1!6FDNw;RYw!J9DF9^O&TPp6|o zGmMu)m#<$iaCcOn?^!a6#C*B<9k7lO=4V#bIhRZ@zTZ-#yDJ&2^Wu!kH%JImi+2LU zxYr~CQi(36ElGPZRz2z9ZY(kv=&&AIYaZq05OFen1h7h56Bt{|EJ9Ec9 zCyDp2`2fI=>X117J|4nAbDWR21=O?+yXAoeixC%>M7_wpjDFFJ`8-s>3H!bvsY9#r zUKP0Fc6@Uo+J{3|oX?A`lj99MAf9id6_%&Ue6k1SJj86~L2(BeiqD|CpNsdY&G?4c ziBPk!)0(!WNN1Ymbk#rNV+RRRTfY;kZRo5Hcs?lq_N0%d_{as*AgG393nHCYIaN4$ z<4r?|pb8*?!QgJ0X$-sgknk6q2lB46#!#~Pi=n!B?Wj5&6Eukd{94<1=;9-ORfMC3 zZ(o}X%pspfp#=mA7;VB&pFJD5lQI zni8ZKA20+UdYzgj2X@mcD!@+QwcNj_+xM}2uKqcfrR0K#DszA9l3EfJCl!YXy_e8+ zgrM-5$C!5S6IVXzouxZKC%ELF&2{Tj#6g#cijO=gNM(-24bDCL z4?1p2z6J)w@mI>h6!XXNVT6;fa&0xAs3M}eqGD6*Nv9fVx@XQ+2PP3w_wF4Lt7MMd zv@K9%Q%v1x3d6^bT)VH+t^n0Q*p7_q>>XXMbS$bQ#5}fNNvUfMrPab5qTx)sa3Ua6 zyIfWEU%l?|C1vBCj<)aQP7<^duS}qgx&Og-{uG?3_=3q)f?vmIY|eq7^8KJ$(G7wp zT%BQUBqq6GNBx~KGU}uuoUcyf)JRadu!qjlF%`dzLXSI5&okQ_ZH_eV`biQ^nPtlN zsFq~*7TJwO0;~m%Fzge80MQS40{C5l(kHgvIMnX8soHU zhvvPqd2Ug4e|VB1v`+A=!z#`wv~@ol&Fvi-hUZC-?wacE)D*hjh30(GOG|Z;*f3)d zd!`3hc&5Lc@(?!KP3RfoA>pL(D|w|`GPFqx;72?qGX3mob!1GtivtYe+p3`6H8b10 z;>0j=G#0N+@TyuV{oxM!wdC;Ywv{8A{oJK&@>+++g9CmU9#vU@VPGB>Bqc0mVWha&LeD<9bZdipSzp-lvnv;wUMsPRqHCmD^WOr%=o=sxXat@kqBf^~MDlfP5|@!T#D zUS)ujvr;f#%Sx6xqbr~wWp1!}!s}0iJ^+abw!>*e1 zC^QMeHgKE53@~cUKLR}>M*3M%rB{B|%$tX|DQsU-qisO6`R)2X%Prry!02={T`OxH z-VpnEa-wy!lzCF2#Slcq*8jqJUS*Pv6f=X8{;BX63032R*<9vJnDFUt-q&2YW|zB{ zCPB?La~l(&h0|Iz@8f3~JLLhlzI*Xl!Pr!S?;^@Zlc?C>U)`74@5k;N_T2Nz1v~XG&$yh*n zEMXu#2e@3iV8*Od@oEFSLUy?}FsK_A2cfx>-gH1~;5Okz6URm;R6|9cO61oN&p2U~`tmuX6;c^)6{}9wTA{0*g~K9s>LIi z`+M#_(;FcsjFJ;Hm1mCBX?@c6EIap>_qLcMeR6&y!#Fnur!rubRf-z6t3-1LXB)8> zc*T3a?GB1I#k8Ip6+iP5XDHXc-ep`ys)?{;6`Rs325e@v@_ocY>9o#%TS)Ww)Eo6+ z;?11|&+ymr0?`7wgJ1D7&R&nFt02Y|M-XPjFbue^?<5wWtFczqK6Li%8?I_-q8Obn z?%NX>O9xqgGjHlQZQiZ3g`G^IF*p<4(g5efp}iKzNnQi$(wJz(Y)=Wg63tg5!VZnu z`{sJKP~4`pDesBYP0)R#umB7$yIB>L8iiZr+d(y}nY}|L=v0(zP!m_scM|qspVWoj zG2S9*2V3p9Zx^tn=#j@(#IO#3T{lKcba?)gT-mPpS%1y>_gK%eo|eHd$LT=>3W%eU zzPzLQ3R1HUlA-Ufb8{OH{5XdpZ@8GXzT*k76;igw#6;Y;IHK-;bjisbb80`GoyE~q zI^#p?dHVTqJ2_(~xAeW&7xC!T(X1!nOmjVvZ%lA|$Q+h%v-_;Hin@C01-r2}{oz8r z&i)UEiut9La~AF8G{8H0mDn;+-iP+tzAF`0|LW_eP158ymMDm$VaNSfcb$ra%v>{} z9dDAQ=d+I}o5slR7;0Kbwvyjw?ss|bG{ufBwAhoMr*QYKK$ZI}ozeqJ)Waz+0&HO% zzI*KsH|Z}s{7A`Hz(rAEU8F>WgvCb-1Rrl4WGq~-e)1!q${Ar>Q;`+_;CU_7rNS^{ zOYsqNfKl_Rjr!78wDXtC2RV4vo!#L(Q}l5AE*AVLEbOWlDiK2*h{NZ|u2eR8HMMI~ zKBH#LL)>rs*+1(Pye1v3*+636lDkQ8mGLHZnBicmYde` zlUZv_aGRP!C5*BfF;{MPU*u?al3c|`6@f0-*ZPLJpF46KOSAW}09v8OlYE+L7Phtz zaU_o3Zd>=_GlaIlw^!rmDY%9#N7(}@iN~rONxXlIBw!e!gni1om34VguNEZ6CTW)7 zCUBJI5$w>F5KW8CM7(5Hf*bJ7DI4ARIxFXkRBOE2r>H)+gzxvQV-;wDdLMGRwz6Ui zk@V*mW68v`Zc6kPuQU@Wu(kFJ*q?;D4@%}&G?RC$g44hrT#OGF?W+Fsma&&+@D*1Ek!F9!p=ioae)1?iGuE!xk~2~EXES2|u0e=g3`X={c7 z?sVfme_F*9t9ao{9-W*9xoD2dCSplZKkUU)Elp=`@X6i^nnm#Qz4P`HrOLVayXnqe zYV@ajY-9;Y&q?so{pCXC4p}prl38-mQVbvbLtK+C7HEYEy9)Z<__iT$zei!g+qcxK ztE)j5uhf`wZOot94eell%6M|`EM4FZjPUxZcQ+DwoZxv%nSFm-u|Ra zm~7W}MS1a5t3|?Cn!#u?WAOz9v!CI)39&0mP2*hy*&*ViuUYO0Kle+%EsvvKrFHZZ zi_S{z;5+k5Ooag$LdWxT=ov3=5J257(eL-IiJ7T^cr-SY>Y;&auRAR#svNBb1)Y=% zNU#p%n-5?DX*}m-oFgtV-TQ%U__>v;#LQA2ME=pa&Zd)$j)ZphP}}7>mZ6%Sz1g*} zAI~3|hf4(>z-Ij5Yr1X08UBQ<6RG2|^x{(Xs7iR2T~9JlG%x)X;FVu>P-&!oP_;X3 z8|7j<;-stR!OhW#VQ}KAVo7YFcxVdM4G#oBv20vJ^g}g>*cl12r(UlHj@k()6U^#j&E}V_#%Mi+&zTC{LoWZ#0 z)7ipYA0yx}*&IJg$x*(8BE`4wQBF@Dm$&Nj1F;!?+{L*FJ?Ac&IaFIvvC8GHz8{Pn-wnQiMdshD7Jfkz=I;L1wAsZl68i*&; z5gFd@HynRNrpJLZ(}mDpDHe&Lg;B65Qt;lvvgvrByUxRFA-&atHmGN>r{|%d+V!#m z6%A{*i|9I9oZo(Q9nNIfJFiOidb$1S56hZ&zT$R3yVt^6%4H*OMW{)kk(aJsV~tPe zGHQKDi#Ojs6hgVOva;1LPw@U|dy#>G;eG#R3^+3YPX3YO^U${nIN)bqe`{#+1PJSh zt_>bC_lJWeb-=#n znAz}k_2uMIwv7u0A!53TT;E4N*EyK!ySA5;Y{sx$!bN?$jJO~|Z3EZ1PGYQ2)*f=i zL~yhAgiC!CoQhL2r|-Z0w3F5!m*y_7I!7|JlC&tm25_M&o!3~t^*mu(=fe3Qnv(v2 zv-yH#2cx;1pDJ{jD(RqGhpUvqQ~@zSNGWKG#cllN!{iOf280^#0snd|ds87y5#cgm z&tB*JB%k6gMOE}3i>T=ggcz2=BRY@eC&tLN3#bKj{KKKAvc#~jb;1XFnD{hFT6gobD?L<% zF>&U;kE;<(lz3svw^q}(4r)SyhP&DfO9we+Sm;sciVt1NVArvQ zxY$iBBfy#Hgk6#A;O>i45>kyj8)cor8yrc;E3P%8>STm)73No=mm!Zu-E+JJArnzg z&Q6xiB?$WKt0nEYuVcTKGUb#mEiKN#i2QyLu5B1wmA?YGFP|lDOn!eyG}#7#K%K#x zhgtjXqnfBaKewvo17^5Aj^Uz`DuT*Ei6vY+j36TA>?2D`GeixUdP8jltIeLs-j`(5 zz6!Pv1$3Jh{KQ07rSeujY&W7@K9FcZh943;heUg^p3;$>B z>@r03AR74F{A(=^s=c$_j~hAq%+Y}m=44CHboIl7@)6@OwvUSyVr>o`@uCKmq7-Tr zuF->0*YnD&^!&{RXRkAekA2mjm(PW+lx77<_nkON)@jv#tA_$+CPV_6W0BK=oDqMN z-gPXN(p&2NvQ(s6gF#oj;5qmGy7IgsvqYH_x*`x0LrqLRuqXw17P3H3PHV&7J#+7w z%IDz9v%SD*{_erbRb%SqVb2;4;lf7LkUQa1sP?_QU*Ri1OmKRhVPr~FeD+oQ<2%Cu z&upsu;tc3|-|(WxHZI?sy-9&q(voANmQ+Pga3+E@CT( zAn!!m=e=#g^!GOI?HOsp$?E!7lr;pCC2YpDiK*aBx>eba@*p0pq>}q#*fi@qG(&5j zb|<=k;M$}5Ca>i?UrP4O*BEN{T-vbq1O8fE-TIJ<{J0$5Y5L{3PwOy|ab^%IQH$lu zcL_!V4x8h$=ocOO^ZpV~wJ#J*)(5P+&M_{m#>62UzoK1^sojpXlI)!iH6^ZDDmfDb z#GU&>Uzl?R0loX4vW^Ha@(^a61Co#HqBmZMT&|a~uUA_U%Cv|^CY za1iBhCb9;+O#M`zjL#Z;xpw22RLx?hm_2KIfIW77q%}>7Qvtlj2n6Dna_J9Fz@=QJ z;SF5o(Jw3rj)kbSV;7tFL$MW2j!#Vsr0h+)l8Fhn&nLp)Ipf4`&T37R7G@lg{NT2@ zlPi+cUl1A-8B%rS z=m){n=H!ySPrj_9x6D%IdRS0y5VQqbU+B4pP|ej~oRx~}ai8M|`{+fID8E%cG3rwa zc^b^!aL6=I?6elZ!<_5DzQ;d*VL0`TQ_Y{6O#3-Tipt+Sf7L=4CY|kN-j6pX=Fn-* z3GB*g3H(zy?&!5sghDS~)sptA{!B(?Q|OQ=V&dz4yMu#0YJ6jX1re<`U(8Av2g2i{ zeB0X9Xj{HDF@N#uLtG=nM!vplV~ySB1y)q5(eaT)o7%KPHn`53(91J39}h_Lp#teX zE*A>kDC+{UO3V@dON*N9;6CLJKjm#@S++QgwysCBiLx)tvQ*vu z>n5q6kD89s!+$DcjEzZwUI&!S+5c+RUk@3Ay};rxpxL6SBI6du}-p9t{1F;&?PSJ<;`Hk z1adeeE-WduR?_-Nu}~7{#{5TxH=?7dVJ}rfntV zbxp1e=dR9&aq~MIywC2TJE=+9mN)cnwT-&Sl+X~t*1%hH(Dz3Kh_l7AkayI6#y49u zPsPNBZk9&BpWynPmt{zMSRkfa5XXpzVA%U&$Zthuc^V8huM~2u*YC}aDP~~q$2Frn z6-pEjV!B^+{6+j!wA++ez?PyEHDaU1%rYbY4{OsO}zgl(i6f z(DmqxO$MV^D_%BgFug7#NTEdn&)tCD%Gh8w|XrgJLi&*P91gsW{Nd z+pCy~j3Hj_4kqBSJg$n8r@vT*L>1~B%`>p*X0gCU-TCKQbhR1bg)w@BK=K*~Wc#93 z4-{b3v`~o)59!depu`gB{lZhG{)zI$UcRWRx}Ig`Wt`1h`5jy@;5uoQD`?Kf(NQ7i zkjeM<1*W;!#1AP=CT++RMQko=iL%^C@Hn(jggx5d??DdAH?jX$*_HH!s5*Q2Dre z3MIu+Bx^Cg8p@YuKgSV;&OhrG%e{(Q2Yy|)S*Dfs#}RRVMFGcAcNkIqw_`p<{nu)$yo(8aMjbN}9*cKRhC=?wm8 z5akNKUsxWV67a*dvJZC=EuDbHYi&gFe(#6!j3-IeD{hmM;bdER71kH&9JG{M-|{5c z@t�LP-gWPmbM@#<}=*EuiSQ+J*YU>SoGXrk6=uq$47C9ZO7ecccmJ*h^Vm&F`0KLa6E-68F1w3I8V>oZ>#ZOS!s`N&HY>kIOK(@;}m$-4WyMepdCVl3O<)miUU?LC<_Q z3iCL8%QF#bI&`T7MY(aGrNSlhjbav=>~PsCHIPqINudRH{CXps)vS1O>C|Pm6JhUd2M&zlB2EGuOl5pPYBqo;SahTW;xr-D+j|3{*gT!elU8 za7ji5?ud@2`>d<_(sF?Ujx9!={Zko_wGO{Xx10Pk4y*BK3|~o?RS%xgUg8N~l5$FT zPBn4NZGDJVLcBIfbeqM}obcHdv7QWdP^gA_9a0+e^*j}r_Zii7j)j7Wi>K-ccFON5 z4(ue+68o=S+j_7@4JR$>0$g57hA`9_oQ`0WqqF(c9}h_A@49HW^c(oFr0WQ|Rx_Sl z``XEW*Vwu9v}`bgMXO9psTAtv*Ni^(?HRxG?Tuk;MdVd7LKCsoM5Y6%izALHOk!`* z>(eIC4&@8cbEm^dPoNNlkORDWp9`5hdqG6-bOl={%YtL#NpWRQ^3&7Gz~fJaoSChTO`e%<4wf zOus(^5$q)erYLtz*(ZB>-2>EiSj1>MW?O0 zO333TVh>sMVh_=x3gNt+m$QKB8(DQ6(JTB^jR;uuHu%2%OL)ro5AS^@=V@@4Xo2Ci z5~xkW>5q87mS9T~A*o7HG1I3wpL@s$jq;!Os}RCAcHMx}ciUgAG@*z_RtvOHqT=pG z&gb;+W2nE!%E0F>^5%zXenQ}QF++Lw5v)=dp zc;lkHHDuw)JW2AoyCq)Hph62Hh|v$9{7KF51G>JG8j%*pMfT7AH*AM1VhZ@RDZh{~ zy{PMxpjxPaxfhcY!=gRnAn)GZA=udM3}YHO;TItnO4J~P7c+)c!hk!$tpnrD5CW;B zUycOcLw}ph=fm%YvGjU$rCbXn2QCXn&2&0DC9)?auiMj?c}rPJY~)aH9PtWjM0@}P#HM&0YDLdaKhl~T58%i=#{dzr^e@Sktr$~Wt9V8W>KzVvwPZPxoTPM!4@87SfO%E1*3b!P z)zP)rTfVof&Uf(w!mz%~V6UM|1%Nmh0Rdzsqjpw8E(V1XvXo z2kL`Ik4D|$*2aS=7~+VTGNo_!8IPHe6-0!nL_l>9h_hTPV)gJbi+#TtvM>Qm!k*cL zq4xE=7hOqhR$T32*wV_Qm0qo5U^IGAAyyJJ)ygY7ShJB|y;~4H^%_!A%)w3<9Fb=~ zX2Mpp$1{X*&3p^^=um+h5)xuEfzl_I!GOo_p-| z30^TbPlogYdS>b3t^L5BlxiF6f%Tu2CG^O~gm6nD*bpc|_H6gd@hh+4A4xi9?C5sN zqckb}>f+M3pO(-eS%=&b(E2+oAsHxPVC;1KM}NMvOw_EAFe?E0&J4> zSWg%^HgZEq-o~O}SA}iqTKzH7ZS62{=DtQ3;~$+tj6gI)A7t?Oub%Xv5hReM^xptA z7*}f`gogpV#lgBk#`<#uKPLeyq*M^beQDgpS91c(rKblF3Tjo3iu>IYDAkifZ3DUf z%*H!tE2lZvhSs)gxEhg^s-M2 zPP|OC1G1gLt6W@M5$`seOi!MU3lA0Z_cnNa(?TY*8ukX&`=2Q%^1HGK$N$=doT_EV z11f&IZc=xWBR`J7#D&v|4L|If<+=9}gYS;U9dh?CM4L8^t?$VdFl8r{Ej8m1r+7BuGgN!&-adkfST~`6WAWwrAGeeDr+% zgM))8mkJarL3(&|#$Ejp?|xHd?==!g)$-<^-KPxfcJe^{t>(aI%@)sEySOo*?LVGp z<$Gp|I1XgbBqFp&H;KWgoqJ>v0;jMD(#^9ZI$}zcTv)%}i$K5UtaGfwLbw(aOU^dX zX6Ooy&b2v3BNI@%v1Jmhb2?)orNRD5F?0N0F%1#F7r_n*`}60|2NA4e5ZLMA=AkjV zwd=dJH=itL8UpT%2%!3W(G)GgZ(WjnAiYbS4TW5L5GO*a|M-YQgS9`$e*xS7Xg)?( z8bl`5fBPH_6F;&yOB7A!UI)(J@*@j1pVyAJL`c55PdplLTzq)5APfBHw&FTjLfWzP zcEHBFO=)&+ZtyX?c~e^aeylyj-}j^a##DXP2h~298Tf(pOK&PX`cxu#^x;J$Hc1eM zrBW#YCOq1;!-0VHnJM(a!J{2zKGd?=(+nQdf-c-+L&kdO;;iV|&A40**};5?Z;VVfXV*bKTD7Q)8;e z&ESjjLF?X~Q)iM-i-{Cpc;7nokF{Zp)>dbzP->8}7mp}@LI8Px9bKE0O$!^>cU!GE zEXk}CurrlxGs-)mompw;m-B5gjbB>MJ{s-4GWx+pV&nO`Ba+5K)VVgOBdnH-!dr+q zaVF+fa?X!{*kWwT0gd`I*}>N~%I)NsWtg-w^j7LrVsmqtdSlAw2&?1m#ZLKL>EMEI z_#>CVy^mT1$8no^-<(Qci*g@|S_n^7B;xEi0!_W1P7%3^t2WQmO zq2|e9dOw6^ygtnMtTlbvEhono3i)bUH60sDVKZIViHlCd3-6v?*^!R3pFLVPJcDQt z5ge;oOxCzQpUrJPY>Lgxao}ox+0*=GJTP~ZgVEJa1<|FL*|2?vCSMiwo-YD}vH!F9 zH=1_#EFO31i=)Ls4U!*q6W-+w!Q6>Koav$8?hggKczA(u^IINY`SrRT8 zob3qqB^6#t51fCT{c#rYsF|V{-{XwuZjHI<3BSk1kxrP<`2al6pjI57LH(!hhP<|qCG_YotO>dYF)?-X6%JvuYaN=;p? zRBM^|lCOxJvK@ca0_N@{Q!#PT3QwkmSS1dE};w&>u-C*LmVMxXO~bWKO<7 zT69ep(sD&o56zlHfojOZVFVF}wUs(MiE?nOGD9Zg&(AJ~$^ygzZY!nY)n&k(3 zo((dbuJ81rl+DJ%2r_}nMpqB+(21hvGyzQ5e8&CHFF)y>${4`N{11z zGm)%o+Kt|~z#mzNUS8r82-=Q&xMy0e-S<&SF`(^H*f@)pFvEvS^J?%!qRa(5DNT_0 zF*9CyxYWUz4Zp)@nu<@{2U^V=Z-0xWacs)len3S8yTH)70P?VY zboOBCO)ovov#3cam33@7Kta7b3q0;gAHmIRMopS3oS_mrw#2Lz*aF>m1bLH%m&-n8 z!;ms9WVFIa;HiIsZK5mV(}y^IhrQGa(l)+7Qhl84fdD0N=rwdr)Vgmt;?rWYvCoA4}+>okFANal-seVu$qiam=_oI$IxS{ge(6+v=A)jma?6K02YE z&JeV^QMIacOow4Tef`GMg&|fRg-Oh%X0kE|M9#O2@8*MVuD_9j;gXi;GSbwug_=M8 zE?!Mo3{BxVs?$w1AW$KAWH?92u@v@Tdwio~96 z6FIx)R?{)=Oe|#uY}fM4Cg$5KUrY`|S_GH??Q7acTy!h2e-9UN#6?yNC0UoXf#5iC zq2#|uRQnMc#|weZ;6fuI7Ycb#^q-!cCJsmW@F$*}SuwIAh8U?n5_G<*c@e-p=lbc9 zhec&Oud-K^?ElZeQ46^kNAW3M_qw?|)~w&><-`J;bHvqBl7pjB=bCO{fkk zg}Hi@x{i<~Yz5q0`)a-3S+lUm86Vr?ZulW6KZbYKy+L5h5;Wluo<5rx{)wfbPo%ny z%=7)HN+uat@LSoj@6a= zP`aJqtP<@U1S_n$B~S}PlNfTjz;C+idCc?^1SM=ksT1!-11y#{Wu8t!ts*w{>f8K7 znro*BsCR!XyXPx2yDe?n)SRQ_>K|Qye=>%mJmXhSmt2eOx}C~m56+yZK@)ebZo~B~ z+^Mn3GEQ-S&d=R17l5W@GM)cOyaik+c-vhdS|8^JBo==5~&Z0TKUU-j4qv3j$fE-C3hY@$U#ke>VSa2#w_m|lX^XHN$^(mD493w2|w}LaDHJDGPLj}#9G#4=#5;d zgQi7c3CV%-rIwva>43Hks7FsQq~N(cFLN}22!NiR-UK8pxNC=L092;xuw*IF3u0|m zF66JXjFAw-lJ82DqywHa7#e2rRzW|%u@cq5su-wKlCNWj5XGti4sHI<8~vicJX*L1 zE8sIft-!BN_{(Yjk2fm>Q{RLW;%B+6W6_s?6*B7m@_Hnj=}=ub^djhXQ&4@Y4RAYk zl5;gOw19|^_A|pK#z{rK6`WNjhnK(s(X8g|Ik5sBvH18%GfO0(&Gh>UcBo0bmitPc zQot{lojwJ7Yg^FR;_*$+Wc{xy8x^#!qTKzD<%P+eWa-}ZJ;Zo1j#~(|$sZ~fYpY3x z;(Rre^#ND~gbvVRY*#t?g*vZoI$MGA zih01HOg`x@wbPyAA4n-Nc}&F`v{AQ+kGvBVHMr&;k!!tdlN!ex*^qc7g&}K1;t)gw z39vZc^lgF9cPAAM?;fVqtYEY` zl+ATW$`1(f%e*DrXSx@9GUSEi3J_2kf$R-y6;esp&Lv>|^cDS!v$OZmsjop;PZTB? z?-)}P!y4#GurbYHuOEI=?7TQXeQqogkYsIM_4_*Q#c>X^C8_4-E3rD7>hL-v`<@i5=p}c@PeLIrqmEtFy8SBir`b6xM+x zJ2Jpif(qTuQ&iq7CvV(UNL6@D@X5j6UTqjqn@6ft5Z9sA2-?{nT~$?;kmFbFrk7ol!j+QFotknYzlGpC1A;Z=$d7r#dQ&yzO`FBbJM|uu4FQ!^ zv?}TdE3=(tL;Gf<3iCm=J9VtnAkC^#wHXc!lSJsQzhsJkYqh-wd$+$VOJ=qpQ~zz+ zGxV)*j9|XHAY=V?p=*6$DqA}{cn(#Su=9w@vG3a0ycefI)8&J*^!S6*s2`e&SB!vv zXGS>0rBd8=0%}IQtbiT+mIe@dHdGnwx$eoTKQ%E*KxV~|@wpDkmgs5wM4=JFu`Fz5I-;Ww zu`<3&>h$n-b2*g|&)iPJHjLCE8X1wa59I|J56sT+)T9;rXc!d?9{Zl`I(&6^`Z7Bq z3gp!VqOjY^++%WGzj#)@zu`hS?a_l#dOeB;nJ4@ytMe+*ALnH({jap|9U`_^hpRjJ?MwDSf(XxG$gs3TsVjInk+eM)l!{wYJU z3(c$U%%zz2C#V?b62rdRvVL{Q{bXkL{YlmQY`Q;tm`ob`{e!ld+%pa>NU8JtriHiQ z%navi$FFbgsf$>D!0IBH07_+8W?AdTbK-Mutnq@nkam#smuY&z;s$ItQBy_vIa1Ze0 z_kXD)AVnr07J6w-B`nnvs6g1*ta$NhX;c09=3_^+87{QIq8G=oFJCGwmj_t-9|*DP zJ*=Q!AzA*JllhqnVN8TFZ_cv9$pa3rSN|=^(Ju?6LU4(D5i4~z>^OhrLo9CsNv9g7 k!fSseK7a(06!_wX5dQT;1 + This template is used to deploy a simple vpc that is partitioned + in a private and public network, with a configured NAT gateway for + the private network and an internet gateway for the public + network. It will also deploy an EventBridge Pipe (utilizing the NAT Gateway) to poll + the events from the specified Self Managed Kafka Cluster IP via Port 9092. + https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-kafka.html + +Parameters: + VpcCidrBlock: + Type: String + Default: 10.1.0.0/16 + Description: CIDR Block for the Virtual Private Cloud + PublicSubnetCidrBlock: + Type: String + Default: 10.1.0.0/24 + Description: CIDR Block for the Public Subnet Segment + PrivateSubnetCidrBlock: + Type: String + Default: 10.1.1.0/24 + Description: CIDR Block for the Private Subnet Segment + KafkaClusterPrimaryIp: + Type: String + Description: Kafka Cluster Public IP + KafkaClusterSecondaryIp: + Type: String + Description: Kafka Cluster Public IP + TopicName: + Type: String + Description: Kafka Cluster Topic Name + +Resources: + +# VPC. This is the main component of our template, within the VPC we will be defining the subnets and +# connect to it all components required to create the stack. +# +# https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-vpc.html + + VPC: + Type: AWS::EC2::VPC + Properties: + CidrBlock: !Ref VpcCidrBlock + EnableDnsSupport: true + EnableDnsHostnames: true + InstanceTenancy: default + Tags: + - Key: Name + Value: !Join ['', [!Ref "AWS::StackName", "-EventBridge-Pipes-VPC"]] + +# Subnets Definition: in this section we define the two subnets, one public and one private +# and also create an ElasticIP and a NAT gateway to attach it to the private network +# so that the Pipe can poll the Kafka Cluster. +# +# https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-subnet.html + + PublicSubnet: + Type: AWS::EC2::Subnet + Properties: + VpcId: !Ref VPC + CidrBlock: !Ref PublicSubnetCidrBlock + MapPublicIpOnLaunch: true + Tags: + - Key: Name + Value: !Join ['', [!Ref "AWS::StackName", "-EventBridge-Pipes-Pub-SubNet"]] + PrivateSubnet: + Type: AWS::EC2::Subnet + DeletionPolicy: Retain + Properties: + VpcId: !Ref VPC + CidrBlock: !Ref PrivateSubnetCidrBlock + MapPublicIpOnLaunch: false + Tags: + - Key: Name + Value: !Join ['', [!Ref "AWS::StackName", "-EventBridge-Pipes-Priv-SubNet"]] + +# Internet Gateway. This component provides internet connectivity to the internet both ways (in and out). +# The gateway needs to be attached to a VPC. Therefore it is important to explicitly attach it to the +# VPC otherwise it won't be connected. +# +# https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-internetgateway.html +# https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-vpc-gateway-attachment.html + + InternetGateway: + Type: AWS::EC2::InternetGateway + DependsOn: VPC + Properties: + Tags: + - Key: Name + Value: !Join ['', [!Ref "AWS::StackName", "-EventBridge-Pipes-IGW"]] + AttachGateway: + Type: AWS::EC2::VPCGatewayAttachment + Properties: + VpcId: !Ref VPC + InternetGatewayId: !Ref InternetGateway + +# ElasticIP definition, we don't need to add many attributes here, simply let Amazon do the +# work for you, we will only need to reference the elastic IP in the gateway. +# +# https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-elasticip.html + + ElasticIPAddress: + Type: AWS::EC2::EIP + Properties: + Domain: vpc + Tags: + - Key: Name + Value: !Join ['', [!Ref "AWS::StackName", "-EventBridge-Pipes-NGW-EIP"]] + +# NAT Gateway. This component requires two references an allocation fo the elastic IP address to +# use and a designated subnet to be attached to. +# +# https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-natgateway.html + + NatGateway: + DependsOn: PublicSubnet + Type: AWS::EC2::NatGateway + Properties: + AllocationId: + Fn::GetAtt: + - ElasticIPAddress + - AllocationId + SubnetId: !Ref PublicSubnet + Tags: + - Key: Name + Value: !Join ['', [!Ref "AWS::StackName", "-EventBridge-Pipes-NGW"]] + +# Routing Tables. The route tables are the logical components that enable the routing of traffic +# within the network, subnets included. +# +# Used Definitions: +# +# https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-route-table.html +# https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-route.html +# +# Public Route Table. This is route table contains the routes for the public subnet. + + PublicRouteTable: + Type: AWS::EC2::RouteTable + Properties: + VpcId: !Ref VPC + Tags: + - Key: Name + Value: !Join ['', [!Ref "AWS::StackName", "-EventBridge-Pipes-Pub-RTB"]] + +# This route enables the in-out traffic from the VPC to the internet via the specified Internet Gateway. + + RouteTrafficToIGW: + Type: AWS::EC2::Route + DependsOn: + - InternetGateway + - AttachGateway + Properties: + RouteTableId: !Ref PublicRouteTable + DestinationCidrBlock: 0.0.0.0/0 + GatewayId: !Ref InternetGateway + +# Private Route Table. This is the route table that contains the routes for the private subnet. + + PrivateRouteTable: + Type: AWS::EC2::RouteTable + Properties: + VpcId: !Ref VPC + Tags: + - Key: Name + Value: !Join ['', [!Ref "AWS::StackName", "-EventBridge-Pipes-Priv-RTB"]] + +# This route enables the outgoing internet traffic from the private subnet to the Internet and +# it is used for patches downloading and access to external services as needed. + + RouteTrafficToNGW: + Type: AWS::EC2::Route + Properties: + RouteTableId: !Ref PrivateRouteTable + DestinationCidrBlock: 0.0.0.0/0 + NatGatewayId: !Ref NatGateway + +# Route Table Associations. The associations are necessary to map the route tables to subnets +# so that traffic can be effectively routed through the components hosted in these networks +# and desired network destinations. +# +# Definitions: +# +# https://docs.amazonaws.cn/en_us/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-subnet-route-table-assoc.html + + PublicSubnetRouteTableAssociation: + Type: AWS::EC2::SubnetRouteTableAssociation + Properties: + SubnetId: !Ref PublicSubnet + RouteTableId: !Ref PublicRouteTable + PrivateSubnetRouteTableAssociation: + Type: AWS::EC2::SubnetRouteTableAssociation + Properties: + SubnetId: !Ref PrivateSubnet + RouteTableId: !Ref PrivateRouteTable + +# Security Group definition. +# We define the rules with the minimum inbound/outbound rules +# More info - https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-kafka.html#pipes-kafka-vpc-config +# +# https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-securitygroup.html + + SecurityGroup: + Type: AWS::EC2::SecurityGroup + DependsOn: VPC + DeletionPolicy: Retain + Properties: + GroupDescription: Allow EventBridge access to the Amazon Virtual Private Cloud (Amazon VPC) resources associated with your Apache Kafka brokers + SecurityGroupEgress: + - IpProtocol: tcp + FromPort: 9092 + ToPort: 9092 + CidrIp: 0.0.0.0/0 + - IpProtocol: tcp + FromPort: 443 + ToPort: 443 + CidrIp: 0.0.0.0/0 + SecurityGroupIngress: + - IpProtocol: tcp + FromPort: 9092 + ToPort: 9092 + CidrIp: 0.0.0.0/0 + Tags: + - Key: Name + Value: !Join ['', [!Ref "AWS::StackName", "-EventBridge-Pipes-Security-Group"]] + VpcId: !Ref VPC + +# CloudWatch Log Group definition. +# This Log Group is used as the target of the Pipe to receive and store polled events. +# +# https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-logs-loggroup.html + + LogGroup: + Type: AWS::Logs::LogGroup + Properties: + Tags: + - Key: Name + Value: !Join ['', [!Ref "AWS::StackName", "-EventBridge-Pipes-Log-Group"]] + +# EventBirdge Pipe IAM Role Policy definition +# +# https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-iam-managedpolicy.html + + PipePolicy: + Type: AWS::IAM::ManagedPolicy + DependsOn: + - LogGroup + Properties: + ManagedPolicyName: !Join ['', [!Ref "AWS::StackName", "-LogsPipeTargetPolciyTemplate"]] + PolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: Allow + Action: + - 'logs:CreateLogStream' + - 'logs:PutLogEvents' + Resource: !GetAtt + - LogGroup + - Arn + - Effect: Allow + Action: + - 'ec2:DescribeNetworkInterfaces' + - 'ec2:DescribeSubnets' + - 'ec2:DescribeSecurityGroups' + - 'ec2:DescribeVpcs' + Resource: '*' + - Effect: Allow + Action: + - 'ec2:CreateNetworkInterface' + - 'ec2:DeleteNetworkInterface' + Resource: '*' + Condition: + StringEqualsIfExists: + 'ec2:SubnetID': + - !Ref PrivateSubnet + +# EventBirdge Pipe IAM Role definition +# IAM role used to allow Pipe Execution +# +# https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-iam-role.html + + PipesIAMRole: + Type: AWS::IAM::Role + DependsOn: + - PipePolicy + Properties: + AssumeRolePolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: Allow + Principal: + Service: + - pipes.amazonaws.com + Action: + - 'sts:AssumeRole' + Description: Role to provide pipes access to SMK and Logs + ManagedPolicyArns: + - !Ref PipePolicy + +# EventBirdge Pipe definition +# +# https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-pipes-pipe.html + + Pipe: + Type: AWS::Pipes::Pipe + DependsOn: + - PrivateSubnet + - PipesIAMRole + - LogGroup + - SecurityGroup + Properties: + RoleArn: !GetAtt + - PipesIAMRole + - Arn + Name: !Join ['', [!Ref "AWS::StackName", "-EventBridge-Pipe"]] + Source: !Join ['', ["smk://", !Ref "KafkaClusterPrimaryIp", ":9092"]] + SourceParameters: + SelfManagedKafkaParameters: + AdditionalBootstrapServers: + - !Join ['', [!Ref "KafkaClusterSecondaryIp", ":9092"]] + TopicName: !Ref TopicName + StartingPosition: LATEST + Vpc: + SecurityGroup: + - !Ref SecurityGroup + Subnets: + - !Ref PrivateSubnet + Target: !GetAtt + - LogGroup + - Arn + +# Outputs define values exported by the template that can be used and captured +# by other templates. +# +# https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/outputs-section-structure.html + +Outputs: + VpcId: + Description: Unique identifier of the VPC + Value: !Ref VPC + PublicSubnetId: + Description: Unique identifier of the Public Subnet segment of the VPC + Value: !Ref PublicSubnet + PrivateSubnetId: + Description: Unique identifier of the Private Subnet segment of the VPC + Value: !Ref PrivateSubnet + Pipe: + Description: Unique identifier of the EventBridge Pipe + Value: !Ref Pipe + LogGroup: + Description: Unique identifier of the CloudWatch Log Group + Value: !Ref LogGroup \ No newline at end of file