Worker E2E: compile and run API ad Worker binaries #1347
Conversation
michaljurecko
force-pushed
the
michaljurecko-PSGO-216-e2e-test-binaries
branch
2 times, most recently
from
8a04405 to
c6624f9
Compare
michaljurecko
force-pushed
the
michaljurecko-PSGO-216-e2e-test-binaries
branch
from
c6624f9 to
ae791e3
Compare
| Inspired by [PhpUnit](https://phpunit.readthedocs.io/en/9.5/assertions.html#assertstringmatchesformat). | ||
| Inspired by [PhpUnit](https://docs.phpunit.de/en/10.0/assertions.html#assertstringmatchesformat). |
There was a problem hiding this comment.
Fixnuty link (404) v Markdown subore.
| # Format code, gofumpt and gci partially overlap, it is needed to run them separately | ||
| # https://github.com/golangci/golangci-lint/issues/1490 | ||
| echo "Running gofumpt ..." | ||
| gofumpt -w ./cmd ./internal ./pkg ./test | ||
| echo "Running gci ..." | ||
| gci write --skip-generated -s standard -s default -s "prefix(github.com/keboola/keboola-as-code)" ./cmd ./internal ./pkg ./test | ||
|
|
There was a problem hiding this comment.
Vramci golangci-lint --fix (nizsie) sa spustaju okrem ineho aj 2 nastroje gci a gofumpt, ktore sa ciastocne prekryvaju vo svojej funkcionalite - oba zoradzuju/formatuju importy v go suboroch.
gci je nastroj iba na tento ucel, gofumpt ma ine uzitocne funkcie, ale zial neda sa tam to formatovane vypnut (aby ho robil iba gci).
Vramci golangci-lint bezia vsetky linters paralelne a niekedy sa stava, ze tieto 2, ak sa pouzije aj --fix, rozbiju go subor.
Viac na uvedenom linku - komentar v kode.
Jednoduche riesenie je, spustit ich samostatne pred golangci-lint, kym to tam nejak nevyriesia.
(ked nechcem prist o to, ze kod v tomto repe ma striktne definovany format).
| } | ||
| }) | ||
| return telemetry.WrapDD(tracerProvider.Tracer("")), nil | ||
| if cfg.DatadogEnabled { |
There was a problem hiding this comment.
Chybali mi tu tieto podmienky, vysledkom boli warnings v testoch, ze sa nedari pripojit k DD agent.
| Attribute("instanceId", String, "ID of the created/updated template instance.") | ||
| Attribute("instanceId", InstanceID, "ID of the created/updated template instance.") |
There was a problem hiding this comment.
Drobny fix, na tieto hodnoty ^^^ su definovane vlastne typy, aby sa predislo preklepom, tu chybali.
Kod nizsie je vygenerovany.
| TelemetryOptions: []middleware.OTELOption{ | ||
| MiddlewareOptions: []middleware.Option{ |
There was a problem hiding this comment.
Nie su to options iba pre OTEL, ale pre vsetky middlewares. Vysvetlene nizsie.
| {"1", `{"key": "payload001"}`, `{"Accept-Encoding":"gzip, br","Content-Type":"application/json","User-Agent":"keboola-go-client"}`}, | ||
| {"2", `{"key": "payload002"}`, `{"Accept-Encoding":"gzip, br","Content-Type":"application/json","User-Agent":"keboola-go-client"}`}, |
There was a problem hiding this comment.
Pouziva sa go-client/request, takze je tam tych hlaviciek viac.
| [worker-node-%s][service][conditions]INFO checked "2" opened slices | ||
| [worker-node-%s][service][conditions]INFO checked "2" opened slices |
There was a problem hiding this comment.
Drobna nekompatibilita medzi Debug a Prod logs.
| ts.t.Logf("-------------------------") | ||
|
|
||
| newTableID := ts.export1.Mapping.TableID + "-v2" | ||
| ts.export1 = ts.UpdateExport(ts.export1.ReceiverID, ts.export1.ID, &server.MappingRequestBody{ |
There was a problem hiding this comment.
UpdateExport metoda ma uz v sebe kontrolu err aj cakanie na dobehnutie task.
| assert.Equal(ts.t, `no free space in the buffer: receiver "my-receiver" has "200.1 KB" buffered for upload, limit is "100.0 KB"`, err.Error()) | ||
| assert.Equal(ts.t, `No free space in the buffer: receiver "my-receiver" has "200.1 KB" buffered for upload, limit is "100.0 KB".`, err.Error()) |
There was a problem hiding this comment.
Chyby sa pred API response konvertuju na vety.
| "exportId": "my-export-1", | ||
| "name": "my-export-1", | ||
| "name": "My Export 1", |
There was a problem hiding this comment.
V testoch som zmenil name aby nebolo rovnake ako id, tak to menim aj v tomto snapshote/ocakavanom stave.
| github.com/keboola/go-client v1.16.1 | ||
| github.com/keboola/go-client v1.16.2-0.20230606141226-ac73db20e551 |
There was a problem hiding this comment.
TO už teď možeš změnit :) (nevím jestli to neděláš v dalším PR)
| } | ||
|
|
||
| // returns true if the request should NOT be logged/traced. | ||
| func isRequestIgnored(req *http.Request) bool { |
There was a problem hiding this comment.
Tady sem si akorát všiml, že to začíná malý i tady aj jinde používáš velké písmena když tu fci používáš jinde v package.
There was a problem hiding this comment.
No velke pismeno zanamena ze je to public, tj. da sa pouzit MIMO package, vramci pkg to nehra rolu.
Na isRequestIgnored asi nemam zatial use-case, aby sa to pouzivalo mimo middleware pkg, ale mohlo by to byt aj public.
Ina situacia je, ze samotny typ je private, tam zvyknem davat metody Public, kedze je uz typ private, a ak je private aj jeho konstruktor funkcia newPrivate() *private, tak sa neda vytvorit mimo package.
type private struct {
}
func (p *private) PublicMethod() {
}
Je to ^^^ vyhoda v pripade refactoringu, ked zistim, ze private utilitku potrebujem vo viacerych packages, tak zmenim iba private -> Private a nemusi prepisovat metody.
^^^^ neviem ci je ten moj koment pochopitelny :) tu je k tomu nieco viac, .... kedy to je a nie je rozdiel: https://stackoverflow.com/questions/37952354/golang-public-method-to-private-struct-is-this-have-any-use-case
jj, v case komentovania to nebolo mergnute, ...upravim v PR |
Jira: https://keboola.atlassian.net/browse/PSGO-216
Intro
V Buffer Service mame 2 typy E2E testov:
requests, kontroluju saresponsesa stavetcddatabazy.V tomto PR sa namiesto testovania Worker a API nodes v Go kode spustia priamo binarky:
etcd.table previewAPI call.Motivacia:
dependencies, ktoru sposob testovania bez spustenia binarky neodhalil a spadlo to az na produkcii.Test Parts
Jedna sa prakticky o jeden test, ktory ma viacero casti:
keboola-as-code/test/buffer/worker/worker_test.go
Lines 22 to 44 in ae791e3
V teste sa vacsinou:
keboola-as-code/test/buffer/worker/002_slice_upload_test.go
Lines 18 to 21 in ae791e3
logs:keboola-as-code/test/buffer/worker/002_slice_upload_test.go
Lines 23 to 30 in ae791e3
etcd+ ze sa nezalogovala ziadna chyba/warning.keboola-as-code/test/buffer/worker/002_slice_upload_test.go
Lines 89 to 94 in ae791e3
etcd expected state
Vramci testu sa do
etcdulozi relativne vela záznamov, no kazdá časť testu mení iba pár záznamov.Preto sú snapshoty etcd rozdelené do viacerých súborov:
A dany snapshot sa posklada zo spolocnych casti, napr.:
keboola-as-code/test/buffer/worker/expected-etcd-state/003-file-import.txt
Lines 1 to 26 in 367142f
.out.outadresar obsahuje reálne výstupy počas testu, ktoré sa dajú použiť na debugovanie:Test Log
Nizsie je log testu, bezi
~80skym sa vsetko nasetupuje (Storage API File Resources, ...), uploadne a importuje do tabulky.>=5zaznamou trigruje upload (v teste je ich6)>=10záznamou trigruje import (v teste je ich10), teda 2 slicesLog testu
go test -v -count 1 -race ./test/buffer/worker/...`(expand)(okomentoavne po commitoch)