|
| 1 | +# :simple-owasp: Contributing to OWASP Amass |
1 | 2 |
|
| 3 | +Welcome to the OWASP Amass Project! This guide covers |
| 4 | +everything you need to contribute — whether you're filing |
| 5 | +a bug, writing documentation, developing a data source |
| 6 | +plugin, or working on the engine internals. |
| 7 | + |
| 8 | +## :material-forum: Getting Involved |
| 9 | + |
| 10 | +Start by joining the community: |
| 11 | + |
| 12 | +- **Discord:** <https://discord.gg/ANTyEDUXt5> — the |
| 13 | + primary place for discussion and questions |
| 14 | +- **GitHub Issues:** |
| 15 | + <https://github.com/owasp-amass/amass/issues> — |
| 16 | + check what needs help |
| 17 | +- **Repos:** All under <https://github.com/owasp-amass>: |
| 18 | + `amass`, `asset-db`, `open-asset-model`, `docs` |
| 19 | + |
| 20 | +## :material-source-branch: Git Workflow |
| 21 | + |
| 22 | +### Forking |
| 23 | + |
| 24 | +Go requires code to live at its original import path. |
| 25 | +Use `git remote` to set up your fork against the |
| 26 | +original repository: |
| 27 | + |
| 28 | +```bash |
| 29 | +# 1. Fork on GitHub, then from your local clone: |
| 30 | +git remote rename origin upstream |
| 31 | +git remote add origin \ |
| 32 | + git@github.com:YOUR_USERNAME/amass.git |
| 33 | + |
| 34 | +# 2. Fetch updates from upstream |
| 35 | +git fetch upstream |
| 36 | +git rebase upstream/develop |
| 37 | +``` |
| 38 | + |
| 39 | +### Branch and PR Rules |
| 40 | + |
| 41 | +- Branch from the tip of `develop`, **not** `main` |
| 42 | +- Do not `--force` push onto `develop` (only allowed |
| 43 | + when reverting a broken commit) |
| 44 | +- Rebase your branch on top of `develop` before |
| 45 | + opening a PR: `git rebase upstream/develop` |
| 46 | +- All PRs target `develop` — do not open PRs |
| 47 | + against `main` |
| 48 | + |
| 49 | +## :material-code-tags: Code Standards |
| 50 | + |
| 51 | +```bash |
| 52 | +# Required before every commit |
| 53 | +gofmt -w . |
| 54 | + |
| 55 | +# Lint (same flags as CI) |
| 56 | +golangci-lint run --timeout=60m ./... |
| 57 | + |
| 58 | +# Build (CGO must be disabled) |
| 59 | +CGO_ENABLED=0 go install -v ./... |
| 60 | + |
| 61 | +# Test |
| 62 | +CGO_ENABLED=0 go test -v ./... |
| 63 | +CGO_ENABLED=0 go test -v \ |
| 64 | + -run TestName ./path/to/package/... |
| 65 | +``` |
| 66 | + |
| 67 | +Go version: **1.26.0**. `CGO_ENABLED=0` is required |
| 68 | +for all builds and tests. |
| 69 | + |
| 70 | +## :material-engine-outline: Engine Architecture |
| 71 | + |
| 72 | +Understanding how the engine works is essential for |
| 73 | +plugin development and engine contributions. |
| 74 | + |
| 75 | +```mermaid |
| 76 | +flowchart TD |
| 77 | + A["REST API\n:4000\nPOST /sessions/{id}/assets"] --> B |
| 78 | +
|
| 79 | + B["Session Backlog\nSQLite\nclaim/ack/lease"] |
| 80 | +
|
| 81 | + B -->|"ClaimNext(assetType)"| C |
| 82 | +
|
| 83 | + C["Dispatcher\nrunEvents loop\nauto-scales / 5 s"] |
| 84 | +
|
| 85 | + C -->|"consistent hash ring"| D |
| 86 | +
|
| 87 | + D["PipelinePool\n1 pool per asset type\nFQDN/IP: 4-32\nothers: 1-4"] |
| 88 | +
|
| 89 | + D --> E["Pipeline Stages\nPos 1-50\nFIFO/DynPool\nor Parallel"] |
| 90 | +
|
| 91 | + E --> F["Handler Callback\nfunc(*Event) error"] |
| 92 | +
|
| 93 | + F -->|"DispatchEvent\nnew assets"| B |
| 94 | +``` |
| 95 | + |
| 96 | +### Key Components |
| 97 | + |
| 98 | +**SessionManager** — each enumeration runs in an |
| 99 | +isolated `Session`. Each session has its own database |
| 100 | +connection, `Backlog`, `Scope`, structured logger, |
| 101 | +and temp directory. |
| 102 | + |
| 103 | +**Registry** — plugins register `Handler` structs at |
| 104 | +startup, keyed by `EventType` (asset type) and |
| 105 | +`Position` (priority 1–50). The registry builds |
| 106 | +pipelines from these registrations. |
| 107 | + |
| 108 | +**Dispatcher** — claims events from session backlogs, |
| 109 | +routes them to asset-type-specific `PipelinePool` |
| 110 | +instances via a consistent hash ring, and auto-scales |
| 111 | +pool size every 5 seconds based on backlog depth. |
| 112 | + |
| 113 | +**PipelinePool** — per-asset-type pool. A consistent |
| 114 | +hash ring with 50 slots routes events to specific |
| 115 | +instances (ensuring locality). FQDN and IPAddress |
| 116 | +pools scale between 4 and 32 instances; all other |
| 117 | +types scale between 1 and 4. Hard cap is |
| 118 | +`maxInstances × 2`. |
| 119 | + |
| 120 | +**Backlog** — durable SQLite-backed work queue with |
| 121 | +claim/ack/lease semantics. Items progress through: |
| 122 | +`queued → leased (in-flight) → done`. Expired leases |
| 123 | +automatically return items to `queued`, enabling |
| 124 | +recovery from failures. |
| 125 | + |
| 126 | +### Handler Instance Constants |
| 127 | + |
| 128 | +Defined in `engine/plugins/support/support.go`: |
| 129 | + |
| 130 | +| Constant | Value | Typical Use | |
| 131 | +| :--- | :---: | :--- | |
| 132 | +| `MinHandlerInstances` | 4 | Low-volume handlers | |
| 133 | +| `MidHandlerInstances` | 16 | Standard API sources | |
| 134 | +| `HighHandlerInstances` | 32 | High-throughput | |
| 135 | +| `MaxHandlerInstances` | 64 | DNS apex, bulk res. | |
| 136 | + |
| 137 | +## :material-puzzle-outline: Writing a Plugin |
| 138 | + |
| 139 | +Plugins live in `engine/plugins/` organized by |
| 140 | +category (`api/`, `dns/`, `scrape/`, `whois/`, |
| 141 | +`enrich/`, `brute/`, `horizontals/`, |
| 142 | +`service_discovery/`). |
| 143 | + |
| 144 | +### Step 1 — Implement the Plugin Interface |
| 145 | + |
| 146 | +```go |
| 147 | +// engine/types/registry.go |
| 148 | +type Plugin interface { |
| 149 | + Name() string |
| 150 | + // Called once at startup; register handlers here |
| 151 | + Start(r Registry) error |
| 152 | + // Clean up goroutines and channels |
| 153 | + Stop() |
| 154 | +} |
| 155 | +``` |
| 156 | + |
| 157 | +### Step 2 — Register Handlers in Start |
| 158 | + |
| 159 | +```go |
| 160 | +func (p *myPlugin) Start(r et.Registry) error { |
| 161 | + p.log = r.Log().WithGroup("plugin"). |
| 162 | + With("name", p.name) |
| 163 | + |
| 164 | + if err := r.RegisterHandler(&et.Handler{ |
| 165 | + Plugin: p, |
| 166 | + Name: "MyPlugin-Handler", |
| 167 | + // Priority 1-50; lower runs first |
| 168 | + Position: 21, |
| 169 | + // Max concurrency (4/16/32/64) |
| 170 | + MaxInstances: support.MidHandlerInstances, |
| 171 | + // Asset type that triggers this handler |
| 172 | + EventType: oam.FQDN, |
| 173 | + // Asset types this handler may produce |
| 174 | + Transforms: []string{ |
| 175 | + string(oam.FQDN), |
| 176 | + }, |
| 177 | + Callback: p.check, |
| 178 | + }); err != nil { |
| 179 | + return err |
| 180 | + } |
| 181 | + |
| 182 | + p.log.Info("Plugin started") |
| 183 | + return nil |
| 184 | +} |
| 185 | +``` |
| 186 | + |
| 187 | +Handler field reference: |
| 188 | + |
| 189 | +| Field | Description | |
| 190 | +| :--- | :--- | |
| 191 | +| `Position` | Priority 1–50; lower runs first | |
| 192 | +| `Exclusive` | If `true`, sole handler at this pos | |
| 193 | +| `MaxInstances` | Enables `DynamicPool` when `> 1` | |
| 194 | +| `Transforms` | Asset types you will dispatch | |
| 195 | +| `EventType` | OAM asset type triggering handler | |
| 196 | +| `Callback` | `func(*Event) error` — the logic | |
| 197 | + |
| 198 | +### Step 3 — Implement the Callback |
| 199 | + |
| 200 | +The pattern below is drawn from |
| 201 | +`engine/plugins/api/chaos.go`, the clearest |
| 202 | +minimal example: |
| 203 | + |
| 204 | +```go |
| 205 | +func (p *myPlugin) check(e *et.Event) error { |
| 206 | + // 1. Type-assert the triggering asset |
| 207 | + fqdn, ok := e.Entity.Asset.(*oamdns.FQDN) |
| 208 | + if !ok { |
| 209 | + return errors.New( |
| 210 | + "failed to extract the FQDN asset") |
| 211 | + } |
| 212 | + |
| 213 | + // 2. Confirm the SLD is in scope |
| 214 | + if !support.HasSLDInScope(e) { |
| 215 | + return nil |
| 216 | + } |
| 217 | + |
| 218 | + // 3. Confirm credentials are present |
| 219 | + ds := e.Session.Config(). |
| 220 | + GetDataSourceConfig(p.name) |
| 221 | + if ds == nil || len(ds.Creds) == 0 { |
| 222 | + return nil |
| 223 | + } |
| 224 | + |
| 225 | + // 4. Check TTL — skip if processed recently |
| 226 | + since, err := support.TTLStartTime( |
| 227 | + e.Session.Config(), |
| 228 | + string(oam.FQDN), |
| 229 | + string(oam.FQDN), |
| 230 | + p.name, |
| 231 | + ) |
| 232 | + if err != nil { |
| 233 | + return err |
| 234 | + } |
| 235 | + if support.AssetMonitoredWithinTTL( |
| 236 | + e.Session, e.Entity, p.source, since) { |
| 237 | + return nil |
| 238 | + } |
| 239 | + |
| 240 | + // 5. Query, store, dispatch |
| 241 | + names := p.query(e, fqdn.Name) |
| 242 | + support.MarkAssetMonitored( |
| 243 | + e.Session, e.Entity, p.source) |
| 244 | + |
| 245 | + if len(names) > 0 { |
| 246 | + entities := support.StoreFQDNsWithSource( |
| 247 | + e.Session, names, p.source, |
| 248 | + p.name, p.name+"-Handler") |
| 249 | + support.ProcessFQDNsWithSource( |
| 250 | + e, entities, p.source) |
| 251 | + } |
| 252 | + return nil |
| 253 | +} |
| 254 | +``` |
| 255 | + |
| 256 | +Common support utilities |
| 257 | +(`engine/plugins/support/support.go`): |
| 258 | + |
| 259 | +| Function | Purpose | |
| 260 | +| :--- | :--- | |
| 261 | +| `HasSLDInScope(e)` | True if SLD is in scope | |
| 262 | +| `TTLStartTime(...)` | Cache window cutoff time | |
| 263 | +| `AssetMonitoredWithinTTL(...)` | Already done? | |
| 264 | +| `MarkAssetMonitored(...)` | Record processing | |
| 265 | +| `StoreFQDNsWithSource(...)` | Bulk-create FQDNs | |
| 266 | +| `ProcessFQDNsWithSource(...)` | Dispatch events | |
| 267 | + |
| 268 | +### Step 4 — Register in load.go |
| 269 | + |
| 270 | +Add your constructor to the `pluginNewFuncs` slice |
| 271 | +in `engine/plugins/load.go`: |
| 272 | + |
| 273 | +```go |
| 274 | +var pluginNewFuncs = []func() et.Plugin{ |
| 275 | + // ...existing plugins... |
| 276 | + mypackage.NewMyPlugin, |
| 277 | +} |
| 278 | +``` |
| 279 | + |
| 280 | +### Step 5 — Add Credentials Template |
| 281 | + |
| 282 | +If your plugin requires credentials, add a |
| 283 | +commented-out entry to `resources/datasources.yaml`: |
| 284 | + |
| 285 | +```yaml |
| 286 | +# - name: MyDataSource |
| 287 | +# ttl: 1440 |
| 288 | +# creds: |
| 289 | +# account: |
| 290 | +# apikey: null |
| 291 | +``` |
| 292 | + |
| 293 | +## :material-source-pull: Contributing to Other Repos |
| 294 | + |
| 295 | +The same git workflow applies to all repos under the |
| 296 | +`owasp-amass` organization: |
| 297 | + |
| 298 | +- **`asset-db`** — database schema, repository |
| 299 | + interface, and migrations |
| 300 | +- **`open-asset-model`** — asset, relation, and |
| 301 | + property type definitions in Go |
| 302 | +- **`docs`** — this documentation site; |
| 303 | + `mkdocs build --strict` must pass before |
| 304 | + opening a PR |
| 305 | + |
| 306 | +--- |
| 307 | + |
| 308 | +*© 2025 Jeff Foley — Licensed under Apache 2.0.* |
0 commit comments