Collaborate on prototype of in-workspace image build

[kylos101]

Is your feature request related to a problem? Please describe

IDE team is working #7671, and has started a related prototype to do in-workspace image builds in https://github.com/gitpod-io/gitpod/tree/pd/workspacekit, the proposed design is outlined here. The intent behind https://github.com/gitpod-io/gitpod/tree/pd/workspacekit is not to create a branch that gets merged back to main. However, that could change? It depends how "good" the experience is.

Describe what you'd like

@utam0k , when you return from vacation, can you talk with @iQQBot? He might need help from Workspace Team, he might not. 😄

Are they happy with the new experience for building images in-workspace? Would they like you to review any of the code in the branch, to help improve the experience for the branch?

Run two supervisors in a workspace pod at the same time

We need to create two IP addresses on ring1 for each of supervisor so as to go for this design

This can be done by editing the nsinsider function we have been using.

gitpod/components/ws-daemon/nsinsider/main.go

Lines 264 to 427 in 0c84d03

	Name: "setup-pair-veths",
	Usage: "set up a pair of veths",
	Flags: []cli.Flag{
	&cli.IntFlag{
	Name: "target-pid",
	Required: true,
	},
	},
	Action: func(c *cli.Context) error {
	containerIf, vethIf, cethIf := "eth0", "veth0", "ceth0"
	mask := net.IPv4Mask(255, 255, 255, 0)
	vethIp := net.IPNet{
	IP: net.IPv4(10, 0, 5, 1),
	Mask: mask,
	}
	cethIp := net.IPNet{
	IP: net.IPv4(10, 0, 5, 2),
	Mask: mask,
	}
	masqueradeAddr := net.IPNet{
	IP: vethIp.IP.Mask(mask),
	Mask: mask,
	}
	targetPid := c.Int("target-pid")
	eth0, err := netlink.LinkByName(containerIf)
	if err != nil {
	return xerrors.Errorf("cannot get container network device %s: %w", containerIf, err)
	}
	veth := &netlink.Veth{
	LinkAttrs: netlink.LinkAttrs{
	Name: vethIf,
	Flags: net.FlagUp,
	MTU: eth0.Attrs().MTU,
	},
	PeerName: cethIf,
	PeerNamespace: netlink.NsPid(targetPid),
	}
	if err := netlink.LinkAdd(veth); err != nil {
	return xerrors.Errorf("link %q-%q netns failed: %v", vethIf, cethIf, err)
	}
	vethLink, err := netlink.LinkByName(vethIf)
	if err != nil {
	return xerrors.Errorf("cannot found %q netns failed: %v", vethIf, err)
	}
	if err := netlink.AddrAdd(vethLink, &netlink.Addr{IPNet: &vethIp}); err != nil {
	return xerrors.Errorf("failed to add IP address to %q: %v", vethIf, err)
	}
	if err := netlink.LinkSetUp(vethLink); err != nil {
	return xerrors.Errorf("failed to enable %q: %v", vethIf, err)
	}
	nc := &nftables.Conn{}
	nat := nc.AddTable(&nftables.Table{
	Family: nftables.TableFamilyIPv4,
	Name: "nat",
	})
	postrouting := nc.AddChain(&nftables.Chain{
	Name: "postrouting",
	Hooknum: nftables.ChainHookPostrouting,
	Priority: nftables.ChainPriorityNATSource,
	Table: nat,
	Type: nftables.ChainTypeNAT,
	})
	// ip saddr 10.0.5.0/24 oifname "eth0" masquerade
	nc.AddRule(&nftables.Rule{
	Table: nat,
	Chain: postrouting,
	Exprs: []expr.Any{
	&expr.Payload{
	DestRegister: 1,
	Base: expr.PayloadBaseNetworkHeader,
	Offset: 12,
	Len: net.IPv4len,
	},
	&expr.Bitwise{
	SourceRegister: 1,
	DestRegister: 1,
	Len: net.IPv4len,
	Mask: masqueradeAddr.Mask,
	Xor: net.IPv4Mask(0, 0, 0, 0),
	},
	&expr.Cmp{
	Op: expr.CmpOpEq,
	Register: 1,
	Data: masqueradeAddr.IP.To4(),
	},
	&expr.Meta{Key: expr.MetaKeyOIFNAME, Register: 1},
	&expr.Cmp{
	Op: expr.CmpOpEq,
	Register: 1,
	Data: []byte(fmt.Sprintf("%s\x00", containerIf)),
	},
	&expr.Masq{},
	},
	})
	prerouting := nc.AddChain(&nftables.Chain{
	Name: "prerouting",
	Hooknum: nftables.ChainHookPrerouting,
	Priority: nftables.ChainPriorityNATDest,
	Table: nat,
	Type: nftables.ChainTypeNAT,
	})
	// iif $containerIf tcp dport 1-65535 dnat to $cethIp:tcp dport
	nc.AddRule(&nftables.Rule{
	Table: nat,
	Chain: prerouting,
	Exprs: []expr.Any{
	&expr.Meta{Key: expr.MetaKeyIIFNAME, Register: 1},
	&expr.Cmp{
	Op: expr.CmpOpEq,
	Register: 1,
	Data: []byte(containerIf + "\x00"),
	},
	&expr.Meta{Key: expr.MetaKeyL4PROTO, Register: 1},
	&expr.Cmp{
	Op: expr.CmpOpEq,
	Register: 1,
	Data: []byte{unix.IPPROTO_TCP},
	},
	&expr.Payload{
	DestRegister: 1,
	Base: expr.PayloadBaseTransportHeader,
	Offset: 2,
	Len: 2,
	},
	&expr.Cmp{
	Op: expr.CmpOpGte,
	Register: 1,
	Data: []byte{0x00, 0x01},
	},
	&expr.Cmp{
	Op: expr.CmpOpLte,
	Register: 1,
	Data: []byte{0xff, 0xff},
	},
	&expr.Immediate{
	Register: 2,
	Data: cethIp.IP.To4(),
	},
	&expr.NAT{
	Type: expr.NATTypeDestNAT,
	Family: unix.NFPROTO_IPV4,
	RegAddrMin: 2,
	RegProtoMin: 1,
	},
	},
	})
	if err := nc.Flush(); err != nil {
	return xerrors.Errorf("failed to apply nftables: %v", err)
	}
	return nil
	},

                                      Pod Network Namespace(ring1)
┌────────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                                                                                    │
│            Workspace Network Namesapce                         Debug Network Namesapce             │
│ ┌────────────────────────────────────────────┐       ┌─────────────────────────────────────────┐   │
│ │                                            │       │                                         │   │
│ │              default via veth0             │       │           default via veth1             │   │
│ │                                            │       │                                         │   │
│ │                                            │       │                                         │   │
│ │     ┌──────┐  ┌──────────────┐             │       │  ┌──────┐  ┌──────────────┐             │   │
│ │     │  lo  │  │    ceth0     │ 10.0.5.1/24 │       │  │  lo  │  │    ceth1     │ 10.0.6.2/24 │   │
│ │     └──────┘  └──▲────────┬──┘             │       │  └──────┘  └──▲────────┬──┘             │   │
│ │                  │        │                │       │               │        │                │   │
│ └──────────────────┼────────┼────────────────┘       └───────────────┼────────┼────────────────┘   │
│                    │        │                                        │        │                    │
│                 ┌──┴────────▼──┐                                  ┌──┴────────▼──┐                 │
│   ┌───────────► │    veth0     │ 10.0.2.1/24          10.0.2.2/24 │    veth1     │ ◄────────────┐  │
│   │             └───────────┬──┘                                  └───────────┬──┘              │  │
│   │                         │                                                 │                 │  │
│   │          ┌──────────────▼─────────────────────────────────────────────────▼─────┐           │  │
│   │          │                                                                      │           │  │
│   │          │                               nftables                               │           │  │
│   │          │                            (ip masquerade)                           │           │  │
│   │          └───────────────────────────────────────┬──────────────────────────────┘           │  │
│   │                                                  │                                          │  │
│   │                        ┌──────┐      ┌───────────▼──┐                                       │  │
│   │ other ports            │  lo  │      │     eth0     │         if a port within 24999-25002  │  │
│   │                        └──────┘      └────▲──────┬──┘                                       │  │
│   │                         if a port is 25003│      │                                          │  │
│   │                                           │      │                                          │  │
│   │          ┌────────────────────────────────┴──────▼──────────────────────────────┐           │  │
│   │          │                                                                      │           │  │
│   └──────────┤                              nftables                                ├───────────┘  │
│              │                         (port redirecter)                            │              │
│              └──────────────────────────────▲────────┬──────────────────────────────┘              │
│                                             │        │                                             │
└─────────────────────────────────────────────┼────────┼─────────────────────────────────────────────┘
                                              │        │
                                              │        ▼
                                             o u t s i d e

go to ASCIIFlow link)

or I have an idea to use eBPF but it is not simple.

Additional context

Thoughts about potential risks if we run the image builder CLI in-workspace:

1. image builder would need a pull secret, which a developer may not have at runtime, to pull from a private registry. Consider ignoring this use-case for now? Which would mean in-workspace builds are only available for dockerfiles where the base image is public.
2. image builder pushes as part of the build, but, this won't be able to go to GCR because they won't have our pull secret. So, it'd need a way to not push, but still do the build, so the image is available for docker run

@loujaybee @laushinka @akosyakov How does IDE team plan to collect feedback on the prototype? @akosyakov shared that the experience is currently poor / not doable in preview environments. Also, I am creating this issue because our team is highly distributed, therefore, we must default to async communication to share context.

@atduarte 👋

[kylos101]

@loujaybee do y'all have any target dates for which you'd like to have a prototype working by (not in main, of course)? I ask to help set expectations and timebox this activity. Let us know? 🙏

cc: @iQQBot @utam0k @laushinka

[kylos101]

@utam0k does IDE team need anything else from you? If yes, can you share a summary of what is needed and update the issue description? You may need to talk synchronously with @iQQBot, to better understand remaining challenges/concerns.

To review, none of the workspacekit experimental branches from IDE team are planned to merge to main. If IDE team does intend to merge any of the experimental workspacekit branches to main, please review thoroughly with both @Furisto and @csweichel . 🙏

cc: @jenting @WVerlaek , should you see a workspacekit PR enter our queue for review, please defer to ☝️

[utam0k]

I have updated the description

[kylos101]

Thanks for the 🏆 diagram, @utam0k ! 🤝

[kylos101]

Hi @laushinka, is there a time box for this activity? I ask because it's been running for a couple weeks, and appears to be starting to hit limits.

Hi @utam0k , @akosyakov bumped into some trouble here, wdyt? Are there any opportunities to help improve performance? 🤔

[utam0k]

@kylos101 I'll check it 👀

[laushinka]

is there a time box for this activity? I ask because it's been running for a couple weeks, and appears to be starting to hit limits.

Hi, @kylos101! Sorry for the lack of updates in this issue.
As you pointed out that we are currently hitting limits in the POC that needs the Workspace team's help, so my proposal is to:

1. Close this issue
2. Once our POC(s) are in a good enough state for Workspace team to review, we will write on Slack. Our estimate is it will roughly take ~50 minutes of @utam0k's (or someone else's) time.

How does that sound to you and @utam0k? 🙏🏽

[utam0k]

@laushinka Thanks for giving me your specific plan. Make sense a lot. But we are having trouble implementing the prototype. I think we need to continue to collaborate closely.
https://gitpod.slack.com/archives/C01KGM9BH54/p1673816111387239?thread_ts=1671437352.598149&cid=C01KGM9BH54
#15766

[laushinka]

@laushinka Thanks for giving me your specific plan. Make sense a lot. But we are having trouble implementing the prototype. I think we need to continue to collaborate closely.

Really appreciate your help, @utam0k 🧡 There is no pressure from our side, and as long as it doesn't interfere with your other priorities.
@kylos101 I leave it to you to close/keep the issue.

[kylos101]

Thanks @laushinka ! That sounds great Re: closing this issue to focus on other priorities. As a heads up, if a PR review and subsequent approval are desired in the future, we'll need to be careful and loop in others (Chris), too.

@utam0k is there anything you'd like to record or save, as part of this collaboration, before closing this issue? Closing this issue will help us focus on other priorities for Workspace Team.

[kylos101]

FYI, I closed #15766

[utam0k]

@utam0k is there anything you'd like to record or save, as part of this collaboration, before closing this issue? Closing this issue will help us focus on other priorities for Workspace Team.

I don't have anything more for now.

@laushinka If your team needs help again, feel free to ask our team! Let's make it together!