ros service enable always pulls from registry and ignores locally generated images

[albertdb]

RancherOS Version: v1.0.2

Docker Version: 1.10.3

Where are you running RancherOS? openstack

This worked perfectly on v0.4.3

The issue is very simple to reproduce:

1. Build image locally
2. Write custom service .yml referencing image of step 1 > /var/lib/rancher/conf/custom_service.yml
3. ros service enable /var/lib/rancher/conf/custom_service.yml

[ 8133.468164] DEBU[0000] START: [ros service enable /var/lib/rancher/conf/custom_service.yml] in /home/rancher/te 
[ 8133.586103] DEBU[0000] Using cached file: /var/lib/rancher/cache/5766785a73996b5975746bfc09d463aa 
[ 8133.587155] DEBU[0000] Loaded /var/lib/rancher/conf/custom_service.yml from /var/lib/rancher/conf/custom_service.yml 
[ 8135.313122] ERRO[0001] Failed Pulling custom_service : Status: Error: image library/local_image not found, Code: 1 
[ 8135.313477] ERRO[0001] Failed to start: custom_service : Status: Error: image library/local_image not found, Code: 1 
[ 8135.313815] FATA[0001] Status: Error: image library/local_image not found, Code: 1 

time="2017-06-07T14:49:56.941340098Z" level=debug msg="Calling GET /v1.20/info" 
time="2017-06-07T14:49:56.941448841Z" level=debug msg="GET /v1.20/info" 
time="2017-06-07T14:49:57.041674189Z" level=debug msg="Calling POST /v1.20/images/create" 
time="2017-06-07T14:49:57.041799213Z" level=debug msg="POST /v1.20/images/create?fromImage=docker.io%2Flibrary%2Flocal_image&tag=latest" 
time="2017-06-07T14:49:57.042098925Z" level=debug msg="Trying to pull local_image from https://registry-1.docker.io v2" 
time="2017-06-07T14:49:58.231224204Z" level=debug msg="Trying to pull local_image from https://index.docker.io v1" 
time="2017-06-07T14:49:58.231390061Z" level=debug msg="hostDir: /etc/docker/certs.d/docker.io" 
time="2017-06-07T14:49:58.231744951Z" level=debug msg="[registry] Calling GET https://index.docker.io/v1/repositories/library/local_image/images" 
time="2017-06-07T14:49:58.661443013Z" level=debug msg="Not continuing with error: Error: image library/local_image not found" 

Is there any quick fix for this?

Thank you.

[SvenDowideit]

this seems really surprising - it works for the system services that we pre-load in the OS boot sequence.

I'll look into it and add some tests (assuming there aren't some already)

[albertdb]

@SvenDowideit Exactly the same happens with system-docker when declaring label io.rancher.os.scope=system

[albertdb]

If instead of defining a service with its own .yml and enabling it via ros service enable, I merge it to cloud-config via ros config merge, then it works.

Anyway, I think this should be fixed quite urgently, anyone that reads the manual will procede with ros service approach.

[zimme]

I think I've found the where the "problem" lies.

os/compose/project.go

Line 226 in a425c4c

bytes, err := network.LoadServiceResource(service, true, cfg)

Ii just tries to get the service with network instead of first trying without network and if that fails, try again with network.

If this is correct I could try a PR for this.

[albertdb]

@zimme Nice to see that. I wonder if @SvenDowideit can confirm your findings.

[johnko]

Is it related to #1817 ?

[SvenDowideit]

@albertdb sorry for the delay, I had to get v1.0.3 out urgently due to Stack Clash. I'll be proding this issue asap.

@johnko yeah, it does seem like an under-speced area with inconsistent outcomes.

[albertdb]

Related to this problem area, I am occasionally (once every 20 attempts) experiencing that the custom services (at least the ones defined via cloud-config) containers are not created and started. Im seeing in dmesg that services are "started" and then keep waiting for Docker. Once Docker is ready, I see something like "connected to /var/run/docker.sock", but nothing else happens afterwards. Normally, containers are created and started following that log trace.

[SvenDowideit]

@zimme wanna make a PR for that?

[zimme]

No problem, I just wanted some confimation that I was in the right srea as I'm not too familiar with the code.
Should be able to have it done over the weekend

edit: Sorry I didn't get the time to do this as I had hoped and I forgot to update you guys.