Add Yocto-Layer's Recipe to OE4T Image

[CalebProvost]

Clarification:

Prior issue #15 successfully added the layer to the build. However, due to such long build times, current validation demonstrates that the uartserver code was not being installed/packaged with the end image.

This issue is opened to resolve the current blocker which is impeding the installation of our the binaries into the end image.

DoD:

• binaries compiled via bitbake -c compile final-project
• binaries installed to staged rootfs via bitbake -c install final-project
• binaries within end system's SD Card image via bitbake demo-image-full and successive SD Creation steps

[CalebProvost]

Discussion documentation:

• Caleb Provost
  Tested the image this morning and it still does not contain our uartserver... ideas?
  https://github.com/cu-ecen-5013/final-project-CalebProvost/tree/yocto-layer

• Dan
  It looks like you are missing the yocto build files? What image are you building and where are you adding final-project to it?

• Caleb Provost
  The build script (tegra_install_n_build.sh):

# Clone L4T Yocto Base
YL4T_SUCCESS="false"
git clone https://github.com/OE4T/tegra-demo-distro.git ${CUR_DIR}/tegra-demo-distro
cd "${CUR_DIR}/tegra-demo-distro/" || echo "Could not enter L4T directory"
git checkout ${BRANCH}
git submodule update --init --recursive# Clone AESD Final Project Layer & add it to L4T layer
git clone https://github.com/cu-ecen-5013/final-project-CalebProvost.git --branch yocto-layer layers/meta-aesd-final
. ./setup-env --machine ${MACHINE} --distro ${DISTRO}
bitbake-layers add-layer "${CUR_DIR}/tegra-demo-distro/layers/meta-aesd-final"
# Begin L4T Tegra Build
bitbake ${BUILD_IMAGE} && export YL4T_SUCCESS="true"

• Caleb Provost
  At built time, bitbake shows it as a layer which it's building, and when cancelling the build and checking the layers (bitbake-layers show-recipes) it shows that it's betting built... But whenever I flash the created SD Card image... The uartserver is never in the image...

• Caleb Provost
  The script which builds the SD Card image for flashing:

    echo "Yocto Build of L4T Complete. Preping Image to flash to SD Card."
    mkdir -p "${CUR_DIR}/tegraflash"
    cd "${CUR_DIR}/tegraflash" || echo "Could Not Enter SD Card Staging Directory"
    cp "${CUR_DIR}/tegra-demo-distro/build/tmp/deploy/images/${MACHINE}/${BUILD_IMAGE}-${MACHINE}.tegraflash.tar.gz" "${CUR_DIR}/tegraflash/"
    tar -xf "${CUR_DIR}/tegra-demo-distro/build/tmp/deploy/images/${MACHINE}/${BUILD_IMAGE}-${MACHINE}.tegraflash.tar.gz"
    [ -f "../${BUILD_IMAGE}-${MACHINE}.img" ] && rm -rf "../${BUILD_IMAGE}-${MACHINE}.img"
    ./dosdcard.sh "../${BUILD_IMAGE}-${MACHINE}.img" && SDCARD_IMAGE="true"

• Dan
  what is BUILD_IMAGE and where does this image include the final-project recipe?

• Caleb Provost
  bitbake-layers add-layer "${CUR_DIR}/tegra-demo-distro/layers/meta-aesd-final"
  And BUILD_IMAGE is a passed in variable (or environment default) which for my builds is demo-image-full
  The build script (tegra_install_n_build.sh):

# Clone L4T Yocto Base
YL4T_SUCCESS="false"
git clone https://github.com/OE4T/tegra-demo-distro.git ${CUR_DIR}/tegra-demo-distro
cd "${CUR_DIR}/tegra-demo-distro/" || echo "Could not enter L4T directory"
git checkout ${BRANCH}

• Caleb Provost
  More specifically, it's included with:

# Clone AESD Final Project Layer & add it to L4T layer
git clone https://github.com/cu-ecen-5013/final-project-CalebProvost.git --branch yocto-layer layers/meta-aesd-final
. ./setup-env --machine ${MACHINE} --distro ${DISTRO}
bitbake-layers add-layer "${CUR_DIR}/tegra-demo-distro/layers/meta-aesd-final"

• Dan
  adding a layer won't include every recipe in the layer

• Caleb Provost
  But when I run bitbake-layers show-recipes it's in the list...

• Dan
  yes but that doesn't mean it's included in the image

• Caleb Provost
  Crap...

• Dan
  Look at assignment 6 for instance, to see how we did it there with a custom image

• Caleb Provost
  So I need to add an image append huh?

• Caleb Provost
  So... add this to local.conf?
  IMAGE_INSTALL_append = " final-project"

• Caleb Provost
  I'm adding demo-image-base.bbappend to our layer now.

• Caleb Provost
  @dan, I've pushed an update to https://github.com/cu-ecen-5013/final-project-CalebProvost/tree/yocto-layer and the image append fails to add the files...

• Caleb Provost

ERROR: Error executing a python function in exec_python_func() autogenerated:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:extend_recipe_sysroot(d)
     0003:
File: '/home/user/build/tegra-demo-distro/layers/meta/classes/staging.bbclass', lineno: 582, function: extend_recipe_sysroot
     0578:                    if "/bin/" in l or "/sbin/" in l:
     0579:                        # defer /*bin/* files until last in case they need libs
     0580:                        binfiles[l] = (targetdir, dest)
     0581:                    else:
 *** 0582:                        staging_copyfile(l, targetdir, dest, postinsts, seendirs)
     0583:
     0584:    # Handle deferred binfiles
     0585:    for l in binfiles:
     0586:        (targetdir, dest) = binfiles[l]
File: '/home/user/build/tegra-demo-distro/layers/meta/classes/staging.bbclass', lineno: 157, function: staging_copyfile
     0153:        os.symlink(linkto, dest)
     0154:        #bb.warn(c)
     0155:    else:
     0156:        try:
 *** 0157:            os.link(c, dest)
     0158:        except OSError as err:
     0159:            if err.errno == errno.EXDEV:
     0160:                bb.utils.copyfile(c, dest)
     0161:            else:
Exception: FileNotFoundError: [Errno 2] No such file or directory: '/home/user/tegra-demo-distro/build/tmp/sysroots-components/armv8a/gcc-runtime/sysroot-providers/virtual_aarch64-oe4t-linux-compilerlibs' -> '/home/user/build/tegra-demo-distro/build/tmp/work/armv8a-oe4t-linux/final-project/1.0+gitAUTOINC+bb18ccaaa1-r0/recipe-sysroot/tegra-demo-distro/build/tmp/sysroots-components/armv8a/gcc-runtime/sysroot-providers/virtual_aarch64-oe4t-linux-compilerlibs'
DEBUG: Python function extend_recipe_sysroot finished

• Dan
  what is at the path referenced there? Are the subdirectories present?

• Dan
  Have you tried building outside your docker container?

• Dan
  Look at permissions on the files/directories referenced there, I'm worried that docker builds might be confusing things.

• Dan
  Also attach the log file referenced

• Caleb Provost
  log.do_patch.314

DEBUG: Executing python function extend_recipe_sysroot

NOTE: Direct dependencies are ['/home/user/build/tegra-demo-distro/layers/meta/recipes-devtools/quilt/quilt-native_0.66.bb:do_populate_sysroot', 'virtual:native:/home/user/build/tegra-demo-distro/layers/meta/recipes-devtools/patch/patch_2.7.6.bb:do_populate_sysroot']

NOTE: patch-native exists in sysroot, but is stale (patch-native.f8fc7e3bfe25fc4e17d449ee25e5f2322d82189ad4bd192c27c9b2a851cce59a vs. patch-native.f8fc7e3bfe25fc4e17d449ee25e5f2322d82189ad4bd192c27c9b2a851cce59a), removing.

DEBUG: Removing manifest: /home/user/build/tegra-demo-distro/build/tmp/work/armv8a-oe4t-linux/final-project/1.0+gitAUTOINC+bb18ccaaa1-r0/recipe-sysroot-native/sysroot-providers/patch-native

DEBUG: Removing manifest: /home/user/build/tegra-demo-distro/build/tmp/work/armv8a-oe4t-linux/final-project/1.0+gitAUTOINC+bb18ccaaa1-r0/recipe-sysroot-native/sysroot-providers/patch-replacement-native

• Dan
  Looks like the error message changed, now referenced /home/user/tegra-demo-distro/build/tmp/sysroots-components/x86_64/quilt-native/sysroot-providers/quilt-native?

• Dan
  Do these directories exist?

• Dan
  what are the permissions?

• Caleb Provost
  I'm trying to settup another build system outside of docker so that I can validate the issue exists outside of docker

• Caleb Provost
  Looks like I gave the wrong log. 1 sec

• Caleb Provost
  Zip
  logs.zip
  39 kB Zip

• Caleb Provost
  what permissions for what files are you asking about?

• Caleb Provost
  Above is the log to the imaged issue.
  I finally got it to run without having to rebuild the entire thing from scratch (a whole lot of sym links were used...)

• Caleb Provost
  Both in docker and out of docker I get the same error. I don't think its a docker container permissions issue.:
  In Docker Log is .376 and our of docker log is .7253
  2 files
  log.do_prepare_recipe_sysroot.376
  173 kB Plain Text
  Click to view details
  log.do_prepare_recipe_sysroot.7253
  129 kB Plain Text
  Click to view details

• Dan
  What parts of this path exist? /home/user/tegra-demo-distro/build/tmp/sysroots-components/x86_64/gcc-cross-aarch64/sysroot-providers/virtual_aarch64-oe4t-linux-gcc

• Dan
  You can make this problem disappear if you just remove your new recipe, correct?

• Dan
  Then the build succeeds?

• Caleb Provost
  Interestingly enough, /home/user/tegra-demo-distro/build/tmp/sysroots-components/x86_64/gcc-cross-aarch64/sysroot-providers/virtual_aarch64-oe4t-linux-gcc doesn't exist, but /home/user/build/tegra-demo-distro/build/tmp/sysroots-components/x86_64/gcc-cross-aarch64/sysroot-providers/virtual_aarch64-oe4t-linux-gcc does. How is it missing the directory?
  Result:

ls -alh /home/user/build/tegra-demo-distro/build/tmp/sysroots-components/x86_64/gcc-cross-aarch64/sysroot-providers/virtual_aarch64-oe4t-linux-gcc
-rw-r--r-- 391 user user 17 Apr 11 22:14 /home/user/build/tegra-demo-distro/build/tmp/sysroots-components/x86_64/gcc-cross-aarch64/sysroot-providers/virtual_aarch64-oe4t-linux-gcc

• Caleb Provost
  Removing recipe now and rerunning.

• Caleb Provost
  Successfully builds without our layer/recipe:

aesd@aesdvm:~/yocto (dl4t %=)$ docker attach dl4t 
user@f416ed90a70d:~/build/tegra-demo-distro/build$ bitbake demo-image-full
Loading cache: 100% |##############################################| Time: 0:00:01
Loaded 3940 entries from dependency cache.
Parsing recipes: 100% |############################################| Time: 0:00:00
Parsing of 2588 .bb files complete (2587 cached, 1 parsed). 3941 targets, 124 skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies
NOTE: Multiple providers are available for runtime libxcb-glx0 (libxcb, libxcb-native)
Consider defining a PREFERRED_RPROVIDER entry to match libxcb-glx0Build Configuration:
BB_VERSION           = "1.49.2"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "universal"
TARGET_SYS           = "aarch64-oe4t-linux"
MACHINE              = "jetson-nano-2gb-devkit"
DISTRO               = "tegrademo"
DISTRO_VERSION       = "3.2+snapshot-20210415"
TUNE_FEATURES        = "aarch64 armv8a crc"
TARGET_FPU           = ""
meta                 = "HEAD:ea455ca8671d3bc2a1097989bfaabe92f3ca37ab"
meta-tegra           
contrib              = "HEAD:40701c99c42563a871b8fca0116440f0958a1124"
meta-oe              
meta-python          
meta-networking      
meta-filesystems     = "HEAD:6e99122ed81b7dcb8bffd82f0fefaed3898de8b0"
meta-virtualization  = "HEAD:438ed8b5065aac2bfd69c57e807befb42c8cc889"
meta-tegra-support   
meta-demo-ci         
meta-tegrademo       = "HEAD:c4ef10f44d92ac9f1e4725178ab0cefd9add8126"Initialising tasks: 100% |#########################################| Time: 0:00:07
Sstate summary: Wanted 0 Local 0 Network 0 Missed 0 Current 3598 (0% match, 100% complete)
NOTE: Executing Tasks
NOTE: Tasks Summary: Attempted 8738 tasks of which 8738 didn't need to be rerun and all succeeded.
user@f416ed90a70d:~/build/tegra-demo-distro/build$ 

• Dan
  Can you try just moving your recipe into one of the existing layers in tegra-demo-distro as a test point?

• Dan
  to remove your layer from the mix...

• Caleb Provost
  nice idea. 1 sec

• Caleb Provost
  Also succeeds outside of docker btw (was running after I posted the prior)

• Caleb Provost
  recipe was added to tegra-demo-distro/layers/meta-tegrademo/recipes-demo/final-project/final-project.bb
  Running now

• Caleb Provost
  success.... looking for binary uartserver now...

• Dan
  OK so take a close look at your layer config and compare that to the layer you added it to, something must be wrong there.

• Caleb Provost
  where would the binary be staged?

• Caleb Provost
  I ask because it's gonna take forever to validate the binary was compiled via: find tmp/ -type f -name uartserver

• Caleb Provost
  Should I have added the image.bbappend recipe too?

• Dan
  you'll need to add to your image, you don't need your own custom image though, you could add to the distro packages

• Caleb Provost
  I ask because I'm basing that append off of the layer I just used.

• Caleb Provost
  How so?

• Caleb Provost
  How do I add the binaries to the packaged image. Everything I've been trying fails.

• Caleb Provost
  I've tried the Assignment 6 method and that leads to the errors that started this thread.

• Caleb Provost
  So.. The bitbake is never actually building the recipe. None of the functions are getting triggered. When I try to manually trigger a function bitbake -c do_compile final-project it fails for the same reason that started all this.

• Caleb Provost
  Just doing a straight bitbake demo-image-full with this recipe in the layer, it never triggers it...

• Caleb Provost
  Why is the recipe being seen, but not being ran?
  It's pulling it from github... just not running oe_runmake...

[CalebProvost]

From what I interpreted from the discussion, it wasn't a docker user permissions issue, it wasn't the way our layer is structured (proved as it had the same error inside the demo layer), it's not getting compiled by OE/Bitbake...

[CalebProvost]

In lieu of waiting for the entire build executing, I deleted the directory and started the build like normal until bitbake started.
I then executed bitbake -c compile final-project so I could test if it was the recipe or something else entirely.
The recipe now compiles successfully, with no issues, and now the binaries are showing up, unlike before:
file /home/user/build/tegra-demo-distro/build/tmp/work/armv8a-oe4t-linux/final-project/1.0+gitAUTOINC+bb18ccaaa1-r0/git/uartserver
uartserver: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=3d13bc92198a46c68cdf06c996f3742106e703bc, for GNU/Linux 3.14.0, with debug_info, not stripped

Now running build like normal...

[CalebProvost]

Stopped the full build to quickly check and verify if the do_install portion of the recipe worked correctly.:

bitbake -c install final-project
Completed successfully and verified binary was "installed" to pre-image-packaging location with find temp/ -type f -name uartserver
Location confirmed to be tmp/work/armv8a-oe4t-linux/final-project/1.0+gitAUTOINC+bb18ccaaa1-r0/image/usr/bin/uartserver.

So far... nothing was changed besides deleting the existing build and starting from scratch.

[CalebProvost]

Just flashed and loaded the SD Card image... The uartserver binary is not on it...

[CalebProvost]

From the discussion with the professor (attached below), we'll need to use a distro append/creation method instead of the image append method that we were taught with assignment 6.

The key points of take-away (for me at least) is that we'll need to change our meta-aesd-final layer's structure to be a distro based off of the OE4T image we want. Then, add our recreated repo as a submodule under the OE4T's repos directory which is sym-linked to from within OE4T's layers directory. From there its apparently as easy as "clone, submodule update, setup-env and build."

• Dan

How do I add the binaries to the packaged image. Everything I've been trying fails.

You can do an image install append in the distro file like you see at https://github.com/OE4T/tegra-demo-distro/blob/dunfell-l4t-r32.4.3/layers/meta-tegrademo/conf/distro/tegrademo-mender.conf#L38
tegrademo-mender.conf
IMAGE_INSTALL_append_tegra = " tegra-eeprom-tool i2c-tools tegra-bup-payload"

• Dan
  Look at this section of the yocto reference manual: https://www.yoctoproject.org/docs/1.8/ref-manual/ref-manual.html#image-generation-dev-environment

• Dan
  A good way to start though is just to look at how/where a distribution is including packages and add yours there.

• Caleb Provost
  The build I currently have running is using IMAGE_INSTALL_append = " final-project"... should this be IMAGE_INSTALL_append_tegra = " final-project" instead?

• Dan
  no it shouldn't matter

• Dan
  so are you saying you still don't have your package included in the build?

• Caleb Provost
  It's still building...
  The only thing I could think of was that something got corrupt with the build packages... So I deleted it all and am starting from scratch. But I made sure to run bitbake -c compile final-project before doing so to validate that the recipe does actually build the binary.

• Caleb Provost
  I just stopped it and am about to run bitbake -c install final-project in order to see if it'll stage it for the rootfs

• Caleb Provost
  That ran without a problem... Checking to see with find tmp/ -type f -name uartserver results with:

tmp/work/armv8a-oe4t-linux/final-project/1.0+gitAUTOINC+bb18ccaaa1-r0/image/usr/bin/uartserver
tmp/work/armv8a-oe4t-linux/final-project/1.0+gitAUTOINC+bb18ccaaa1-r0/git/uartserver

• Caleb Provost
  @dan, is image/usr/bin/ the rootfs staging area? If so... then is everything working so far.

• Dan
  there's a separate area used for the image when you get to that point in the build, but this looks like the right place for the recipe output

... {Omitted progress notifications from Caleb Provost} ...

• Caleb Provost
  Build just completed successfully. No issues encountered.
  Flashing SD Card now for final test...

• Caleb Provost
  @dan,
  It didn't work. The image still doesn't have our binary in it.
  You mentioned:

there's a separate area used for the image when you get to that point in the build, but this looks like the right place for the recipe output

What am I missing for it to go from the staging to inside the rootfs?

• Caleb Provost
  Whelp... I changed it from IMAGE_INSTALL_append to IMAGE_INSTALL_append_tegra and placed that line inside the recipe itself... and now it's showing up in more locations. I don't know how to validate if any of these are going to go into the final image though.

user@1fe92c6db610:~/build/tegra-demo-distro/build$ find tmp/ -type f -name uartserver
tmp/work/armv8a-oe4t-linux/final-project/1.0+gitAUTOINC+bb18ccaaa1-r0/packages-split/final-project-dbg/usr/bin/.debug/uartserver
tmp/work/armv8a-oe4t-linux/final-project/1.0+gitAUTOINC+bb18ccaaa1-r0/packages-split/final-project/usr/bin/uartserver
tmp/work/armv8a-oe4t-linux/final-project/1.0+gitAUTOINC+bb18ccaaa1-r0/package/usr/bin/.debug/uartserver
tmp/work/armv8a-oe4t-linux/final-project/1.0+gitAUTOINC+bb18ccaaa1-r0/package/usr/bin/uartserver
tmp/work/armv8a-oe4t-linux/final-project/1.0+gitAUTOINC+bb18ccaaa1-r0/image/usr/bin/uartserver
tmp/work/armv8a-oe4t-linux/final-project/1.0+gitAUTOINC+bb18ccaaa1-r0/git/uartserver

• Dan
  it won't help to put in the recipe

• Dan
  Can you please check in what you are building and then send me a link to the image install location as well as the commands you are using for setup_env and bitbake build?

• Caleb Provost

check in what you are building

I don't understand what that means.

link to the image install location

like... the repo where the code that is doing this, or the physical location in my build tree?

the commands you are using for setup_env and bitbake build

docker pull calebprovost/dockter-l4t:sdk_installed
docker run -it -v /home/user/build:/home/user/build --name dl4t calebprovost/dockter-l4t:sdk_installed bash

then within the docker container:

git clone https://github.com/OE4T/tegra-demo-distro.git tegra-demo-distro
cd tegra-demo-distro
git checkout c4ef10f44d92ac9f1e4725178ab0cefd9add8126
git submodule update --init --recursive
git clone https://github.com/cu-ecen-5013/final-project-CalebProvost.git --branch yocto-layer layers/meta-aesd-final
. ./setup-env --machine jetson-nano-2gb-devkit --distro tegrademo
bitbake-layers add-layer ../layers/meta-aesd-final
bitbake demo-image-full

• Dan
  Where are you adding your IMAGE_INSTALL package?

• Dan

I don't understand what that means.

Typically what you'd do here is add your meta-aesd-final as a submodule under the repos directory

• Dan
  and then symlink from layers

• Dan
  and then make a new fork of tegra-demo-distro with your customized content

• Dan
  so then all you need to do is clone, submodule update, setup-env and build

[CalebProvost]

I shared the output build tree with Dan and asked if the latest changes actually look like it'll have our code in it. He says it looks like it should be good...

The commit to the changes which should be the new structure which yields our work in the image:
https://github.com/cu-ecen-5013/final-project-CalebProvost/tree/eb528d29b93e22669ea777678928a2d766027269

Flashing the image to SD card now, confirmation pending...

[CalebProvost]

Nope... Didn't work. Not only was our code not in it, it also was not the full demo image...

I'm just gonna copy and paste what already exists with OE4T's distro has and insert our stuff into it. None of the "append" stuff we've learned is working and Dan hasn't been pointing out why... Just tells me that I need to back track.

[CalebProvost]

This took way too long to do.
I got it working. I just had to recreate the entire image that OE4T already did as a simple image.bbappend didn't want to work...

https://github.com/cu-ecen-5013/final-project-CalebProvost/tree/c4ca47058b9b14bc6c87c27984c22828e96ce770