Moves nix-tools standard templates into iohk-nix #30

angerman · 2019-01-23T05:56:04Z

As such we can now define:

let
  localLib = import ./lib.nix;
in
{ pkgs, ... }@args
localLib.nix-tools.default-nix ./nix/pkgs.nix args // {
    # additional non-haskell packages we want to build on CI.
    byronLedgerSpec = import ./specs/ledger/latex {};
    byronChainSpec = import ./specs/chain/latex {};
    semanticsSpec = import ./specs/semantics/latex {};
}

as default.nix

and

let
  localLib = import ./lib.nix;
in
localLib.nix-tools.release-nix {
  package-set-path = ./.;

  # packages from our stack.yaml or plan file (via nix/pkgs.nix) we
  # are intereted in building on CI via nix-tools.
  packages = [ "cardano-chain" "cs-ledger" "small-steps" "cs-blockchain" ];

  # The set of jobs we consider crutial for each CI run.
  # if a single one of these fails, the build will be marked
  # as failed.
  #
  # The names can be looked up on hydra when in doubt.
  #
  # custom jobs will follow their name as set forth in
  # other-packages.
  #
  # nix-tools packages withh be prefixed with nix-tools and
  # follow the following naming convention:
  #
  #   namespace                      optional cross compilation prefix                 build machine
  #   .-------.                              .-----------------.                .--------------------------.
  #   nix-tools.{libs,exes,tests,benchmarks}.{x86_64-pc-mingw-,}$pkg.$component.{x86_64-linux,x86_64-darwin}
  #             '--------------------------'                    '-------------'
  #                 component type                          cabal pkg and component*
  #
  # * note that for libs, $component is empty, as cabal only
  # provides a single library for packages right now.
  #
  # Example:
  #
  #   libs.cardano-chain.x86_64-darwin -- will build the cardano-chain library on and for macOS
  #   libs.cardano-chain.x86_64-linux -- will build the cardano-chain library on and for linux
  #   libs.x86_64-pc-mingw32-cardano-chain.x86_64-linux -- will build the cardano-chain library on linux for windows.
  #   tests.cs-ledger.ledger-delegation-test.x86_64-linux -- will build and run the ledger-delegation-test from the
  #                                                          cs-ledger package on linux.
  #
  required-name = "cardano-chain-required-checks";
  required-targets = jobs: [
    jobs.byronLedgerSpec
    jobs.byronChainSpec
    jobs.semanticsSpec

    jobs.nix-tools.libs.cs-blockchain.x86_64-darwin
    # ...

    # windows cross compilation targets
    jobs.nix-tools.libs.x86_64-pc-mingw32-cardano-chain.x86_64-linux
    # ...
  ];
}

as release.nix.

nix-tools-release.nix

michaelpj · 2019-01-23T12:00:49Z

default.nix

+    # default and release templates that abstract
+    # over the details for CI.
+    default-nix = import ./nix-tools-default.nix commonLib;
+    release-nix = import ./nix-tools-release.nix commonLib;


Are these usable as part of a larger default.nix or release.nix? For example, in the Plutus repo we also have a bunch of latex that's buildable via our top-level default.nix and included in release.nix.

Also, writing a little about how to use these in a README or similar would be good.

It should be perfectly fine to use the default-nix template and just merge it with custom package definitions. The release-nix will still rely on the default.nix.

@michaelpj does the example (See the PR message atop) suite your needs?

Good example, do put them somewhere in the actual repository so they're available for future people!

nix-tools-default.nix

michaelpj · 2019-01-24T10:21:55Z

default.nix example

I'd rather do something like this, does that work?

let localLib = import ./lib.nix;
in {
    nix-tools = import localLib.nix-tools.default-nix ./nix/pkgs.nix {};
    # other stuff
    ...
}

Similarly for the release.nix. Basically, I don't like the inversion of control: if I want to have other non-Haskell stuff I have to pass it in to the nix-tools template. I'd much rather have it be a plain attribute set or something that I can just use normally.

angerman · 2019-01-24T11:20:13Z

@michaelpj maybe I'm misreading you...

if I want to have other non-Haskell stuff I have to pass it in to the nix-tools template. I'd much rather have it be a plain attribute set or something that I can just use normally.

{ pkgs, ... }@args:
localLib.nix-tools.default-nix ./nix/pkgs.nix args // {
    # additional non-haskell packages we want to build on CI.
    byronLedgerSpec = import ./specs/ledger/latex {};
    byronChainSpec = import ./specs/chain/latex {};
    semanticsSpec = import ./specs/semantics/latex {};
}

This is equivalent to

{ pkgs, ... }@args:
{
    inherit (localLib.nix-tools.default-nix ./nix/pkgs.nix args) nix-tools;

    # additional non-haskell packages we want to build on CI.
    byronLedgerSpec = import ./specs/ledger/latex {};
    byronChainSpec = import ./specs/chain/latex {};
    semanticsSpec = import ./specs/semantics/latex {};
}

The slightly annoying but important part is that we pass args to the nix-tools template. Otherwise cross compilation would not work as hydra through release.nix, which will call default.nix with arguments, provides the right arguments and pkgs configuration for cross compilation to work.

Alright, I lied. It's technically equivalent to inherit (localLib.nix-tools.default-nix ./nix/pkgs.nix args) nix-tools _lib;

michaelpj · 2019-01-24T11:29:41Z

Okay, seems fine, but I'd like to have both documented.

Have a look at the Plutus default.nix to see how much stuff there is in there. Ideally this would replace the haskellPackages bit and leave the rest much the same.

michaelpj · 2019-01-25T10:38:45Z

Looks like you're still working on this - let me know if you'd like me to have another look.

jbgi · 2019-01-28T09:56:37Z

@angerman Should we also include packaging of remote-iserv and friends with this PR? I think it would make sense.

nix-tools-default.nix

angerman · 2019-01-31T08:33:39Z

@angerman Should we also include packaging of remote-iserv and friends with this PR? I think it would make sense.

@jbgi yes! yes, we should. Was added in c101f9f.

michaelpj

Looks fine, examples would be great!

default.nix

haskell-hackage-stackage.nix

nix-tools-default.nix

nix-tools-iohk-module.nix

nix-tools-release.nix

patches/clock-0.7.2.patch

As such we can now define: ``` let localLib = import ./lib.nix; in localLib.nix-tools.default-nix ./nix/pkgs.nix ``` as `default.nix` and ``` let localLib = import ./lib.nix; in localLib.nix-tools.release-nix { packages = [ "cardano-chain" "cs-ledger" "small-steps" "cs-blockchain" ]; required-name = "cardano-chain-required-checks"; other-packages = { byronLedgerSpec = import ./specs/ledger/latex {}; byronChainSpec = import ./specs/chain/latex {}; semanticsSpec = import ./specs/semantics/latex {}; }; package-set-path = ./.; } ``` as `release.nix`.

This should remove the datadir mismatch when using component builds with cabal.

disassembler · 2019-02-04T19:18:21Z

@angerman good with this, but can we as a follow-up add bors since we're adding CI checks now?

angerman requested review from rvl, cleverca22, deepfire, disassembler and jbgi January 23, 2019 05:56

angerman force-pushed the feature/nix-tools-generators branch from f8797f4 to a2c9a3a Compare January 23, 2019 09:39

michaelpj reviewed Jan 23, 2019

View reviewed changes

angerman force-pushed the feature/nix-tools-generators branch from 40ee24c to 420910d Compare January 24, 2019 01:52

angerman mentioned this pull request Jan 25, 2019

Alex and Happy in more recent versions start to break input-output-hk/haskell.nix#31

Closed

jbgi reviewed Jan 28, 2019

View reviewed changes

nix-tools-default.nix Outdated Show resolved Hide resolved

angerman force-pushed the feature/nix-tools-generators branch from 4dd072e to 340e2ef Compare January 30, 2019 02:48

michaelpj reviewed Jan 31, 2019

View reviewed changes

angerman added 12 commits February 4, 2019 14:47

Less magic, let the consumer be explicit about the required targets.

a42e1f5

Drop other-packages in favour of pulling them from default.nix

84f2d01

drop _lib as well.

30a6e41

Adds Cabal patch

6a8c291

This should remove the datadir mismatch when using component builds with cabal.

Overlay, extended module, more patches, mingw32 logic.

cf319d7

Now with even more cross compilation support.

bb5918b

patches, patches, patches!

2e4a534

bump packages

61a4ad1

Add prebuilt rocksdb expression

e5d6a95

Bump default nixpkgs

5aef828

bump haskell.nix

6b5fb09

angerman added 8 commits February 4, 2019 14:47

change hackage/haskell/stackage repo

7e42c85

Minor adjustments

380a29b

forgot self

22c0f17

🔫

b75fd48

Allow injecting the th-packages.

8af92c5

🔥 😊

fc774df

bump stackage and hackage

ed70549

Adds more documentation.

917e491

angerman force-pushed the feature/nix-tools-generators branch from 1310a07 to 917e491 Compare February 4, 2019 08:09

angerman added 2 commits February 4, 2019 16:21

:-/

ac8ae44

Apparently hsc2hs *is* missing in 86 as well :-/

c92f011

michaelpj approved these changes Feb 4, 2019

View reviewed changes

angerman mentioned this pull request Feb 4, 2019

[DEVOPS-1214] Nix-toolification of cardano-chain input-output-hk/cardano-ledger-byron#246

Merged

disassembler approved these changes Feb 4, 2019

View reviewed changes

disassembler merged commit f9a1654 into master Feb 4, 2019

disassembler deleted the feature/nix-tools-generators branch February 4, 2019 19:19

Moves nix-tools standard templates into iohk-nix #30

Moves nix-tools standard templates into iohk-nix #30

Uh oh!

Conversation

angerman commented Jan 23, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

michaelpj Jan 23, 2019

Choose a reason for hiding this comment

Uh oh!

angerman Jan 24, 2019

Choose a reason for hiding this comment

Uh oh!

angerman Jan 24, 2019

Choose a reason for hiding this comment

Uh oh!

michaelpj Jan 24, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

michaelpj commented Jan 24, 2019

Uh oh!

angerman commented Jan 24, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

michaelpj commented Jan 24, 2019

Uh oh!

michaelpj commented Jan 25, 2019

Uh oh!

jbgi commented Jan 28, 2019

Uh oh!

Uh oh!

angerman commented Jan 31, 2019

Uh oh!

michaelpj left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

disassembler commented Feb 4, 2019

Uh oh!

Uh oh!

angerman commented Jan 23, 2019 •

edited

Loading

angerman commented Jan 24, 2019 •

edited

Loading