Aztec3 Specific Work in Barretenberg (AztecProtocol/barretenberg#142)

* Split Pedersen Hash & Commitment Gadgets (AztecProtocol/barretenberg#95)

* [SQUASHED] Pedersen refactor into hash and commitment.

Use lookup pedersen for merkle tree, fixed-base pedersen for commitments.
---------
Co-authored-by: Suyash Bagad <[email protected]>

Port `copy_as_new_witness`.

Port `must_imply`.

`operator++`.

Port changes from `common`.

Port `ecc/groups`.

* [CPM] add missing dependencies to libbarretenberg.a (AztecProtocol/barretenberg#154)
---------

* Increase Pedersen Generator indices and subindices. (AztecProtocol/barretenberg#169)

* Remove a3 specific types. (AztecProtocol/barretenberg#252)

* Address Luke's Comments on `aztec3 -> master` (AztecProtocol/barretenberg#263)

* Add must_imply tests.

* Added a test for `field_t::copy_as_new_witness`

* add test for `conditional_assign`

* Added `infinity` test.

* Add `add_affine_test`.

* Tests for Array Object in `stdlib` (AztecProtocol/barretenberg#262)

* basic array tests.

* Add `composer_type` while hashing/compressing a vkey.

* Add `contains_recursive_proof` to Recursive VK (AztecProtocol/barretenberg#268)

* feat: debug utility for serialization (AztecProtocol/barretenberg#290)

* feat: enable asan config

* `array_push` for Generic Type (AztecProtocol/barretenberg#291)

* Add Indexed Merkle Tree  (AztecProtocol/barretenberg#281)

* remove ts (consulted with Adam and we're good to go). (AztecProtocol/barretenberg#292)

* Add cout for verification_key struct (AztecProtocol/barretenberg#295)

* compute tree (AztecProtocol/barretenberg#298)

* [SQUASHED] fixing `push_array_to_array` method. (AztecProtocol/barretenberg#304)

* feat(memory_tree|a3): add sibling path calculations (AztecProtocol/barretenberg#301)

* feat(memory_tree): frontier paths

* fix array and resolve merge conflicts (AztecProtocol/barretenberg#305)

* Mc/hash vk (AztecProtocol/barretenberg#306)

* Increase number of sub-generators to 128.

* Build a3crypto.wasm (AztecProtocol/barretenberg#311)

* More Tests on A3 `stdlib` methods (AztecProtocol/barretenberg#316)

* test: more vk tests to compare circuit/native/vk_data (AztecProtocol/barretenberg#310)

* Mc/hash vk (AztecProtocol/barretenberg#306)

* inc num_generators_per_hash_index to 128. (AztecProtocol/barretenberg#309)

* fix. (AztecProtocol/barretenberg#318)

* Added test for `compute_tree_native`. (AztecProtocol/barretenberg#319)

* Install instructions for apt on ubuntu (AztecProtocol/barretenberg#312)

* Fix address compilation. (AztecProtocol/barretenberg#329)

---------

Co-authored-by: David Banks <[email protected]>
Co-authored-by: Michael Connor <[email protected]>
Co-authored-by: dbanks12 <[email protected]>
Co-authored-by: Santiago Palladino <[email protected]>
Co-authored-by: ludamad <[email protected]>
Co-authored-by: Maddiaa <[email protected]>
Co-authored-by: Santiago Palladino <[email protected]>
Co-authored-by: ludamad <[email protected]>
Co-authored-by: cheethas <[email protected]>

Author: 8 people

barretenberg/README.md

@@ -11,17 +11,32 @@
 - libomp (if multithreading is required. Multithreading can be disabled using the compiler flag `-DMULTITHREADING 0`)
 - wasm-opt (part of the [Binaryen](https://github.com/WebAssembly/binaryen) toolkit)
 
+To install on Ubuntu, run:
+```
+sudo apt-get install cmake clang clang-format ninja-build binaryen
+```
+
 ### Installing openMP (Linux)
 
+Install from source:
+
 ```
-RUN git clone -b release/10.x --depth 1 https://github.com/llvm/llvm-project.git \
+git clone -b release/10.x --depth 1 https://github.com/llvm/llvm-project.git \
   && cd llvm-project && mkdir build-openmp && cd build-openmp \
   && cmake ../openmp -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DLIBOMP_ENABLE_SHARED=OFF \
   && cmake --build . --parallel \
   && cmake --build . --parallel --target install \
   && cd ../.. && rm -rf llvm-project
 ```
 
+Or install from a package manager, on Ubuntu:
+
+```
+sudo apt-get install libomp-dev
+```
+
+> Note: on a fresh Ubuntu Kinetic installation, installing OpenMP from source yields a `Could NOT find OpenMP_C (missing: OpenMP_omp_LIBRARY) (found version "5.0")` error when trying to build Barretenberg. Installing from apt worked fine.
+
 ### Getting started
 
 Run the bootstrap script. (The bootstrap script will build both the native and wasm versions of barretenberg)

barretenberg/cpp/CMakeLists.txt

@@ -18,6 +18,8 @@ option(BENCHMARKS "Build benchmarks" ON)
 option(FUZZING "Build fuzzing harnesses" OFF)
 option(DISABLE_TBB "Intel Thread Building Blocks" ON)
 option(COVERAGE "Enable collecting coverage from tests" OFF)
+option(SERIALIZE_CANARY "Build with serialize canary" OFF)
+option(ENABLE_ASAN "Address sanitizer for debugging tricky memory corruption" OFF)
 option(ENABLE_HEAVY_TESTS "Enable heavy tests when collecting coverage" OFF)
 option(INSTALL_BARRETENBERG "Enable installation of barretenberg. (Projects embedding barretenberg may want to turn this OFF.)" ON)
 option(USE_TURBO "Enable the use of TurboPlonk in barretenberg." OFF)
@@ -39,6 +41,15 @@ if(CMAKE_SYSTEM_PROCESSOR MATCHES "aarch64" OR CMAKE_SYSTEM_PROCESSOR MATCHES "a
     set(DISABLE_TBB 0)
 endif()
 
+if(ENABLE_ASAN)
+    add_compile_options(-fsanitize=address)
+    add_link_options(-fsanitize=address)
+endif()
+
+if(SERIALIZE_CANARY)
+    add_definitions(-DENABLE_SERIALIZE_CANARY)
+endif()
+
 if(FUZZING)
     add_definitions(-DFUZZING=1)
 

barretenberg/cpp/dockerfiles/Dockerfile.wasm-linux-clang

@@ -5,10 +5,11 @@ WORKDIR /usr/src/barretenberg/cpp/src
 RUN curl -s -L https://github.com/CraneStation/wasi-sdk/releases/download/wasi-sdk-12/wasi-sdk-12.0-linux.tar.gz | tar zxfv -
 WORKDIR /usr/src/barretenberg/cpp
 COPY . .
-# Build both honk_tests barretenberg.wasm
+# Build both honk_tests barretenberg.wasm primitives.wasm
 # This ensures that we aren't using features that would be incompatible with WASM for Honk
-RUN cmake --preset wasm && cmake --build --preset wasm --target honk_tests --target barretenberg.wasm
+RUN cmake --preset wasm && cmake --build --preset wasm --target honk_tests --target barretenberg.wasm --target primitives.wasm
 
 FROM alpine:3.17
 COPY --from=builder /usr/src/barretenberg/cpp/build-wasm/bin/barretenberg.wasm /usr/src/barretenberg/cpp/build/bin/barretenberg.wasm
+COPY --from=builder /usr/src/barretenberg/cpp/build-wasm/bin/primitives.wasm /usr/src/barretenberg/cpp/build/bin/primitives.wasm
 COPY --from=builder /usr/src/barretenberg/cpp/build-wasm/bin/*_tests /usr/src/barretenberg/cpp/build/bin/

barretenberg/cpp/scripts/bb-tests

@@ -2,7 +2,7 @@ crypto_aes128_tests
 crypto_blake2s_tests
 crypto_blake3s_tests
 crypto_ecdsa_tests
-crypto_pedersen_tests
+crypto_pedersen_commitment_tests
 crypto_schnorr_tests
 crypto_sha256_tests
 ecc_tests
@@ -18,7 +18,7 @@ stdlib_blake2s_tests
 stdlib_blake3s_tests
 stdlib_ecdsa_tests
 stdlib_merkle_tree_tests
-stdlib_pedersen_tests
+stdlib_pedersen_commitment_tests
 stdlib_schnorr_tests
 stdlib_sha256_tests
 transcript_tests

barretenberg/cpp/src/CMakeLists.txt

@@ -76,22 +76,28 @@ if(WASM)
         $<TARGET_OBJECTS:crypto_blake3s_objects>
         $<TARGET_OBJECTS:crypto_keccak_objects>
         $<TARGET_OBJECTS:crypto_schnorr_objects>
-        $<TARGET_OBJECTS:crypto_pedersen_objects>
+        $<TARGET_OBJECTS:crypto_generators_objects>
+        $<TARGET_OBJECTS:crypto_pedersen_hash_objects>
+        $<TARGET_OBJECTS:crypto_pedersen_commitment_objects>
         $<TARGET_OBJECTS:ecc_objects>
         $<TARGET_OBJECTS:polynomials_objects>
         $<TARGET_OBJECTS:plonk_objects>
         $<TARGET_OBJECTS:honk_objects>
         $<TARGET_OBJECTS:proof_system_objects>
         $<TARGET_OBJECTS:stdlib_primitives_objects>
         $<TARGET_OBJECTS:stdlib_schnorr_objects>
-        $<TARGET_OBJECTS:stdlib_pedersen_objects>
+        $<TARGET_OBJECTS:stdlib_pedersen_hash_objects>
+        $<TARGET_OBJECTS:stdlib_pedersen_commitment_objects>
         $<TARGET_OBJECTS:stdlib_blake2s_objects>
         $<TARGET_OBJECTS:stdlib_blake3s_objects>
         $<TARGET_OBJECTS:stdlib_sha256_objects>
         $<TARGET_OBJECTS:stdlib_aes128_objects>
         $<TARGET_OBJECTS:stdlib_merkle_tree_objects>
         $<TARGET_OBJECTS:acir_format_objects>
         $<TARGET_OBJECTS:acir_proofs_objects>
+        $<TARGET_OBJECTS:stdlib_sha256_objects>
+        $<TARGET_OBJECTS:stdlib_aes128_objects>
+        $<TARGET_OBJECTS:stdlib_merkle_tree_objects>
     )
 
     # With binaryen installed, it seems its wasm backend optimiser gets invoked automatically.
@@ -105,12 +111,35 @@ if(WASM)
         -nostartfiles -O2 -Wl,--no-entry -Wl,--export-dynamic -Wl,--import-memory -Wl,--allow-undefined -Wl,--stack-first -Wl,-z,stack-size=1048576
     )
 
+    # Repeat the above but for the smaller primitives.wasm
+    # Used in packages where we don't need the full contents of barretenberg
+    add_executable(
+        primitives.wasm
+        $<TARGET_OBJECTS:srs_objects>
+        $<TARGET_OBJECTS:numeric_objects>
+        $<TARGET_OBJECTS:crypto_sha256_objects>
+        $<TARGET_OBJECTS:crypto_aes128_objects>
+        $<TARGET_OBJECTS:crypto_blake2s_objects>
+        $<TARGET_OBJECTS:crypto_blake3s_objects>
+        $<TARGET_OBJECTS:crypto_generators_objects>
+        $<TARGET_OBJECTS:crypto_keccak_objects>
+        $<TARGET_OBJECTS:crypto_schnorr_objects>
+        $<TARGET_OBJECTS:crypto_pedersen_hash_objects>
+        $<TARGET_OBJECTS:crypto_pedersen_commitment_objects>
+        $<TARGET_OBJECTS:ecc_objects>
+    )
+
+    target_link_options(
+        primitives.wasm
+        PRIVATE
+        -nostartfiles -O2 -Wl,--no-entry -Wl,--export-dynamic -Wl,--import-memory -Wl,--allow-undefined -Wl,--stack-first -Wl,-z,stack-size=1048576
+    )
+
     # TODO(blaine): Figure out how to Asyncify the wasm output.
     # Binaryen's Asyncify transform produces wasm that has too many local variables to run in a WebAssembly
     # instance. This likely would be "solved" by enabling the optimizations to reduce the number of locals,
     # but using any optimization level results in a wasm file that takes an unusable amount of time to solve the
     # most simple prood.
-
     # find_program(WASM_OPT wasm-opt)
 
     # if(NOT WASM_OPT)
@@ -124,6 +153,13 @@ if(WASM)
     #     VERBATIM
     # )
 
+    add_custom_command(
+        TARGET primitives.wasm
+        POST_BUILD
+        COMMAND wasm-opt "$<TARGET_FILE:primitives.wasm>" -O2 -o "$<TARGET_FILE:primitives.wasm>"
+        VERBATIM
+    )
+
     if(INSTALL_BARRETENBERG)
         install(TARGETS barretenberg.wasm DESTINATION ${CMAKE_INSTALL_BINDIR})
     endif()
@@ -142,15 +178,18 @@ if(WASM)
         $<TARGET_OBJECTS:crypto_blake3s_objects>
         $<TARGET_OBJECTS:crypto_keccak_objects>
         $<TARGET_OBJECTS:crypto_schnorr_objects>
-        $<TARGET_OBJECTS:crypto_pedersen_objects>
+        $<TARGET_OBJECTS:crypto_generators_objects>
+        $<TARGET_OBJECTS:crypto_pedersen_hash_objects>
+        $<TARGET_OBJECTS:crypto_pedersen_commitment_objects>
         $<TARGET_OBJECTS:ecc_objects>
         $<TARGET_OBJECTS:polynomials_objects>
         $<TARGET_OBJECTS:plonk_objects>
         $<TARGET_OBJECTS:honk_objects>
         $<TARGET_OBJECTS:proof_system_objects>
         $<TARGET_OBJECTS:stdlib_primitives_objects>
         $<TARGET_OBJECTS:stdlib_schnorr_objects>
-        $<TARGET_OBJECTS:stdlib_pedersen_objects>
+        $<TARGET_OBJECTS:stdlib_pedersen_hash_objects>
+        $<TARGET_OBJECTS:stdlib_pedersen_commitment_objects>
         $<TARGET_OBJECTS:stdlib_blake2s_objects>
         $<TARGET_OBJECTS:stdlib_blake3s_objects>
         $<TARGET_OBJECTS:stdlib_sha256_objects>
@@ -174,15 +213,18 @@ else()
         $<TARGET_OBJECTS:crypto_blake3s_objects>
         $<TARGET_OBJECTS:crypto_keccak_objects>
         $<TARGET_OBJECTS:crypto_schnorr_objects>
-        $<TARGET_OBJECTS:crypto_pedersen_objects>
+        $<TARGET_OBJECTS:crypto_generators_objects>
+        $<TARGET_OBJECTS:crypto_pedersen_hash_objects>
+        $<TARGET_OBJECTS:crypto_pedersen_commitment_objects>
         $<TARGET_OBJECTS:ecc_objects>
         $<TARGET_OBJECTS:polynomials_objects>
         $<TARGET_OBJECTS:plonk_objects>
         $<TARGET_OBJECTS:honk_objects>
         $<TARGET_OBJECTS:proof_system_objects>
         $<TARGET_OBJECTS:stdlib_primitives_objects>
         $<TARGET_OBJECTS:stdlib_schnorr_objects>
-        $<TARGET_OBJECTS:stdlib_pedersen_objects>
+        $<TARGET_OBJECTS:stdlib_pedersen_hash_objects>
+        $<TARGET_OBJECTS:stdlib_pedersen_commitment_objects>
         $<TARGET_OBJECTS:stdlib_blake2s_objects>
         $<TARGET_OBJECTS:stdlib_blake3s_objects>
         $<TARGET_OBJECTS:stdlib_sha256_objects>

barretenberg/cpp/src/barretenberg/common/container.hpp

@@ -48,4 +48,16 @@ InnerCont flatten(Cont<InnerCont, Args...> const& in)
         result.insert(result.end(), e.begin(), e.end());
     }
     return result;
+}
+
+// Return the first index at which a given item can be found in the vector.
+// Only safe for vectors with length less than the size_t overflow size.
+template <typename T> long index_of(std::vector<T> const& vec, T const& item)
+{
+    auto const& begin = vec.begin();
+    auto const& end = vec.end();
+
+    auto const& itr = std::find(begin, end, item);
+
+    return itr == end ? -1 : std::distance(begin, itr);
 }

barretenberg/cpp/src/barretenberg/common/map.hpp

@@ -10,13 +10,29 @@ template <template <typename, typename...> typename Cont,
           typename... Args,
           typename F,
           typename OutElem = typename std::invoke_result<F, InElem const&>::type>
-Cont<OutElem> map(Cont<InElem, Args...> const& in, F op)
+Cont<OutElem> map(Cont<InElem, Args...> const& in, F&& op)
 {
     Cont<OutElem> result;
     std::transform(in.begin(), in.end(), std::back_inserter(result), op);
     return result;
 }
 
+/*
+ * Generic map function for mapping a std::array's elements to another type.
+ * TODO: this has only been added because I (Mike) couldn't get the above to work
+ * with an array.
+ */
+template <std::size_t SIZE,
+          typename InElem,
+          typename F,
+          typename OutElem = typename std::invoke_result<F, InElem const&>::type>
+std::array<OutElem, SIZE> map(std::array<InElem, SIZE> const& in, F&& op)
+{
+    std::array<OutElem, SIZE> result;
+    std::transform(in.begin(), in.end(), result.begin(), op);
+    return result;
+}
+
 /*
  * Generic map function for mapping a containers element to another type.
  * This version passes the element index as a second argument to the operator function.