Skip to content

[native_assets_cli] BuildOutput extension: addDataAssetDirectories #2097

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 37 commits into from
Mar 20, 2025
Merged
Show file tree
Hide file tree
Changes from 28 commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
80af89c
add BuildOutput extensions addFoundCodeAssets and addDataAssetDirecto…
MichealReed Mar 12, 2025
28e4dd2
update changelog
MichealReed Mar 12, 2025
ec9b98c
should use targetOS
MichealReed Mar 12, 2025
ba20b2d
tests should use targetOS
MichealReed Mar 12, 2025
80f7576
mappings instead of names, libraries don't always match convention.
MichealReed Mar 12, 2025
42d26a3
fix comments
MichealReed Mar 12, 2025
6795c87
test nesting
MichealReed Mar 12, 2025
703a5c5
optional outputDirectory URI
MichealReed Mar 12, 2025
6c91856
normalize keys instead of edit generated file, needs new issue
MichealReed Mar 12, 2025
7da0b8f
use dependency from dart
MichealReed Mar 12, 2025
303e701
remove main from ffigen
MichealReed Mar 12, 2025
4e99e19
clean-up unnecessary show
MichealReed Mar 12, 2025
d78a647
show AddDataAssetsDirectoryExtension and GetLinkMode
MichealReed Mar 12, 2025
2836592
BuildOutputBuilder in changelog.
MichealReed Mar 12, 2025
82a4b05
remove AddFoundCodeAssets extension
MichealReed Mar 16, 2025
09cbb6b
remove AddFoundCodeAssets extension
MichealReed Mar 16, 2025
a9b87b9
remove GetLinkMode extension
MichealReed Mar 16, 2025
379a76e
remove addfound tests
MichealReed Mar 16, 2025
b76ec8f
dont add directories
MichealReed Mar 16, 2025
768827c
dont add directories
MichealReed Mar 16, 2025
590811f
remove directory tests
MichealReed Mar 16, 2025
8f29e61
remove addfoundlibrary from download_asset example
MichealReed Mar 16, 2025
7b0d8fb
update changelog
MichealReed Mar 16, 2025
ca389e5
no need for code asset import in config
MichealReed Mar 16, 2025
6423e27
remove config important from validation tests
MichealReed Mar 16, 2025
56cbc0b
actually add the data asset
MichealReed Mar 17, 2025
3cf0742
fix asset name
MichealReed Mar 17, 2025
7719d15
fix config imports
MichealReed Mar 17, 2025
a60dff5
restore download_asset
MichealReed Mar 18, 2025
af8218c
move to data_assets
MichealReed Mar 18, 2025
c21eba0
restore download_asset example
MichealReed Mar 18, 2025
8034a81
fix validation test, cleanup new extension
MichealReed Mar 18, 2025
7424003
restore base config.dart imports
MichealReed Mar 18, 2025
7d6c35d
addDependency directory
MichealReed Mar 18, 2025
2c316ea
remove import
MichealReed Mar 19, 2025
bc33b02
restore build_runner_test
MichealReed Mar 19, 2025
57d6d41
add directory uri
MichealReed Mar 20, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,34 +1,11 @@
// Copyright (c) 2024, the Dart project authors. Please see the AUTHORS file
// for details. All rights reserved. Use of this source code is governed by a
// BSD-style license that can be found in the LICENSE file.
import 'dart:io';

import 'package:native_assets_cli/data_assets.dart';
import 'package:native_assets_cli/native_assets_cli.dart';

void main(List<String> args) async {
await build(args, (input, output) async {
final assetDirectory = Directory.fromUri(
input.packageRoot.resolve('assets/'),
);
// If assets are added, rerun hook.
output.addDependency(assetDirectory.uri);

await for (final dataAsset in assetDirectory.list()) {
if (dataAsset is! File) {
continue;
}

// The file path relative to the package root, with forward slashes.
final name = dataAsset.uri
.toFilePath(windows: false)
.substring(input.packageRoot.toFilePath(windows: false).length);

output.assets.data.add(
DataAsset(package: input.packageName, name: name, file: dataAsset.uri),
);
// TODO(https://github.com/dart-lang/native/issues/1208): Report
// dependency on asset.
output.addDependency(dataAsset.uri);
}
await output.addDataAssetDirectories(['assets'], input: input);
});
}
1 change: 1 addition & 0 deletions pkgs/native_assets_cli/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

- Added validation that all URLs in the `Input` and `Output` of hooks are
absolute.
- Added `addDataAssetDirectories` extension method on `BuildOutputBuilder`.

## 0.11.0

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,12 +32,19 @@ void main(List<String> args) async {
outputDirectory,
);
final fileHash = await hashAsset(file);
final expectedHash =
assetHashes[createTargetName(
targetOS.name,
targetArchitecture.name,
iOSSdk?.type,
)];
final targetName = createTargetName(
targetOS.name,
targetArchitecture.name,
iOSSdk?.type,
);

// Create a lookup where the keys have their file extension removed.
final normalizedHashes = {
for (final entry in assetHashes.entries)
entry.key.substring(0, entry.key.lastIndexOf('.')): entry.value,
};

final expectedHash = normalizedHashes[targetName];
if (fileHash != expectedHash) {
throw Exception(
'File $file was not downloaded correctly. '
Expand Down
1 change: 1 addition & 0 deletions pkgs/native_assets_cli/lib/native_assets_cli.dart
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ export 'src/api/link.dart' show link;
export 'src/api/linker.dart' show Linker;
export 'src/config.dart'
show
AddDataAssetsDirectoryExtension,
BuildConfig,
BuildConfigBuilder,
BuildInput,
Expand Down
74 changes: 73 additions & 1 deletion pkgs/native_assets_cli/lib/src/config.dart
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,10 @@ import 'dart:io';
import 'package:crypto/crypto.dart' show sha256;
import 'package:pub_semver/pub_semver.dart';

import '../data_assets.dart';
import 'api/deprecation_messages.dart';
import 'encoded_asset.dart';
import 'hook/syntax.g.dart' as syntax;
import 'metadata.dart';
import 'utils/datetime.dart';
import 'utils/json.dart';

Expand Down Expand Up @@ -404,6 +404,78 @@ class BuildOutputBuilder extends HookOutputBuilder {
syntax.BuildOutput.fromJson(super._syntax.json);
}

extension AddDataAssetsDirectoryExtension on BuildOutputBuilder {
/// Extension on [BuildOutput] to handle data asset directories and files.
///
/// This extension provides a convenient way for build hooks to add
/// [DataAsset] dependencies from one or more directories or individual files.
///
/// If any specified path does not exist, a [FileSystemException] is thrown.
/// Any error during the directory listing is caught and rethrown with
/// additional context.
///
/// When recursive is set to true, the method will also add all subdirectories
/// and their files as dependencies.
Future<void> addDataAssetDirectories(
List<String> paths, {
required BuildInput input,
bool recursive = false,
}) async {
final packageName = input.packageName;
final packageRoot = input.packageRoot;
final rootPath = packageRoot.toFilePath(windows: false);

String assetName(Uri assetUri) =>
assetUri.toFilePath(windows: false).substring(rootPath.length);

for (final path in paths) {
final resolvedUri = packageRoot.resolve('$packageName/$path');
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we have $packageName/ here? Wouldn't we simply want to resolve $path?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Files created in the tests were showing up under $packageName, is there something else moving the assets to this path?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was kind of expecting this function to do what pkgs/native_assets_builder/test_data/simple_data_asset/hook/build.dart does, and that file being refactored to simply a call to this function.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How can test_data/simple_data_asset be tested? It depends on a nonexistent dart:asset and ByteAsset.

Pushed a fix for name that mirrors that example, we still must resolve the relative $packageName/$path to find the file though. Do you want this to be refactored for compatibility with system absolute paths or relative only?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The way to test for now is to run dart test in package:native_assets_builder.

we still must resolve the relative $packageName/$path to find the file though.

I don't understand, if I change the helper function to final resolvedUri = packageRoot.resolve(path);, then dart test in package:native_assets_builder succeeds.

Do you want this to be refactored for compatibility with system absolute paths or relative only?

I think relative paths in the package is fine for now. That's the most common use case. (For desktop applications that use an LLM that's installed on the system, I can imagine wanting to support absolute paths. So we could add such support in the future. We should be able to distinguish a string containing an absolute path from a relative path.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be good now, the data assets validation tests makeDataBuildInput was not resolving the packageRoot correctly, which caused the failure in the validation test. Now both simple_data_asset and the validation tests pass with resolve(path).

final directory = Directory.fromUri(resolvedUri);
final file = File.fromUri(resolvedUri);

if (await directory.exists()) {
try {
await for (final entity in directory.list(
recursive: recursive,
followLinks: false,
)) {
if (entity is File) {
assets.data.add(
DataAsset(
package: packageName,
name: assetName(entity.uri),
file: entity.uri,
),
);
addDependency(entity.uri);
}
}
} on FileSystemException catch (e) {
throw FileSystemException(
'Error reading directory "$path": ${e.message}',
directory.path,
e.osError,
);
}
} else if (await file.exists()) {
assets.data.add(
DataAsset(
package: packageName,
name: assetName(file.uri),
file: file.uri,
),
);
addDependency(file.uri);
} else {
throw FileSystemException(
'Path does not exist',
resolvedUri.toFilePath(windows: Platform.isWindows),
);
}
}
}
}

extension type EncodedAssetBuildOutputBuilder._(BuildOutputBuilder _output) {
/// Adds [EncodedAsset]s produced by this build.
///
Expand Down
102 changes: 102 additions & 0 deletions pkgs/native_assets_cli/test/data_assets/validation_test.dart
Original file line number Diff line number Diff line change
Expand Up @@ -108,4 +108,106 @@ void main() {
);
expect(errors, contains(contains('More than one')));
});

test('addDataAssetDirectories processes multiple directories', () async {
final input = makeDataBuildInput();
final outputBuilder = BuildOutputBuilder();

final assetsDir1Uri = packageRootUri.resolve('assets1');
final assetsDir1 = Directory.fromUri(assetsDir1Uri);
await assetsDir1.create(recursive: true);

final assetsDir2Uri = packageRootUri.resolve('assets2');
final assetsDir2 = Directory.fromUri(assetsDir2Uri);
await assetsDir2.create(recursive: true);

// Create a file in assets1.
final file1Uri = assetsDir1.uri.resolve('file1.txt');
final file1 = File.fromUri(file1Uri);
await file1.writeAsString('Hello World');

// Create a file in assets2.
final file2Uri = assetsDir2.uri.resolve('file2.txt');
final file2 = File.fromUri(file2Uri);
await file2.writeAsString('Hello Dart');

final output = BuildOutput(outputBuilder.json);
await outputBuilder.addDataAssetDirectories([
'assets1',
'assets2',
], input: input);

// Check that the files in both directories were added as dependencies.
expect(output.dependencies, contains(file1Uri));
expect(output.dependencies, contains(file2Uri));
});

test('addDataAssetDirectories processes one file', () async {
final input = makeDataBuildInput();
final outputBuilder = BuildOutputBuilder();

final assetsDirUri = packageRootUri.resolve('single_assets');
final assetsDir = Directory.fromUri(assetsDirUri);
await assetsDir.create(recursive: true);

// Create a file in the single_assets directory.
final fileUri = assetsDir.uri.resolve('single_file.txt');
final file = File.fromUri(fileUri);
await file.writeAsString('Test content');

final output = BuildOutput(outputBuilder.json);
await outputBuilder.addDataAssetDirectories([
'single_assets/single_file.txt',
], input: input);

// Check that the file in the directory was added as a dependency.
expect(output.dependencies, contains(fileUri));
});

test('addDataAssetDirectories processes nested directories', () async {
final input = makeDataBuildInput();
final outputBuilder = BuildOutputBuilder();

// Create top-level assets directory.
final assetsDirUri = packageRootUri.resolve('assets3');
final assetsDir = Directory.fromUri(assetsDirUri);
await assetsDir.create(recursive: true);

// Create nested subdirectory.
final nestedDirUri = assetsDir.uri.resolve('subdir');
final nestedDir = Directory.fromUri(nestedDirUri);
await nestedDir.create(recursive: true);

final nestedDir2Uri = nestedDir.uri.resolve('subdir2');
final nestedDir2 = Directory.fromUri(nestedDir2Uri);
await nestedDir2.create(recursive: true);

// Create a file in the top-level assets directory.
final fileTopUri = assetsDir.uri.resolve('top_file.txt');
final fileTop = File.fromUri(fileTopUri);
await fileTop.writeAsString('Top level file');

// Create a file in the nested subdirectory.
final nestedFileUri = nestedDir.uri.resolve('nested_file.txt');
final nestedFile = File.fromUri(nestedFileUri);
await nestedFile.writeAsString('Nested file');

// Create a file in the nested subdirectory.
final nestedFile2Uri = nestedDir2.uri.resolve('nested_file2.txt');
final nestedFile2 = File.fromUri(nestedFile2Uri);
await nestedFile2.writeAsString('Nested file 2');

final output = BuildOutput(outputBuilder.json);
await outputBuilder.addDataAssetDirectories(
['assets3'],
input: input,
recursive: true,
);

// Verify that the top-level directory, nested directory, and both files are
// added.
expect(output.dependencies, contains(fileTopUri));
expect(output.dependencies, contains(nestedFileUri));
expect(output.dependencies, contains(nestedFile2Uri));
});
}