Skip to content

Commit 77b4880

Browse files
cynthiajoannatebosch
cynthiajoan
and
natebosch
authored
feat(vertexai)!: Vertex AI in Firebase is now Generally Available (GA) and can be used in production apps. (#13453)
* chore(vertexai): fork vertexai sdk away from generative ai sdk. (#13298) * fix the sdk name and support platform * forking off model and chat class * fork api class * remove some export content since the next vertexai release will be breaking change. * fork function calling * fork off content * fix for analyzer * more analyzer fix * chore(vertexai): Add corresponding unit test for the fork (#13324) * Add corresponding unit test for the fork * fix issues from analyzer * feat(vertexai): add json schema support and update schema structure (#13378) * Add schema support for prompt response * update schema object, replace requiredProperties to optionalProperties * api updates (breaking) * more tweak during api review * Add format for Schema.string * fix test files * Update packages/firebase_vertexai/firebase_vertexai/lib/src/vertex_api.dart Co-authored-by: Nate Bosch <[email protected]> * add review feedback --------- Co-authored-by: Nate Bosch <[email protected]> * feat(vertexai): update function calling and sample (#13395) * Add schema support for prompt response * update schema object, replace requiredProperties to optionalProperties * api updates (breaking) * more tweak during api review * Add format for Schema.string * fix test files * first sample update * remove embedding * api update for function calling * Update sample code to match the new api * more code snippet for toolconfig * apply documentation update * tweak of the tools * remove embed related class * Update packages/firebase_vertexai/firebase_vertexai/example/lib/main.dart Co-authored-by: Nate Bosch <[email protected]> * Update packages/firebase_vertexai/firebase_vertexai/example/lib/main.dart Co-authored-by: Nate Bosch <[email protected]> * address review comments * more fix * keep name consistency * keep using getWeather to keep align with devsite documentation * Update the static name --------- Co-authored-by: Nate Bosch <[email protected]> * feat(vertexai): update enums and citation and DataPart (#13410) * Update SafeRating * Rename citation source * add blocked for SafetyRating parse * Rename DataPart to InlineDataPart * Update readme * apply review comment * feat(vertexai): Error message for service api not enabled. (#13435) * Error message for service api not enabled. * add quota exceed test * Apply suggestions from code review Co-authored-by: Nate Bosch <[email protected]> * Update for review comments --------- Co-authored-by: Nate Bosch <[email protected]> * chore(vertexai): remove vertex_ file name for most of files (#13439) * remove vertex_ file name for most of files * organize import * fix the optional attributes while parsing SafetyRating (#13452) * chore(vertexai): function calling sample update (#13492) * fix the optional attributes while parsing SafetyRating * add doc comments * update function sample * comments * Shorten lines over 80 columns (#13493) Reflow comments and break long strings into adjacent string literals. --------- Co-authored-by: Nate Bosch <[email protected]>
1 parent f010b46 commit 77b4880

29 files changed

+3321
-1362
lines changed

‎CHANGELOG.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1926,7 +1926,7 @@ Packages with other changes:
19261926

19271927
#### `firebase_vertexai` - `v0.1.0`
19281928

1929-
- Initial release of the Vertex AI for Firebase SDK (public preview). Learn how to [get started](https://firebase.google.com/docs/vertex-ai/get-started) with the SDK in your app.
1929+
- Initial release of the Vertex AI in Firebase SDK (public preview). Learn how to [get started](https://firebase.google.com/docs/vertex-ai/get-started) with the SDK in your app.
19301930

19311931

19321932
## 2024-05-07

‎packages/firebase_app_check/firebase_app_check_web/pubspec.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
name: firebase_app_check_web
2-
description: The web implementation of firebase_auth
2+
description: The web implementation of firebase_app_check
33
homepage: https://github.com/firebase/flutterfire/tree/main/packages/firebase_app_check/firebase_app_check_web
44
version: 0.1.3+2
55

‎packages/firebase_vertexai/firebase_vertexai/CHANGELOG.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -62,4 +62,4 @@
6262

6363
## 0.1.0
6464

65-
- Initial release of the Vertex AI for Firebase SDK (public preview). Learn how to [get started](https://firebase.google.com/docs/vertex-ai/get-started) with the SDK in your app.
65+
- Initial release of the Vertex AI in Firebase SDK (public preview). Learn how to [get started](https://firebase.google.com/docs/vertex-ai/get-started) with the SDK in your app.

‎packages/firebase_vertexai/firebase_vertexai/README.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,15 @@
1-
# Vertex AI for Firebase Flutter
1+
# Vertex AI in Firebase Flutter
22
[![pub package](https://img.shields.io/pub/v/firebase_vertexai.svg)](https://pub.dev/packages/firebase_vertexai)
33

44
A Flutter plugin to use the [Vertex AI](https://firebase.google.com/docs/vertex-ai/).
55

66
To learn more about Vertex AI, please visit the [website](https://cloud.google.com/vertex-ai)
77

8-
**Preview**: Vertex AI for Firebase is in Public Preview, which means that the product is not subject to any SLA or deprecation policy and could change in backwards-incompatible ways.
8+
**Generally Available**: Vertex AI in Firebase is now Generally Available (GA) and can be used in production apps
99

1010
## Getting Started
1111

12-
To get started with Vertex AI for Firebase Flutter, please [see the documentation](https://firebase.google.com/docs/vertex-ai/get-started?platform=flutter).
12+
To get started with Vertex AI in Firebase Flutter, please [see the documentation](https://firebase.google.com/docs/vertex-ai/get-started?platform=flutter).
1313

1414
## Usage
1515

‎packages/firebase_vertexai/firebase_vertexai/example/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# firebase_vertexai_example
22

3-
Sample app to show how to use Vertex AI for Firebase.
3+
Sample app to show how to use Vertex AI in Firebase.
44

55
## Getting Started
66

‎packages/firebase_vertexai/firebase_vertexai/example/lib/main.dart‎

Lines changed: 149 additions & 60 deletions
Original file line numberDiff line numberDiff line change
@@ -91,52 +91,66 @@ class _ChatWidgetState extends State<ChatWidget> {
9191

9292
initFirebase().then((value) {
9393
_model = FirebaseVertexAI.instance.generativeModel(
94-
model: 'gemini-1.5-flash-preview-0514',
94+
model: 'gemini-1.5-flash',
9595
);
9696
_functionCallModel = FirebaseVertexAI.instance.generativeModel(
97-
model: 'gemini-1.5-flash-preview-0514',
97+
model: 'gemini-1.5-flash',
9898
tools: [
99-
Tool(functionDeclarations: [exchangeRateTool]),
99+
Tool.functionDeclarations([fetchWeatherTool]),
100100
],
101101
);
102102
_chat = _model.startChat();
103103
});
104104
}
105105

106-
Future<Map<String, Object?>> findExchangeRate(
107-
Map<String, Object?> arguments,
108-
) async =>
109-
// This hypothetical API returns a JSON such as:
110-
// {"base":"USD","date":"2024-04-17","rates":{"SEK": 0.091}}
111-
{
112-
'date': arguments['currencyDate'],
113-
'base': arguments['currencyFrom'],
114-
'rates': <String, Object?>{arguments['currencyTo']! as String: 0.091},
115-
};
116-
117-
final exchangeRateTool = FunctionDeclaration(
118-
'findExchangeRate',
119-
'Returns the exchange rate between currencies on given date.',
120-
Schema(
121-
SchemaType.object,
122-
properties: {
123-
'currencyDate': Schema(
124-
SchemaType.string,
125-
description: 'A date in YYYY-MM-DD format or '
126-
'the exact value "latest" if a time period is not specified.',
127-
),
128-
'currencyFrom': Schema(
129-
SchemaType.string,
130-
description: 'The currency code of the currency to convert from, '
131-
'such as "USD".',
132-
),
133-
'currencyTo': Schema(
134-
SchemaType.string,
135-
description: 'The currency code of the currency to convert to, '
136-
'such as "USD".',
137-
),
138-
},
139-
),
106+
// This is a hypothetical API to return a fake weather data collection for
107+
// certain location
108+
Future<Map<String, Object?>> fetchWeather(
109+
double latitude,
110+
double longitude,
111+
String date,
112+
) async {
113+
// TODO(developer): Call a real weather API.
114+
// Mock response from the API. In developer live code this would call the
115+
// external API and return what that API returns.
116+
final apiResponse = {
117+
'location': '$latitude, $longitude',
118+
'date': date,
119+
'temperature': 38,
120+
'chancePrecipitation': '56%',
121+
'cloudConditions': 'partly-cloudy',
122+
};
123+
return apiResponse;
124+
}
125+
126+
/// Actual function to demonstrate the function calling feature.
127+
final fetchWeatherTool = FunctionDeclaration(
128+
'fetchWeather',
129+
'Get the weather conditions for a specific city on a specific date.',
130+
parameters: {
131+
'location': Schema.object(
132+
description: 'The longitude and latitude of the city for which to get '
133+
'the weather. Must always be a nested object of '
134+
'`longitude` and `latitude`. The values must be floats.',
135+
properties: {
136+
'latitude': Schema.number(
137+
format: 'float',
138+
description: 'A numeric value indicating the latitude of the '
139+
'desired location between -90 and 90',
140+
),
141+
'longitude': Schema.number(
142+
format: 'float',
143+
description:
144+
'A numeric value indicating the longitude of the desired '
145+
'location between -180 and 180',
146+
),
147+
},
148+
),
149+
'date': Schema.string(
150+
description: 'The date for which to get the weather. '
151+
'Date must be in the format: YYYY-MM-DD.',
152+
),
153+
},
140154
);
141155

142156
Future<void> initFirebase() async {
@@ -274,6 +288,20 @@ class _ChatWidgetState extends State<ChatWidget> {
274288
: Theme.of(context).colorScheme.primary,
275289
),
276290
),
291+
IconButton(
292+
tooltip: 'schema prompt',
293+
onPressed: !_loading
294+
? () async {
295+
await _promptSchemaTest(_textController.text);
296+
}
297+
: null,
298+
icon: Icon(
299+
Icons.schema,
300+
color: _loading
301+
? Theme.of(context).colorScheme.secondary
302+
: Theme.of(context).colorScheme.primary,
303+
),
304+
),
277305
if (!_loading)
278306
IconButton(
279307
onPressed: () async {
@@ -304,6 +332,51 @@ class _ChatWidgetState extends State<ChatWidget> {
304332
);
305333
}
306334

335+
Future<void> _promptSchemaTest(String subject) async {
336+
setState(() {
337+
_loading = true;
338+
});
339+
try {
340+
final content = [Content.text('Create a list of 20 $subject.')];
341+
342+
final response = await _model.generateContent(
343+
content,
344+
generationConfig: GenerationConfig(
345+
responseMimeType: 'application/json',
346+
responseSchema: Schema.array(
347+
items: Schema.string(
348+
description: 'A single word that a player will need to guess.',
349+
),
350+
),
351+
),
352+
);
353+
354+
var text = response.text;
355+
_generatedContent.add((image: null, text: text, fromUser: false));
356+
357+
if (text == null) {
358+
_showError('No response from API.');
359+
return;
360+
} else {
361+
setState(() {
362+
_loading = false;
363+
_scrollDown();
364+
});
365+
}
366+
} catch (e) {
367+
_showError(e.toString());
368+
setState(() {
369+
_loading = false;
370+
});
371+
} finally {
372+
_textController.clear();
373+
setState(() {
374+
_loading = false;
375+
});
376+
_textFieldFocus.requestFocus();
377+
}
378+
}
379+
307380
Future<void> _sendStorageUriPrompt(String message) async {
308381
setState(() {
309382
_loading = true;
@@ -358,8 +431,8 @@ class _ChatWidgetState extends State<ChatWidget> {
358431
Content.multi([
359432
TextPart(message),
360433
// The only accepted mime types are image/*.
361-
DataPart('image/jpeg', catBytes.buffer.asUint8List()),
362-
DataPart('image/jpeg', sconeBytes.buffer.asUint8List()),
434+
InlineDataPart('image/jpeg', catBytes.buffer.asUint8List()),
435+
InlineDataPart('image/jpeg', sconeBytes.buffer.asUint8List()),
363436
]),
364437
];
365438
_generatedContent.add(
@@ -444,29 +517,35 @@ class _ChatWidgetState extends State<ChatWidget> {
444517
setState(() {
445518
_loading = true;
446519
});
447-
final chat = _functionCallModel.startChat();
448-
const prompt = 'How much is 50 US dollars worth in Swedish krona?';
520+
final functionCallChat = _functionCallModel.startChat();
521+
const prompt = 'What is the weather like in Boston on 10/02 this year?';
449522

450523
// Send the message to the generative model.
451-
var response = await chat.sendMessage(Content.text(prompt));
524+
var response = await functionCallChat.sendMessage(
525+
Content.text(prompt),
526+
);
452527

453528
final functionCalls = response.functionCalls.toList();
454529
// When the model response with a function call, invoke the function.
455530
if (functionCalls.isNotEmpty) {
456531
final functionCall = functionCalls.first;
457-
final result = switch (functionCall.name) {
458-
// Forward arguments to the hypothetical API.
459-
'findExchangeRate' => await findExchangeRate(functionCall.args),
460-
// Throw an exception if the model attempted to call a function that was
461-
// not declared.
462-
_ => throw UnimplementedError(
463-
'Function not implemented: ${functionCall.name}',
464-
)
465-
};
466-
// Send the response to the model so that it can use the result to generate
467-
// text for the user.
468-
response = await chat
469-
.sendMessage(Content.functionResponse(functionCall.name, result));
532+
if (functionCall.name == 'fetchWeather') {
533+
Map<String, dynamic> location =
534+
functionCall.args['location']! as Map<String, dynamic>;
535+
var date = functionCall.args['date']! as String;
536+
var latitude = location['latitude']! as double;
537+
var longitude = location['longitude']! as double;
538+
final functionResult = await fetchWeather(latitude, longitude, date);
539+
// Send the response to the model so that it can use the result to
540+
// generate text for the user.
541+
response = await functionCallChat.sendMessage(
542+
Content.functionResponse(functionCall.name, functionResult),
543+
);
544+
} else {
545+
throw UnimplementedError(
546+
'Function not declared to the model: ${functionCall.name}',
547+
);
548+
}
470549
}
471550
// When the model responds with non-null text content, print it.
472551
if (response.text case final text?) {
@@ -483,11 +562,21 @@ class _ChatWidgetState extends State<ChatWidget> {
483562
});
484563

485564
const prompt = 'tell a short story';
486-
var response = await _model.countTokens([Content.text(prompt)]);
487-
print(
488-
'token: ${response.totalTokens}, billable characters: ${response.totalBillableCharacters}',
489-
);
490-
565+
var content = Content.text(prompt);
566+
var tokenResponse = await _model.countTokens([content]);
567+
final tokenResult = 'Count token: ${tokenResponse.totalTokens}, billable '
568+
'characters: ${tokenResponse.totalBillableCharacters}';
569+
_generatedContent.add((image: null, text: tokenResult, fromUser: false));
570+
571+
var contentResponse = await _model.generateContent([content]);
572+
final contentMetaData = 'result metadata, promptTokenCount:'
573+
'${contentResponse.usageMetadata!.promptTokenCount}, '
574+
'candidatesTokenCount:'
575+
'${contentResponse.usageMetadata!.candidatesTokenCount}, '
576+
'totalTokenCount:'
577+
'${contentResponse.usageMetadata!.totalTokenCount}';
578+
_generatedContent
579+
.add((image: null, text: contentMetaData, fromUser: false));
491580
setState(() {
492581
_loading = false;
493582
});

‎packages/firebase_vertexai/firebase_vertexai/lib/firebase_vertexai.dart‎

Lines changed: 18 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -12,25 +12,13 @@
1212
// See the License for the specific language governing permissions and
1313
// limitations under the License.
1414

15-
export 'src/firebase_vertexai.dart'
15+
export 'src/api.dart'
1616
show
17-
// TODO(next breaking): Remove defaultTimeout
18-
defaultTimeout,
19-
FirebaseVertexAI,
20-
RequestOptions;
21-
export 'src/vertex_api.dart'
22-
show
23-
BatchEmbedContentsResponse,
2417
BlockReason,
2518
Candidate,
2619
CitationMetadata,
27-
CitationSource,
28-
ContentEmbedding,
20+
Citation,
2921
CountTokensResponse,
30-
// TODO(next breaking): Remove CountTokensResponseFields
31-
CountTokensResponseFields,
32-
EmbedContentRequest,
33-
EmbedContentResponse,
3422
FinishReason,
3523
GenerateContentResponse,
3624
GenerationConfig,
@@ -41,29 +29,31 @@ export 'src/vertex_api.dart'
4129
SafetyRating,
4230
SafetySetting,
4331
TaskType,
44-
// TODO(next breaking): Remove parse* methods
45-
parseCountTokensResponse,
46-
parseEmbedContentResponse,
47-
parseGenerateContentResponse;
48-
export 'src/vertex_chat.dart' show ChatSession, StartChatExtension;
49-
export 'src/vertex_content.dart'
32+
UsageMetadata;
33+
export 'src/chat.dart' show ChatSession, StartChatExtension;
34+
export 'src/content.dart'
5035
show
5136
Content,
52-
DataPart,
37+
InlineDataPart,
5338
FileData,
5439
FunctionCall,
5540
FunctionResponse,
5641
Part,
57-
TextPart,
58-
// TODO(next breaking): Remove parseContent
59-
parseContent;
60-
export 'src/vertex_function_calling.dart'
42+
TextPart;
43+
export 'src/error.dart'
44+
show
45+
VertexAIException,
46+
VertexAISdkException,
47+
InvalidApiKey,
48+
ServerException,
49+
UnsupportedUserLocation;
50+
export 'src/firebase_vertexai.dart' show FirebaseVertexAI;
51+
export 'src/function_calling.dart'
6152
show
6253
FunctionCallingConfig,
6354
FunctionCallingMode,
6455
FunctionDeclaration,
65-
Schema,
66-
SchemaType,
6756
Tool,
6857
ToolConfig;
69-
export 'src/vertex_model.dart' show GenerativeModel;
58+
export 'src/model.dart' show GenerativeModel;
59+
export 'src/schema.dart' show Schema, SchemaType;

0 commit comments

Comments
 (0)