[firebase_ai]: Unhandled format for Content: {}

[devberkay]

Is there an existing issue for this?

• I have searched the existing issues.

Which plugins are affected?

Other

Which platforms are affected?

No response

Description

Unforunately as of now, to my experience, package is broken on anything more than basic chat response generation . Error is always same. attached below.

firebase_ai 2.1.0

To reproduce try any model with tool call generating json response, around %50 of the tool calls are failing with this error:

[log] AI Error: Error details: Unhandled format for Content: {}
      This indicates a problem with the Vertex AI in Firebase SDK. Try updating to the latest version (https://pub.dev/packages/firebase_ai/versions), or file an issue at https://github.com/firebase/flutterfire/issues.

2.5 models are worst , they are generating this response much more frequent than other models

I also tried settings SafetySettings to none as i saw another bug report causing this but no it didnt solve.

Reproducing the issue

To reproduce try any model with tool call generating json response, around %50 of the tool calls are failing with this error:

Firebase Core version

3.14.0

Flutter Version

3.32.0

Relevant Log Output

[log] AI Error: Error details: Unhandled format for Content: {}
      This indicates a problem with the Vertex AI in Firebase SDK. Try updating to the latest version (https://pub.dev/packages/firebase_ai/versions), or file an issue at https://github.com/firebase/flutterfire/issues.

Flutter dependencies

Expand Flutter dependencies snippet

Replace this line with the contents of your `flutter pub deps -- --style=compact`.

Additional context and comments

No response

[SelaseKay]

Hi @devberkay, kindly provide a complete minimal sample that reproduces this issue.

[atebitftw]

@SelaseKay I have the similar issue when prompting to generate an image in response to an image that I send.

So this works...

(flutter/dart web app, sourcing image data from a webcam capture)

final XFile result = await cameraController.takePicture();
final Uint8List bytes = await result.readAsBytes();
final List<Content> prompt = [
  Content.text(
      'Tell me about this image.',
    ),
  Content.inlineData(
      'image/png',
      bytes,
    ),
  ];
  
 final response = await model.generateContent(prompt);
 print(response.text);

The AI SDK will return a text description of the image that was sent along with the prompt.

However, using the same exact code, if I change the prompt to "Generate a new image with the image provided, but put a spaceship somewhere in the image."

I get this exception:

DartError: Unhandled format for Content: {role: model}
This indicates a problem with the Vertex AI in Firebase SDK. Try updating to the latest version (https://pub.dev/packages/firebase_ai/versions), or file an issue at https://github.com/firebase/flutterfire/issues.
dart-sdk/lib/_internal/js_dev_runtime/private/ddc_runtime/errors.dart 266:3        throw_
errors.dart:266
package:firebase_ai/src/developer/api.dart 302:10                                  <fn>
api.dart:302
package:firebase_ai/src/developer/api.dart 295:10                                  _parseGoogleAIContent
api.dart:295
package:firebase_ai/src/developer/api.dart 185:11                                  _parseCandidate
api.dart:185
dart-sdk/lib/_internal/js_dev_runtime/private/ddc_runtime/operations.dart 117:77   tear

I am not registered or using the Vertex AI, just the Gemini Developer API in Firebase. Is this a matter of registering with the Vertex API as well?

UPDATE: I activated the Vertex API in firebase but any time I prompt with "Generate an image..." I get the above.

UPDATE 2: I tried to use the model and modality configuration from this documentation:
https://firebase.google.com/docs/ai-logic/generate-images-gemini?api=dev#edit-images

Like so:

  final model = FirebaseAI.googleAI().generativeModel(
    generationConfig: GenerationConfig(
      candidateCount: 1,
      responseModalities: [ResponseModalities.text, ResponseModalities.image],
    ),

    //model: 'gemini-2.5-flash',
    model: 'gemini-2.0-flash-preview-image-generation',
  );

But I receive this error, using the same code and prompt from above:

DartError: UnimplementedError: inlineData content part not yet supported

Is this just an issue of the firebase_ai Dart library needing more work, or is it intended that this functionality will not be supported? Could I use JS-Interop for now as a workaround?

[rubendepaz1]

same here, Error: Unhandled format for Content: {role: model}. The same prompt, model, parameters, and context produces an error or a good response with simply a small change in the user question injected in the prompt like: "how was the last time i ran 10km" -> response ok. "i'm a bit fatigued today" -> Error: Unhandled format for Content: {role: model} (the context is a json with training data history for the user/athlete)

[sydneyagcaoili]

I got same problem, and it only happens when I expect the AI to respond with "Function Calls"

[doughywilson]

Same problem. I get this issue whenever the model tries to call a function.

[atebitftw]

As far as I can tell from the docs, the API does not yet support "image transformation" or as openai calls it "image edit" capability. My workaround for now was to switch over to openai apis, which is disappointing because I'd rather have all of this capability within firebase/gemini.

[sydneyagcaoili]

I figured in my case, the issue only happens when:

• The prompt input is expected to respond with functionCall based on instructions
• Your GenerationConfig is set. e.g. FirebaseAI.vertexAI().generativeModel(generationConfig: ...)

So when I removed the config in the params, everything worked fine again.

...Weirdly, I get a different behavior with the same situation as above using FirebaseAI**.googleAI().**generativeModel(generationConfig: ...) -> sendChatMessage/(Stream) just does not respond.

I got same problem, and it only happens when I expect the AI to respond with "Function Calls"

[doughywilson]

This seems to be a problem when calling functions that have a fair amount of schema to them. For example, my code has a simple "save" function call, and one that is more complex. This issue only happens on the complex function call.

@SelaseKay I think this one might be hard to generate a simple code example, given the need to integrate credentials with the code. Do we really need to mark this issue as "blocked by customer response" until you have this? This is what I could deem to be a critical issue with firebase_ai that certainly prevents my use of the package for production.

[doughywilson]

Another discovery, I am NOT having this problem when I change my AI model from "gemini-2.5-flash" to "gemini-2.5-pro". So, it is likely that the model itself is spitting out garbage function calls. I was going to try to increase the thinking budget on flash to see if that helped this issue, but it appears this library does not yet support doing that.