Testing simple examples with but seem to get stuck with ollama/llama3:instruct

[kevin-mitchell]

Sorry for what is admittedly a lazy question, but I was interested in checking out this project figured I'd just follow the simplest quick start guides to see what I could do with ollama and the llama3:instruct model by piecing together / trying a few simple examples (some with structured output, some without). In all of the cases I ended up with an error (that I assume is a catch all of some sort?) eventually before completing the task involving select_dropdown_option or action 'fill' with id= failed with error.

It seems like (and this still seems like a miracle to me!) things are working-ISH, in that when I watch the agent / browser it will, for example, go to the website I said, and identify different parts of the page, or even navigate to another page sometimes, but very quickly it seems to get "stuck" and based on the errors it's unclear to me if this is the model being confused / not good enough to do the job, or something else.

I very much appreciate the open source nature of this, for now I'm just "kicking the tires" and wanted to see what was possible using purely locally running models on my modest MBP laptop. I mentioned this was a lazy question because I haven't really dug into the code yet and it's entirely possible I'm simply doing this completely wrong (sorry in advance if so, and thanks for your time!)

Here are some specific examples with code.

Simple task on simple website

import notte
from dotenv import load_dotenv

load_dotenv()


with notte.Session(headless=False) as session:
    agent = notte.Agent(
        session=session, reasoning_model="ollama/llama3:instruct", max_steps=30
    )
    response = agent.run(
        task="go to kevinmitchell.io tell me the title of the homepage"
    )
print(response.answer)

results in

WARNING  - 🪟 No viewport set in headful mode, using default viewport in playwright
INFO     - 💡 Step 1
INFO     - Session url is 'about:blank': returning empty observation. Perform goto action before observing to get a more meaningful observation.
INFO     - 📝 Current page: unknown
INFO     - 🔬 Previous goal: ❓ unknown
INFO     - 🧠 Memory: unknown
INFO     - 🎯 Next goal: go to kevinmitchell.io tell me the title of the homepage
INFO     - 🆔 Relevant ids:
INFO     - ⚡ Taking action:
   ▶ goto with url=https://kevinmitchell.io
INFO     - Goto for url='https://kevinmitchell.io' succeeded with HTTP 200: 
INFO     - ✅ action 'goto' succeeded: 'Navigated to 'https://kevinmitchell.io' in current tab'


INFO     - 💡 Step 2
INFO     - 📝 Current page: Kevin Mitchell | kevinmitchell.io - Homepage with links to blog, resume, GitHub, and LinkedIn profiles
INFO     - 🔬 Previous goal: ❓ unknown
INFO     - 🧠 Memory: unknown
INFO     - 🎯 Next goal: Tell me the title of the homepage
INFO     - 🆔 Relevant ids:
   ▶ L1: Internal link to blog
   ▶ L2: Resume download link
   ▶ L3: External link to GitHub profile
INFO     - ⚡ Taking action:
   ▶ fill with id 
INFO     - ❌ action 'fill' with id= failed with error: Execution failed because interactive element is not found or not visibleHint: wait 5s and try again, check for any modal/dialog/popup that might be blocking the element, or try another action.


INFO     - 💡 Step 3
INFO     - 📝 Current page: Kevin Mitchell | kevinmitchell.io - Homepage with links to blog, resume, GitHub, and LinkedIn profiles
INFO     - 🔬 Previous goal: ❓ unknown
INFO     - 🧠 Memory: 
INFO     - 🎯 Next goal: Tell me the title of the homepage
INFO     - 🆔 Relevant ids:
   ▶ L1: Internal link to blog
   ▶ L3: External link to GitHub
INFO     - ⚡ Taking action:
   ▶ fill with id 
INFO     - ❌ action 'fill' with id= failed with error: Execution failed because interactive element is not found or not visibleHint: wait 5s and try again, check for any modal/dialog/popup that might be blocking the element, or try another action.


INFO     - 💡 Step 4
INFO     - 📝 Current page: Kevin Mitchell | kevinmitchell.io - Homepage with links to blog, resume, GitHub, and LinkedIn profiles
INFO     - 🔬 Previous goal: ❓ unknown
INFO     - 🧠 Memory: 
INFO     - 🎯 Next goal: Tell me the title of the homepage
INFO     - 🆔 Relevant ids:
INFO     - ⚡ Taking action:
   ▶ fill with id 
ERROR    - Error during agent run: Max consecutive failures reached in a single step: 3.
Traceback (most recent call last):

Using response_format with HackerNews example

import notte
from dotenv import load_dotenv
from pydantic import BaseModel

load_dotenv()


class HackerNewsPost(BaseModel):
    title: str
    url: str
    points: int
    author: str
    comments_count: int


class TopPosts(BaseModel):
    posts: list[HackerNewsPost]


with notte.Session(headless=False) as session:
    agent = notte.Agent(
        session=session, reasoning_model="ollama/llama3:instruct", max_steps=30
    )
    response = agent.run(
        task="Go to Hacker News (news.ycombinator.com) and extract the top 5 posts with their titles, URLs, points, authors, and comment counts.",
        response_format=TopPosts,
    )
print(response.answer)

Error:

INFO     - 💡 Step 2
INFO     - 📝 Current page: The current page appears to be a list of 30 articles or posts from Hacker News. Each post has a title, author, and number of comments.
INFO     - 🔬 Previous goal: ❓ unknown
INFO     - 🧠 Memory: The last action was not specified.
INFO     - 🎯 Next goal: Select the next goal and action based on the current page content.
INFO     - 🆔 Relevant ids:
   ▶ vote: Upvote the article
   ▶ hide: Hide the article
   ▶ comment: Leave a comment on the article
INFO     - ⚡ Taking action:
   ▶ select_dropdown_option with id next_goal
ERROR    - NotteBaseError: Execution failed because interactive element is not found or not visible (locator='css=#more')
INFO     - ❌ action 'select_dropdown_option' with id=next_goal failed with error: Execution failed because interactive element is not found or not visibleHint: wait 5s and try again, check for any modal/dialog/popup that might be blocking the element, or try another action.

HackerNews without structured output

import notte
from dotenv import load_dotenv

load_dotenv()


with notte.Session(headless=False) as session:
    agent = notte.Agent(
        session=session, reasoning_model="ollama/llama3:instruct", max_steps=30
    )
    response = agent.run(
        task="Go to Hacker News (news.ycombinator.com) tell me what the first post in the list is about.",
    )
print(response.answer)

INFO     - 🆔 Relevant ids:
   ▶ vote: Upvote or downvote a post
   ▶ hide: Hide a post from view
   ▶ comment: Leave a comment on a post
INFO     - ⚡ Taking action:
   ▶ select_dropdown_option with id vote
INFO     - ❌ action 'select_dropdown_option' with id=vote failed with error: An unexpected playwright error occurred: Locator.evaluate: Unexpected token "" while parsing css selector "Select a post to upvote or downvote.". Did you mean to CSS.escape it?
Call log:
  2 × waiting for Select a post to upvote or downvote.

[giordano-lucas]

Hey @kevin-mitchell tbh, reasoning capabilities of ollama/llama3:instruct are pretty bad (+ it's really slow). If I were you, I would use use a more powerful cloud hosted model like gemini, claude or openai. It will resolve all of your issues.

However, if you do have a specific use-case in mind that requires running these model on premise, feel free to book a meeting to discuss it