Building an AI Agent - Open Source LLMs or GPT4, Anthropic & Co?

Dear All!

As you already know, I am currently preparing some simple AI Agents for Procurement. There are currently 3 in total:

• Finley, the Finance Assistant

• Negotiation Assistant

• Valuation Assistant for AI Projects 

You can already test Finley. The others will be added in the next few weeks. It takes a lot of effort to get everything on the right track. But I'm not giving up! On the contrary - I'm learning a lot every day.

Today I wanted to address a slightly different topic. As I've been very busy with prompting and Playground over the last few weeks, I've been able to immerse myself in the world of Open Source LLMs. I have to admit that I have completely neglected this area up to now. Unfortunately!

Since a few days I have an account on Hugging Face, a platform that I am sure many people are familiar with. For readers who haven't heard much about it, it's a platform that has a lot of open source LLMs, as well as many data sets for training these models. 

My main tools for configuring the AI agents were the OpenAI Assistant Playground and the Anthropic Console - both paid platforms. In my private tests, the costs are very manageable. After a few days, I spent about 10 USD on each of the playgrounds. But think a little further: if you are a medium-sized company, for example, and are thinking about AI Assistants in some areas (e.g. Document Automation), we quickly get into the five- to six-figure USD range. 

You can do the maths very easily. I used this calculator: https://docsbot.ai/tools/gpt-openai-api-pricing-calculator .

A small calculation example from our Procurement world: 

• 300 RFx processes per year

• each RFx document has approx. 5 pages

• we would like to have this document largely created with AI

• so we have approx. 300 tokens per 1 page and certainly approx. 10 API queries (prompts)

What are the costs?

• GPT4 Turbo: 48.000 USD

• GPT4: 103.500 USD

• Anthropic Claude 3 Opus: 112.500 USD

And this is where the open source LLMs come into play. What are their advantages:

• No costs for the LLM; of course you have to have your own infrastructure. Often even a laptop is enough!

• you can train and fine-tune the LLM with your own data

• high level of security because the data is stored on your servers

• Platforms such as Hugging Face offer low-code solutions for Agent creation

Of course someone can say: all well and good! But GPT4 or Anthropic are much more powerful and trained on vast amounts of data. Yes, that's all true. BUT we should be aware of which processes we want to optimise with the LLMs. If we are talking about relatively simple document automation, we probably don't need an LLM with 400 billion parameters. Probably 7 billion is enough. Of course, if we are doing scientific projects, we probably need a bit more :) But here the open source world has a lot to offer.

I have done a test myself. My Finley Assistant is in general on GPT4. To test Open Source LLMs I used the same instruction on Hugging Face Playground and used the model "mistralai/Mixtral-8x7B-Instruct-v0.1". And I was really amazed at how close the result was to the GPT4. Moreover, the Mixtral LLM understood the reporting structure correctly from the beginning, GPT4 needed some discussions with me :) On the other hand, GPT4 delivered slightly better analysis texts. Here is the link to the HF Finley.

But for making a decision about the financial situation of potential partners, it is perfectly adequate! We always have to make sure what we want to optimise with AI and act accordingly. Don't overdo it!

You can drive from A to B in a Rolls Roys. But a Renault Megane does it too :) A little less comfort perhaps. But the cost-benefit ratio will certainly be better :) 

But if you want to speed across the sand dunes, you'll need a Toyota Land Cruiser. You won't get far with a Megane ;) 

Tell me if you already have experience with Open Source LLMs? What do you think of them?