Reviewing Relevance AI Automation Framework

In my search for the ultimate AI automation framework, I’ve so far reviewed n8n and Gumloop; today it’s time for a Relevance AI review.

ProfitSwarm is a challenge to make a 100% AI business. For this I need a kind of general structural piece of software which can rig all of the various agents, scripts, inputs, and triggers together.

I’m in search of a manager agent, or a framework to structure an AI agent team.

Relevance AI is the third of these AI workflow tools I’m trying, but I still have 4 on my list 😅, so if you’re new here, subscribe at the top of the page, or check back later – I’ll do a summary post once I’m done with my AI Workflow framework tests.

My Relevance AI Review

What is Relevance AI?

Relevance AI is a workflow designing tool, much like n8n or Gumloop. It offers you a way to wire together lots of tools in order to create automatic workflows.

Where Relevance is different, is that it builds each ‘workflow’ as an AI employee (agent). This means you end up with an AI team instead of a set of AI workflows.

Like the others, Relevance offers you a no-code/half-code editor where you can organise flows of actions. It doesn’t do this in a drag-and-drop canvas style, but rather each action is initiated by prompting their ‘agent builder’.

Whereas I could provide super simple examples of this for n8n and Gumloop, Relevance AI’s closest thing to a template directory is it’s pre-rolled agents. Here’s an example:

The steps of a Relevance workflow are more obscure than the other tools I’ve tried, because they operate agents not workflows directly.

Digging into “Elli’s” config:

… you can see the backbone of each AI agent is a meaty prompt. I both love this, and struggle with it, but more on that later.

On top of each agents prompt, they can also have “tools” and “sub-agents”. In this case the agent has tools:

Each tool can be referenced in the prompt by typing /, which is a nice touch.

As you can see in some ways Relevance AI is a simpler beast than the others. But it’s also as powerful once you dig into the details.

Let’s dig in and see whether prompt engineering beats workflow canvas alternatives.

What do I want to do with Relevance AI?

I’m testing Relevance AI to see if it’s a competent middle manager for my AI agent team. Further I’m looking to see how reliably it executes tasks, how easy it is to make agents that do what I want them too, and whether it’s worth paying extra for that AI-generating-AI layer that Relevance currently has over Gumloop and n8n.

What I’m hoping Relevance AI can help me with:

• Retrieving data with a search engine research API
• Collation and organising of search and trend data
• Matrixing potential viable business ideas
• Interacting with my architect API
• Social media marketing
• Content generation
• Email outreach
• Data enriching
• … and more

So far in my tests n8n and Gumloop have achieved some of this, but both hit slight blockers. Let’s see what Relevance AI can do.

First impressions of Relevance AI

Having started these tests on the open source n8n, the paid SaaS alternatives are seeming much cleaner. I love n8n, don’t get me wrong, but like Gumloop, Relevance AI feels more refined when you first try it.

Having hit issues signing into Gumloop with Github, I tried the same in Relevance AI. It failed too! But at least Relevance had a helpful message (Github email is private) 🤦. Anyhow, signup was otherwise easy and I quickly got started, before I knew it, I had it ‘inventing an agent’:

Like the other workflow tools I’ve reviewed, Relevance has a library of pre-written AI agents. I haven’t dipped into these much as I want very specific outcomes here, but they look solid and there’s a fair amount of them covering the basics.

Personally, I jumped straight in to making a new agent.

My first Impressions:

• Easy sign up
• Easy to jump straight into creating a new AI agent
• Was expecting a more ‘workflow canvas’ style, but actually the UI was easy to navigate and the concepts made sense
• Pricing seemed palatable (I have been thinking about team licensing), at 0->$20->$200+
• Credits system seems a little unintelligible

Relevance AI Review: Testing

In testing Gumloop I’d seen how far I could get building parts for my Directory Prospector (the first agent I need for my first experiment), so I did the same here on Relevance AI. It took some getting used to as I had started to gel with the workflow canvas style of Gumloop.

To begin with I didn’t have any idea as to what the nested structure of these agents needed to look like, so I dabbled.

Pretty soon I had something which almost worked.

Like the others, Relevance had it’s own hiccups.

Test Workflow: Directory Prospector

I jumped right in at a high level for this test – seeing whether Relevance AI could make the whole first agent for this experiment, doing these tasks:

For this Relevance AI review I wanted to see if one agent (and sub-agents) could take on the full task. In the other workflow tools I’d seen that individual parts are fairly achievable, but could Relevance combine them cleanly first time?

Here’s where I started:

1. Created a new AI agent “Directory Prospector” (prompt + settings)
2. Created a new “tool” for this AI agent to use (retrieve search engine data)

… and it kind of worked.

After iterating a few times on this tool, I got to see the ‘backend’ of each workflow, which is the Relevance AI equivalent of the workflow canvas drag-and-drop organisers of n8n/Gumloop.

Here you can see the API call I was doing to RapidAPI. It was such a pleasant site when I saw this UI, where the parameters were easy to add, and adding in the auth headers was common-sense, (this was less easy in Gumloop/n8n).

What’s more, you could test the step and see actual response data:

This made adding API calls super simple.

But this was also where I hit my first bug.

What the “AI agent inventor” had done is made this tool which:

1. Grabbed the data from the API
2. (Tried to) Format it in python
3. Returned the formatted data

As Relevance tried to write a smart formatting function, it basically would get itself stuck in a loop and eternally ‘Analyze the request’. This took longer than it should have for me to resolve.

In the end I grabbed the API output myself, put it through chatGPT and made a super simple python script to format the data. Here I think Relevance AI came close to being awesome, but it tripped over itself.

Relevance AI Review: Second Hiccup

It had started so well! Next I simplified my approach, trying to reduce the scope to just a Directory Ideator agent.

After a short while I did have an agent which would do approximately what I wanted. I was super impressed. Then I hit another wall:

In Relevance you interact with agents via a kind of chat window. I got this agent (Directory Ideator) working quite nicely, but then for some reason it just started giving me an update then going to get a cup of coffee and forgetting it had work to do.

If nudged the agent would carry on, but that’s hardly automated!

I went back into the settings and made sure it was setup to not take tea-breaks. I added the following to it’s prompt:

- Do not wait for approval at any step, work through it automatically

… and I upped these settings against the tools / subagents it used, (though I think it was the prompt which fixed it!)

Things started to work..

Fast forward a load of tweaks, and I had the following set-up and sort of working:

• Agent: Directory Ideator
  • Sub-agent: Keyword Researcher
  • Tool: Send to Slack

… this lean hierarchy would work, some of the time. I realised the limitations of this prompt-based agent approach – the results can be inconsistent.

Things I learned in this process:

• Don’t be afraid of splitting things up into simple tasks, each with an agent assigned – this lets you have more granular control and produce more reliable results
• Be hyper aware of Auto run and Approval settings in Relevance AI – it worked best for me when they’re basically set to Auto run and have high limits before approval is required (though this may be expensive in the long run)
• Keep working on your prompts – At first I was rough with my task descriptions, but through many iterations I got closer to working agents
• Try different AI models – Sometimes just shifting the AI model up a notch helped a lot
• As they stand Relevance AI agents are not great at dealing with large data sets

More Hiccups

Things were getting there, but I still hit a few more bugs/issues:

• Connecting Google Sheets was very buggy – Relevance AI’s account connection wizard repeatedly took 10 seconds to load, then would lose the Google Oauth connection randomly, often bugging out the other tools I had on the agent (e.g. it changed my Slack connection to use the Google Oauth?) I never got this to work
• Agents get stuck thinking (even on simple tasks) and seem to burn through credits until you kill them – This was beyond frustrating and near impossible to debug (as no thinking log is provided). It also cost a few thousand credits trying to fix it. There is an ‘Agent timeout’ function, but it sits behind the fat $599 a month business plan 🤦🏻‍♂️

Test AI Agent outcomes

I achieved about 80% of what I wanted in Relevance AI:

• I have agents doing their individual tasks well 80%+ of the time
• I’ve tried other search engine data sources, and have managed to reduce the steps needed in the Directory Prospector agent (no need for Google trends scraping!)
• The results are still inconsistent
• I still get inexplicable agent day-dreams that seem to cost a lot of credits and go nowhere
• I cannot seem to get Google sheets to connect

But it’s close to being great!

Will I continue to test after this Relevance AI review?

With each new workflow tool I test I see so much potential, but in all three so far I’ve hit irritating bumps in the road.

I’d love to be able to get what I want done in Relevance AI, but in lieu of good debugging / timeout features I’ll likely switch back to Gumloop. It’s hard to justify spending on a tool which randomly loses 2,000 credits overnight when you left nothing running.

If I can overcome a few of these pitfalls, though, it’d be great for this challenge.

My last ditch attempts will include:

• Making the agents use some sort of shared file system to overcome the data-size issue (hopefully)
• Re-tweak (for the 500th time) the prompts on the agents which are getting stuck
• Dump the heaviest AI models possible onto the same

Relevance AI Review: Pro’s

I like Relevance AI a lot, though it has annoyingly got blockers right now. This might not matter for you if you have tasks which are simpler, require less data, or you have the time (and cash) to spend days refining the prompts/agents. It might ‘just work’ with the extra features in Business plan (at $599 per month!)

Here’s what I love about Relevance AI right now:

• AI generating AI is sweet
• UI is lovely
• Connecting some services is super easy (e.g. Slack)
• Price seems okay ($19.99 is accessible, though credits do seem to bleed randomly)
• I like the concepts of Agents, Sub-Agents, and Tools
• In theory it has everything I need for this challenge

Relevance AI Review: Con’s

Overall Relevance AI is a capable tool, but I seem to keep hitting issues one way or another with these AI workflow tools. Maybe it’s just me.

Here’s what I think are the con’s for using Relevance AI to build a MAS (multi-agent system) right now (they may not matter to you):

• Agents got stuck in various states
• Agent crashes seem frequent and not sure why, tweaking makes little difference (and costs credits each test)
• Agent timeouts only available in biz level accounts (at $599 pm)
• Magically lost 2,000+ credits overnight even though nothing should have been running – no clear log of credits spent
• Connecting (Oauth) accounts is super slow/laggy/broken (Google)
• Agents seem to randomly fire and cost credits even when not asked to. Bulk queue works mystically and cancelling doesn’t always cancel
• Last run says “16 minutes ago” on agents even though they just fired and fired sub agents
• No scroll bars – For some reason these are removed from all UI, might look cool, but this is a bigger pain than I thought it’d be
• Can’t use your own openAI key (obfuscated pricing)

Relevance AI Review Summary

TL;DR; It’s awesome, but potentially expensive and not easy to debug.

In the end I do like Relevance AI. It’s close to being something I think will run a lot of business processes in the near future.

For my use case it fails on a few points, but I’m hoping the kinks get worked out so I can use a tool like this.

For now I’ll keep testing workflows in both this and Gumloop.

Relevance AI Review Final Thoughts

I think it’s worth trying an AI workflow tool like Relevance alongside a schema-based one like Gumloop, each of us and our use cases may find either works better for them.

If Relevance AI can fix the above issues, and provide better debugging for confused agents; I might be all in.

What’s really striking me now though, is why don’t these tools have version control?

What do you think of Relevance AI?

Have you used Relevance AI or another multi-agent system tool like it? I’d love to read your thoughts in the comments below.

Trying Relevance AI

If you’re interested in trying Relevance AI I’d appreciate if you tried it through the following link, I’ll get a small affiliate fee if you choose to pay for it 🙂