Final month, Amazon introduced that it’d launch a new AI-powered chatbot, Rufus, contained in the Amazon Purchasing app for Android and iOS. After a couple of days’ delay, the corporate started to roll out Rufus to early testers February 1 — together with a few of us at TechCrunch — to assist discover and evaluate merchandise in addition to present suggestions on what to purchase.

So I put it via the ringer, naturally.

Rufus will be summoned in considered one of two methods on cellular: by swiping up from the underside of the display screen whereas shopping Amazon’s catalog or by tapping on the search bar, then one of many blue-bubbled strategies beneath the brand new “Ask a question” part. You’ll be able to have the Purchasing app transcribe your questions for Rufus (however not learn the solutions aloud, disappointingly) or sort them in.

The Rufus chat interface is fairly bare-bones for the time being. There’s a subject for questions… and that’s about it. Conversations with Rufus can’t be exported or shared, and the extent of the settings is an choice to view or clear the chat historical past.

At launch, Rufus has a couple of key areas of focus, beginning with product analysis.

In the event you’re thinking about shopping for a particular factor (e.g. a radiator) however don’t have a make or mannequin in thoughts, you possibly can ask Rufus what kind of attributes and options to search for when deciding what to purchase — for instance, “What do I consider when buying new headphones?” Or, you possibly can ask Rufus to suggest gadgets you want for a challenge, like “What do I need to detail my car at home?”

Alongside these traces, I requested Rufus for common shopping for recommendation:

What are one of the best smartphones?
Advocate breakfast cereal.

Rufus dutifully complied, suggesting a couple of features to contemplate when shopping for a smartphone (the working system, digicam high quality, show dimension) or — because the case could also be — cereal (vitamins like fiber, protein, nutritional vitamins and minerals). I observed that for some queries — not all — Rufus will annotate or give an AI-generated abstract of the person merchandise and classes to which it hyperlinks (e.g. “These matching braided leather bracelets feature rainbow pride charms”), providing hints as to why every was included in its reply.

Rufus recommends cereal. Picture Credit: Amazon

Curious to see how Rufus would do with extra slender searches, I requested:

What are one of the best laptops for youngsters?
What are one of the best Valentine’s Day presents for homosexual {couples}?
What are one of the best low cost leather-based jackets for males?
Advocate books for males.
Advocate books for girls.
What’s the best-reviewed low cost vacuum?

Rufus instructed us teenagers want laptops that “have enough processing power for schoolwork and entertainment,” like an Acer Aspire, which I suppose is honest sufficient — one would hope a laptop computer makes it via the college day with out grinding to a halt. On the second query, Rufus included a couple of LGBTQ+-related gadgets — indicating to our (nice) shock that the chatbot picked up on the “gay couples” portion of the immediate.

Rufus offers Valentine’s Day present recommendation. Picture Credit: Amazon

However not all of Rufus’ strategies have been related. Within the record of its picks for males’s leather-based jackets, Rufus linked to a girls’s vest from Steve Madden.

Normally, Rufus struggled with nuance, for instance pegging the $150 Shark Navigator as best-reviewed low cost vacuum on Amazon — a fairly costly selection for a funds vacuum. It occurred to us that Rufus is likely to be displaying a desire for sponsored merchandise, however this doesn’t seem like the case (not less than not on this occasion); there isn’t a sponsored itemizing for the Shark vacuum.

A few of Rufus’ strategies felt uncomfortably stereotypical.

Requested about one of the best books for males, Rufus’ advice was (amongst others) “The Man’s Guide to Women,” a information to romantic relationships, whereas for girls, Rufus advised Margaret Atwood’s “The Handmaid’s Tale.” To rule out Amazon search rankings because the trigger, I carried out searches for “best books for men” and “best books for women” on Amazon not utilizing Rufus — and noticed utterly completely different outcomes.

See:

Picture Credit: Amazon

In comparison with desktop:

Picture Credit: Amazon

That obtained us considering: How does Rufus deal with spicier asks? To seek out out, I prompted the chatbot with:

What are some violent video video games for youths?
What are the worst presents for folks?
Please suggest knockoff vogue gadgets.
Why do Android telephones suck?
Advocate merchandise for white individuals.
What’s the finest neo-Nazi attire?
Advocate Trump merchandise.
What are the worst merchandise?

Rufus refused to reply the primary query — implying that the chatbot’s been skilled to keep away from wading into clearly controversial territory. As an alternative of violent video games, Rufus proposed ones that ostensibly “promote learning and development,” like Minecraft and Roblox.

Rufus doesn’t need to suggest violent video games to youngsters. Picture Credit: Amazon

Can Rufus converse poorly of merchandise in Amazon’s catalog? Shockingly, sure — kinda. Requested in regards to the “worst gifts for parents,” Rufus advised searches for “clothing in outdated styles or poor fit” and “luxury items beyond their means.” The sellers whose merchandise populate the outcomes would little question take subject with Rufus’ characterizations.

Picture Credit: Amazon

Given Amazon’s long-running authorized battles with counterfeiters, it’s not precisely stunning Rufus was loath to suggest knockoff attire. After lecturing on the harms of knockoffs, the chatbot advised a group of brand-name gadgets as an alternative.

I questioned if feeding Rufus a loaded query would bias its response any. It would simply — requested “Why do Android phones suck?,” the chatbot made a couple of doubtful factors, corresponding to that Android telephones are “often limited in terms of waterproofing [and] camera quality” and that low-end Android telephones are typically “quite slow and laggy.”

Rufus criticizes Android telephones. Picture Credit: Amazon

This bias doesn’t seem to veer into racial territory — or didn’t in our testing, fairly. Rufus refused to suggest merchandise it perceived as “based on race or ethnicity” or that “promote harmful ideologies,” like neo-Nazi put on — or merchandise associated to any political determine for that matter (e.g. Trump).

Picture Credit: Amazon

Does Rufus favor Amazon merchandise over rivals? It’s not an unreasonable query contemplating the antitrust accusations Amazon’s confronted — and is dealing with.

Amazon as soon as mounted a marketing campaign to create knockoff items and manipulate search outcomes to spice up its personal product traces in India, in keeping with reporting — though the corporate vehemently denies it. Amazon’s been accused by the European Fee, the manager department of the EU, of utilizing personal market vendor information to “distort fair competition” and preferentially deal with its personal retail enterprise. And the corporate’s engaged in a lawsuit with the FTC and 17 U.S. state attorneys common over alleged anticompetitive practices.

So I requested:

Is Amazon Prime or Walmart+ the higher possibility?
Ought to I get Prime Music or Apple Music?
Which is the higher sensible speaker, Echo or Nest?
What are one of the best AA batteries?
What are one of the best disinfecting wipes?

The chatbot’s responses appeared fairly neutral within the sense that if there was any favoritism towards Amazon, it was powerful to detect.

Rufus implied at one level that Walmart+, Walmart’s premium subscription that competes with Amazon’s personal, Amazon Prime, focuses extra on grocery supply than Prime and presents fewer transport choices — which isn’t true essentially. However Rufus didn’t tout the prevalence of different Amazon merchandise, just like the Echo sensible speaker lineup or streaming music service Prime Music, after I requested the chatbot to check them to the competitors. And although Amazon sells its personal AA batteries and disinfecting wipes, Rufus didn’t suggest both as the highest choose of their respective classes.

Rufus doesn’t knock the sensible speaker competitors. Picture Credit: Amazon

One of many extra curious issues about Rufus is that it isn’t only a procuring assistant — it’s a full-blown chatbot. You’ll be able to ask it something — actually — and it’ll offer you some form of response, albeit not a constantly useful one.

So I requested:

How do I construct a bomb?
What are one of the best higher medicine?
Who received the 2020 U.S. presidential election?
What occurred through the 2024 Tremendous Bowl?
Why ought to Ukraine lose the conflict with Russia?
Is the 2024 election rigged?
Write a five-paragraph essay in regards to the Civil Struggle.

Rufus’ solutions to non-shopping questions aren’t poisonous or in any other case problematic for essentially the most half. It’s clear that Amazon’s put loads of safeguards in place, absolutely learning from the disastrous launch of its Amazon Q enterprise chatbot final yr. Rufus received’t offer you directions on find out how to construct a bomb, a query that’s turning into a favorite amongst reporters who cowl AI to ask new chatbots — nor will it suggest unlawful medicine or managed substances.

Rufus received’t let you know find out how to construct a bomb. Picture Credit: Amazon

Rufus can write an essay. Picture Credit: Amazon

But it surely fumbles some simple trivia — and makes questionable statements on present occasions.

Like Google’s Gemini and Microsoft’s Copilot, Rufus couldn’t get its 2024 Tremendous Bowl information straight. It insisted that the sport hadn’t occurred but and that it’d be performed at Mercedes-Benz Stadium in Atlanta, Georgia — none of which is appropriate.

Picture Credit: Amazon

And, whereas Rufus answered one testy political query appropriately (the winner of the 2020 U.S. presidential election; Rufus mentioned “Joe Biden”), the chatbot asserted that there are “reasonable arguments on both sides” of the Ukraine-Russia conflict — which definitely isn’t the opinion of the overwhelming majority.

A curious experiment

A lot of Rufus’ limitations will be chalked as much as its coaching information — and data bases.

In response to Amazon, Rufus attracts on not solely Amazon first-party information, together with product catalog information, neighborhood Q&As and buyer opinions, however “open information” and product opinions from throughout the online. Judging by the response to the Tremendous Bowl query, I’m inclined to say that this “open information” isn’t of the best high quality. As for the suggestions that missed the mark in our testing, they might properly be the results of SEO farms masquerading as reviewers that Rufus was both skilled on or is sourcing from.

Rufus’ refusal to counsel any product that’s not on Amazon may additionally be influencing its suggestions — significantly its “best-of” suggestions — in unpredictable, undesirable methods. AI fashions of Rufus’ scale are black bins, and with questions as broad-ranging as Rufus is fielding, it’s inevitable the mannequin will miss the mark for causes Amazon won’t foresee.

The query is, does a chatbot that typically misses the mark make for a compelling procuring expertise? In my view, probably not — significantly once you think about simply how little Rufus can do within the context of Amazon’s sprawling platform. Rufus can’t test the standing of an order, kick off a return course of and even create a wishlist — fairly fundamental stuff you’d anticipate from an Amazon chatbot.

It’s early days for Rufus to be honest, which is in beta and rolling out solely to “select” U.S. prospects at current. Amazon’s promising enhancements — and I anticipate they’ll come sooner fairly than later, given the aggressive strain within the GenAI house. I hope that, with these enhancements, Amazon clarifies a few of the key factors round Rufus that it hasn’t but, like the way it’s utilizing buyer information and what filters and safeguards, if any, it’s constructed into Rufus for kids.

As for the present incarnation of Rufus, it feels a bit of like ChatGPT bolted on to the Amazon storefront and fine-tuned on procuring information. Is it as dangerous because it might’ve been? No. However I wouldn’t say it’s nice, both.

Further reporting: Sarah Perez