Amazon’s Solution to Tough AI Cloud Margins: Non-Nvidia Chips

Welcome back! 

Salesforce’s annual customer conference Dreamforce is under way, so check back in the coming days for our coverage of the event and what it means for artificial intelligence in enterprises.

Now on to today’s column…

Last week, our report on the razor-thin gross profit margins Oracle generates from renting out Nvidia chips to AI developers raised questions about how the rest of the cloud industry is faring.

The most profitable cloud provider, Amazon, seems to have found one way to improve some of its AI cloud margins: custom server chips. In a surprise, Amazon Web Services Chief Marketing Officer Julia White said Monday on The Information’s TITV that the cloud provider was powering a majority of its Bedrock AI service using Amazon Trainium chips.

Bedrock lets customers access AI from Anthropic and other providers through an application programming interface, but customers don’t pick which chips power the API.

White’s revelation is significant because it shows the extent to which Amazon is relying on in-house technology to potentially get better gross profit margins from AI, as Trainium chips are less expensive than Nvidia graphics processing units. (Trainium isn’t a GPU but is designed to speed up AI model training and inference).

The fact that a majority of Bedrock runs on Trainium suggests that a significant number of AWS customers don’t require Nvidia chips—which are generally more powerful than Trainium chips—to run AI computing jobs on AWS. 

AWS executives often talk about turning the “building blocks” of computing, from servers to cloud software, into cheap commodities. But it wasn’t previously clear if this strategy could carry over from traditional cloud computing to the AI era.