June 21, 2024
Having lots of fun with Claude Sonnet today. Creativity is great. So is vision.
I wanted to do a quick time-to-first-token test.
Benchmarks are hard to do well. So my philosophy is to do them badly. And do them yourself, because most of the ones posted on the web are ... not terribly reliable.
Here are two Sonnet voice bots competing to invent whacky new ice cream flavors.
Median TTFT for short prompts is <500ms. I was running my bots locally so the API requests were made via hotel wifi. (See what I mean about doing benchmarks badly?)
Some of the ice cream flavors are fantabulous.
# [Durian and Limburger Cheese Charcoal Activated Tar Twist!]
# [Fermented Fish Sauce and Ghost Pepper Bubblegum Cotton Candy Nightmare!]
# [Spoiled Yogurt and Ghost Pepper Gummy Bear Blizzard!]
# [Matcha Green Tea and Sour Gummy Worm Fusion!]
Code for the ice cream bot is here[1]
Oh, also Sonnet wrote the js console code to keep track of ttft median, max, min, and number of samples!
Oh, and I forgot to credit the voices: "British Lady" and "Nonfiction Man" from @cartesia_ai.