January 22, 2025
Using Gemini search metadata in a voice AI application
Filipi added support in @pipecat_ai for Google Gemini's `groundingMetadata`.
This makes it easy to do things like:
- link to URLs that Gemini searched
- log searches for use in your evals/observability tools
- use specific search result chunks as input for your internal RAG content extraction
Links to implementation and example code, below in the thread. Here's a video of Filipi interacting with the simple example in the PR.
The search metadata is encapsulated in an `LLMSearchResponseFrame`.
You can write a `FrameProcessor` to use the metadata within your @pipecat_ai pipeline, or send the metadata to the client. (Or both.)
Here's the PR that implements the `LLMSearchResponseFrame` and related pipeline code:
https://t.co/JG4MPQxqXn
Here's PR that adds support for sending these frames to the client:
https://t.co/MlerAzMaLN
Here's the simple example from the video above:
https://t.co/NuGpPDe3Os
Note that all of the above links are to PRs that haven't been merged to `main`, yet.
I 💚posting PRs publicly, to increase visibility of new, not-yet-shipped, features! Comments and further PRs are welcome, always.