HACKATHON - Agent Eval Harness (by the Friendly Data Meetup)
about this event
SIGN UP HERE 👉 https://luma.com/h8zer2ek
Hi new friends and old! The Friendly Data Meetup Hackathon is back by popular demand! 🎉 After the brilliant turnout (and the even more brilliant demos) at our first hackathon in January, we simply had to do it again. 👩💻
This evening is all about learning, building, and having fun with friendly faces. No pressure, no pitch decks, just inquisitive minds making cool things with data. With perhaps a slight competitive edge for those seeking glory 😉
📍 We're very grateful to **Quantspark** for kindly hosting us at their venue for the evening!
**💡 The challenge: Judge the Agent ⚖️**
*Dawdle Deliveries'* shiny new AI customer support bot goes live next week. Is it actually ready? Your job is to find out, with evidence instead of vibes! AI agents are moving into production, and everyone is asking the same question: how do you know it works? Together, we'll find out by building an evaluation harness from scratch: - 🧪 **Write the tests:** a set of realistic (and sneaky) customer questions - ✅ **Build the checks:** automatic pass/fail checks, plus an AI judge that marks every answer - 🔧 **Fix it and prove it:** improve the bot, then show a before/after scorecard proving your fix worked (without breaking anything else!)
The challenge is split into levels, so every team can work at its own pace, and there are bonus rounds (tracing, red-teaming, model shoot-outs) for the speedy ones. We'll finish with quick 3-minute demos: your scorecard, your most interesting failure, and the big question: *do you trust your judge?* 🤔
Teams of 2–4 will be formed on the night, so don't worry if you're coming solo.
**✅ Pre-flight checklist** This hackathon is very hands-on, so it's best suited to people who already write code and/or have some working knowledge with data and AI. You'll get the most out of the evening if you: - Have decent working experience writing code - Are very comfortable with git, Docker and Python environments - Are happy to dig in and figure things out with your team (organisers will be around to help, but there won't be step-by-step hand-holding)
**🎒 Please bring** - A laptop (charged!) and charger; preinstalled with: - [Docker](https://docs.docker.com/get-docker/) - [uv](https://docs.astral.sh/uv/getting-started/installation/) - An LLM of your choice to work with, either: - your own API key for a hosted provider of your choice, or - free local models running via Ollama in Docker (no key needed, but allow around 10 GB of free disk space) - Curiosity and a willingness to collaborate
**📶 Wi-Fi** will be provided at the venue. We'll share the starter kit with registered participants before the event, so please download everything (especially the local models) in advance. They're a few GB, and the Wi-Fi is no match for a whole room pulling them at once! 😅
**📒 IMPORTANT NOTE** To take part in the hackathon, **please sign up on the Luma link above**. You will be required to show your QR code at the reception. **RSVP on Meetup does not count as registration for this event**, so sign up on Luma now to secure your place!
Many thanks for your enthusiasm and participation! We can't wait to build with you 😄
tracks
Write the tests
Write a set of realistic (and sneaky) customer questions.
Build the checks
Build automatic pass/fail checks, plus an AI judge that marks every answer.
Fix it and prove it
Improve the bot, then show a before/after scorecard proving your fix worked without breaking anything else.
Bonus rounds
Extra rounds for speedy teams: tracing, red-teaming and model shoot-outs.
when
HACKATHON - Agent Eval Harness (by the Friendly Data Meetup)
Evening hackathon: building an evaluation harness for an AI customer support bot, with team work at multiple levels, bonus rounds and 3-minute demos.
◍ Quantspark, 230 Blackfriars Road, London SE1 8NW