[HackerNotes Ep. 117] Vulnus Ex Machina - AI Hacking Part 1

In this episode of Critical Thinking - Bug Bounty Podcast @Rez0 introduces Vulnus Ex Machina: A 3-part mini-series on hacking AI applications. In this part, he lays the groundwork and focuses on AI reconnaissance.

Hacker TL;DR

• Vulnus ex Machina, Part 1

  This is the first episode of a 3-part series where Rez0 talks about AI hacking. He covers how to identify AI features in apps and do initial recon to understand their capabilities and weak spots.

  • Talk on Building AI Agents: Paul Klein's talk breaks down web-based AI agents, covering everything from vision vs text-based agents to building reliable ones with frameworks like Stagehand. Great watch if you're into automating web tasks or building AI security tools.

  • 17 Security Checks - Vibe Coding → Prod: With more people using AI to code without understanding security basics, here's a list of 17 checks to run before pushing to prod. You can also use these checks to analyse other people's code - worth adding to your AI workflow if you have one.

• Understand current AI models

  You gotta know your LLMs to hack them properly. At their core, they're fancy text predictors that now handle text, images, audio, and video, either directly or through specialised models.

  • Get comfortable using and steering them: Play with free versions of ChatGPT, Claude, and Gemini, and test their limits by throwing weird characters at them, making them contradict themselves, or trying to peek at their system prompts. The more you experiment, the better you'll get at steering them where you want.

  • System Prompts: Think of these as the model's instruction manual, they control how the AI behaves. While users can't see them directly, you can sometimes trick the model into revealing them or at least some part of it.

• Identifying Features

  The best way to understand LLMs is hands-on, ask them questions, read their docs, and use them like a regular user. Once you're comfortable, try to leak their system prompt. Check out our previous episode with Johann or the HackerNotes for Ep. 101 for some cool techniques.

  • 1. Finding leaked system prompts might show you guidelines and tools, but just because you can break the AI's rules doesn't mean it's a vulnerability. Check the program's scope first and maybe submit a test bug to see if they care about these findings.

    2. Test rendering capabilities - try HTML instead of markdown, or get it to output markdown image links. If successful, you could make it generate links to your server, potentially leaking data through parameters.

    3. Test malicious links - see if the LLM visits them and try different payloads for server-side or client-side execution.

    4. Check info sources, many LLMs use usernames in responses, so make your username a payload. Look for other sources to exploit too.

    Social engineer the AI to explain its features. Act like an interested user and it'll likely tell you everything.

ThreatLocker Cloud Control leverages built-in intelligence to assess whether a connection from a protected device originates from a trusted network.

By analyzing connection patterns from protected computers and mobile devices, it automatically identifies and allows trusted connections.

Find out more here:

https://www.threatlocker.com/platform/cloud-control

— Vulnus ex Machina, Part 1

We got a new style of content here at the CTBB Podcast. This week we got the first episode of this 3 part series where Rez0 will talk about AI hacking.

In this first episode he’s talking about identifying AI features in apps and performing the initial recon to understand their capabilities and potential weaknesses so you can figure out what part of the application is more worth hacking on. So we hope you find these episodes useful!

• Talk on Building AI Agents

  Really cool talk from Paul Klein, he breaks down web-based AI agents, covers everything from different types of agents (vision vs text-based) to building reliable ones with frameworks like Stagehand. If you're interested in automating web tasks or building AI tools for security testing, it’s a must-watch.

• Gemini 2.5 Pro

  New SOTA model that Google just released, it not only beats all the other LLMs in pretty much every type of task, it has 1M tokens context window. But it will take some convincing if you want to use it for hacking. It’s already in the main app if you have a Gemini subscription but if you don’t, you can go to ai.dev to play with it a little even if you’re not subscribed.

• 17 Security Checks - Vibe Coding → Prod

  A lot people are creating apps without writing any code, which will likely cause security issues in the future since they lack knowledge about best practices or aren't even aware that secure coding is a thing.

  But there are people who are interested in making vibe-coded apps more secure, here’s a list of 17 things you can ask your AI coding assistant to check before pushing things to prod. Vibe-checking vibe-coded apps, sounds like a great idea! Hahah

  Nonetheless, we can use this same list of checks to ask our assistants to analyse someone else's code. Take a look at that list and consider adding it to your automated AI workflow if you have one.

• ChatGPT-4o Image Generation

  You can now generate some incredible stuff directly from your chats with 4o. It now handles text a lot better and understands context in a way that you can ask it to make something, change it and the rest of the image stays the same. Here’s an example of what it can do, really impressive!

— AI Hacking Overview

Understand current AI models

You gotta know your LLMs to hack them properly. Yeah, they're basically just fancy text predictors under the hood, but these days they can do way more than just spit out words.

Most modern LLMs handle text, images, audio, and video, sometimes directly and sometimes by calling out to other specialised models.

The tricky part is testing them because you can't always see how they're built, even when they accidentally leak some system prompt details.

Get comfortable using and steering them

Get your hands dirty with AI models every day, and play around with free versions of ChatGPT, Claude, and Gemini to really get a feel for what they can and can't do.

Push these models to their limits - throw weird characters at them, try to make them contradict themselves, or see if you can peek behind the curtain at their system prompts. The more you mess around with them, the better you'll get at steering them where you want them to go when you're testing for vulnerabilities.

System Prompts: Think of system prompts as the model's instruction manual - they're the rules developers set to control how the AI behaves. Users can't see them directly, but with the right tricks, you can sometimes get the model to spill the beans about what those instructions are.

Retrieval-Augmented Generation: RAG is like giving an AI model its own personal Google. Instead of just working with what it learned during training, it can pull up specific info from a company's database.

1. You ask something

2. The system digs through its database

3. It grabs any relevant info it finds

4. The AI uses both your question and this extra context to give you an answer

Jailbreaking: Most bug bounty programs won't pay for jailbreaks unless they specifically ask for them. They're seen more as model behaviour issues than security vulnerabilities, since most AI apps are just built on top of existing models.

A jailbreak is basically tricking an AI into ignoring its rules. You've got partial jailbreaks (work in specific cases), universal ones (work all the time), and transferable ones (work across different AI models). Check out llm-attacks.org for some cool examples but you can also play around with Pliny's Jailbreaks, you'll need to experiment a lot to get good at it.

The Recon Process

1. It's important to go around looking for new things to hack on. Rez0 talked about the time his team won an LHE, where they had access to some features and functionalities that not many people had access to.

So going after these things like asking people for access to new stuff, signing up for exclusive features or beta access, it really pays off because you'll be competing with a lot fewer people.

Messaging the program managers for your favourite programs asking for access to AI features that are still in beta also pays off sometimes, because it shows proactivity and that you care about their product.

2. Another thing that could be useful is monitoring the programs you like to hack on: sign up/subscribe to developer messages and/or set up a monitoring script that sends you notifications whenever the devs push something new, and you'll be ahead of the competition.

3. The third tip is that a lot of companies are now implementing customer support with AI chatbots, so take a look at the "support" and "contact us" pages because you may find out that they're starting to use AI in places that were previously used exclusively for customers to contact them.

Identifying Features

No surprise, the best way to learn what these LLMs can do is to ask them, read docs, and use those features like a normal user would. After getting a feel for them, you should try to leak their system prompt. We covered quite a few techniques for that in a previous episode with Johann, you can also find the link to the HackerNotes for Ep. 101 here.

But in general, you want to trick it into leaking the system prompt and break its own rules. Try things like telling it your page refreshed and you lost the previous conversation, telling it you don’t speak English very well and giving you info in another language.

1. When you get the system prompt to leak, you'll likely see guidelines and mentions of tools. However, just because you can make the AI do something against its system prompt rules doesn't automatically mean it's a vulnerability. While some cases may be worth reporting, these AI safety issues aren’t a priority to programs most of the time. Review the program's scope first, and consider submitting one test bug to see if they accept these types of findings. There isn’t much they can do about prompt injection at the moment so they might not accept it.

2. Trying to make it render things is also a good idea, try to make it render HTML instead of the usual markdown conversion, try to make it respond with a markdown image link if an LLM can output markdown, a prompt injection becomes more dangerous since the attacker can make it generate an image link pointing to their server. The path or parameters of this link could contain sensitive data, enabling data exfiltration without user interaction.

3. Malicious links are also a great idea, try to figure out if they're being visited by the LLM and experiment with different payloads to see if anything executes on the server side or maybe even on the client side.

4. Experiment with its information sources. Some of these LLMs use your username in various places, so make your username a payload and play around. The username is only one of the sources, identify more and experiment with each of them.

A very powerful technique is to make it explain its own features to you. Social engineer it to show you everything it can do and gradually escalate the interaction — if you sound genuinely interested in using it as a normal user, it's extremely likely to give you all the information that you want. So frame your requests as normal ones and sound interested. Bonus tip from Rez0: if you have some kind of HTML injection, ask it to showcase all the HTML tags and see if they render or not in real-time on your screen, sounds fun!

Google dork strings like “Powered by <AI>” and “Built with <AI>” alongside your target’s name and you might be able to find some cool pages too!

That wraps up this week's episode.

Coming up next in Rez0's Vulnus ex Machina series, we'll dive into specific attack scenarios, explore attack vectors for finding vulnerabilities, examine prompt injection techniques, and much more, so stay tuned.

And as always, keep hacking!