Optimizing Web Fuzzing With Local LLMs

Introducing brainstorm

Brainstorm is an internet fuzzing device that mixes native LLM fashions and ffuf to optimize listing and file discovery. It combines conventional net fuzzing strategies (as carried out in ffuf) with AI-powered path technology to find hidden endpoints, recordsdata, and directories in net purposes. brainstorm often finds extra endpoints with fewer requests.

The device is offered right here:
https://github.com/Invicti-Safety/brainstorm

ffuf

ffuf is without doubt one of the hottest instruments for performing net fuzzing and is my favourite device for such duties. It’s a wonderful device, quick, straightforward to make use of and really configurable.

Ollama

Ollama is a device for operating open LLMs (Massive Language Fashions) regionally. You possibly can run fashions comparable to Llama 3.2, Phi 3, Mistral, Gemma 2, Qwen 2.5 coder and different fashions by yourself machine with out having to pay something. It’s obtainable for macOS, Linux, and Home windows.

How brainstorm works

brainstorm works by producing clever guesses for potential paths and filenames primarily based on some preliminary hyperlinks extracted from the goal web site. It really works by:

• Extracting preliminary hyperlinks from the goal web site
• Utilizing AI (native LLM fashions) to research the construction and counsel new potential paths
• Fuzzing these paths utilizing ffuf
• Studying from discoveries to generate extra focused recommendations
• Repeating the entire course of

Instance of operating brainstorm

Say that we now have an internet site that has two recordsdata: index.php and login.php.

Manually, we might run an area Ollama mannequin comparable to qwen2.5-coder utilizing the next command:

ollama run qwen2.5-coder

(when you don’t have qwen2.5 coder in your machine you would want to obtain it first utilizing ollama pull qwen2.5-coder)

We’d then paste a really lengthy immediate the place we ask the LLM to brainstorm new potential filenames or directories, you could find the complete immediate right here.

qwen answered with the next listing of potential recordsdata primarily based on the unique recordsdata (index.php and login.php).

dashboard.php
profile.php
settings.php
assist.php
phrases.php
privateness.php
contact.php
about.php
weblog.php
articles.php
posts.php
feedback.php
gallery.php
photographs.php
movies.php
audio.php
downloads.php
retailer/index.php
retailer/listing.php
retailer/view.php
retailer/cart.php
retailer/checkout.php
retailer/fee.php
api/v1/customers
api/v1/orders
api/v1/merchandise
api/v1/classes
api/v1/tags
api/v1/feedback
api/v2/customers
api/v2/orders
api/v2/merchandise
api/v2/classes
api/v2/tags
api/v2/feedback
admin/index.php
admin/login.php
admin/logout.php
admin/dashboard.php
admin/customers.php
admin/settings.php
admin/logs.php

Not unhealthy, among the recommendations are fairly good and naturally you may modify the immediate to incorporate completely different pointers on your particular case (to generate various kinds of filenames, directories, APIs, and so on.)

One other essential factor to know is that LLMs have non-deterministic conduct, which means that when you ask the identical query once more you would possibly obtain completely different solutions (completely different filenames). We may use this conduct in our favor to generate different potential filenames and directories.

That is the fundamental gist of how brainstorm work: it automates the entire course of above utilizing the Ollama API. From the unique hyperlinks, it generates new potential hyperlinks, take a look at them utilizing ffuf, if it finds new filenames which might be legitimate, it provides them to the immediate, after which repeats every thing many instances.

Making an attempt out brainstorm and ffuf on a take a look at web site

To check this device, I’ve constructed a take a look at web site utilizing hyperlinks from an actual web site (from a bug bounty program). This take a look at web site is an older Java web site with .jsp recordsdata. This web site has two hyperlinks on the principle web page: index.jsp and userLogin.jsp.

Utilizing ffuf with fuzz.txt

Let’s fuzz this web site with an excellent wordlist that I take advantage of rather a lot in my checks: fuzz.txt. It’s maintained by Bo0oM and it’s a wonderful wordlist, I extremely advocate it.

It discovered solely one endpoint: api. That’s to be anticipated, as fuzz.txt isn’t designed for .jsp recordsdata. Let’s strive with a .jsp particular wordlist.

Utilizing ffuf with jsp.txt

Subsequent, we’ll use a .jsp particular wordlist, that is a part of a set of tech-specific wordlists. The wordlist is jsp.txt. It accommodates 100,000 jsp particular recordsdata.

A lot better, it discovered 5 endpoints—but it surely made 100,000 requests to the goal web site.

Utilizing brainstorm

Now, let’s use the brand new device, brainstorm. It’s designed to obtain a full ffuf command line as a command line argument, so you may run ffuf first, exclude some responses, after which go the complete command line to brainstorm.

Within the first cycles, it discovered some attention-grabbing recordsdata comparable to forgotPassword.jsp, about.jsp, cart.jsp, checkout.jsp, contact.jsp and after a number of extra cycles it discovered different recordsdata comparable to userRegister.jsp. This final one is attention-grabbing as a result of it was brainstormed from the preliminary hyperlink userLogin.jsp. Some API endpoints have been additionally discovered.

After some time, no new recordsdata have been discovered, so I finished the method.

Ultimately, a complete of 10 new endpoints have been found BUT we solely despatched 328 requests. That’s significantly better when put next with the jsp.txt wordlist the place we discovered 5 endpoints however despatched 100,000 requests. Additionally, we didn’t ship all of the requests without delay, we despatched 30 requests, waited till the LLM generated extra doable filenames after which despatched a number of extra requests (solely the brand new/distinctive filenames). That is essential as a result of when you ship 100,000 requests without delay most web sites will block you instantly however when you ship a number of requests on occasion this would possibly get underneath the radar.

Which LLM mannequin to make use of?

As you’ve most likely seen above, I’m utilizing the mannequin qwen2.5-coder by default, I just like the qwen fashions rather a lot and use them each day, I think about them one of the best native fashions obtainable proper now.

However I wished to test perhaps different fashions are higher on this particular job. So, I wrote a python script to check all of the fashions that I had put in on my pc and test what number of endpoints every one discovered.

The fashions that I’ve examined are:

Some fashions are greater (like qwen2.5-coder:14b with 14B) and others smaller (phi3 with 3.8B)—these are merely the fashions I had on my machine.

Ultimately, the outcomes are as follows:

As anticipated, the larger fashions (14B) carry out higher however from the 7/8B parameter fashions the qwen fashions are often fairly good. llama3.1 as additionally doing very nicely. You’ll find the complete benchmark outcomes right here.

One other take a look at web site (PHP)

I examined brainstorm with one other take a look at web site, this time PHP-based. It began with one file auth/login.php and it found 13 new endpoints whereas making 276 requests.

Shortname scanner

The thought behind this device may very well be utilized to different fuzzing issues. It may very well be utilized for fuzzing APIs, subdomains, digital hosts, …

For instance, I’ll present how I utilized this concept to fuzzing IIS brief (8.3) filenames. IIS (Web Info Providers) makes use of brief (8.3) filenames, a legacy function from older file methods like FAT, to keep up compatibility with purposes that require 8-character filenames and 3-character extensions. These brief names are routinely created by the file system for recordsdata and directories with lengthy names.

There are well-known IIS brief (8.3) filenames scanners comparable to IIS-ShortName-Scanner from Soroush Dalili. These instruments reap the benefits of a vulnerability in IIS that enables attackers to enumerate brief filenames. However after getting a brief filename comparable to FORGOT~1.JSP you want a strategy to guess the complete filename. For instance, the complete identify behind this brief filename is forgotPassword.jsp.

I’ve tailored the unique script fuzzer.py to attempt to guess full names when supplied with a brief filename. The brand new script is fuzzer_shortname.py. You present this script with ffuf command line and with a brief filename and the LLM will attempt to brainstorm full filenames.

The LLM immediate that I’ve used on this case is offered right here.

As you may see above, the brand new filenames prompt are fairly good and the device was capable of establish the proper full filename.

Nevertheless, it doesn’t work as nicely in all circumstances. LLMs typically counsel filenames that don’t begin with the brief filename even when the immediate consists of the next requirement: “All of the filenames ought to begin with the filename earlier than the tilde and use the identical extension. DO NOT generate filenames that don’t begin with the filename earlier than the tilde or use a special extension.”

As you may see above, filenames like userReset.jsp have been prompt even when the brief filename is FORGOT~1.JSP. This can be a recognized limitation of native LLMs, it doesn’t apply to greater LLMs. I’m not conscious of an answer to this drawback besides switching to greater LLMs.

Conclusion

I believe that future fuzzing instruments ought to be rewritten to reap the benefits of the advantages that LLMs present. LLMs are nice at brainstorming new objects, and I hope this concept will subsequent be utilized to enhancing subdomain discovery, the place you present the LLM with a listing of recognized subdomains and ask it to generate variations primarily based on these current subdomains. The LLM ought to have the ability to establish patterns within the discovered subdomains and brainstorm new subdomains utilizing the patterns it discovered.

Greater LLMs are higher

Additionally, this device is designed to make use of native LLMs (with sizes of seven/8B and 14B) which you could run in your native pc with out having to pay for entry. I’ve experimented with smarter LLMs comparable to Claude Sonnet 3.5 and the outcomes are significantly better, but it surely prices cash to run the device, so it may not make sense in all circumstances.