--- title: "Connect AI Crawler Logs: Setup Guide | Brantial" description: "See which AI crawlers read your site. Pick the method that fits your site, set it up in about ten minutes and test the connection from Brantial." source_url: "https://brantial.ai/help/ai-crawler-logs/" lang: "en" updated: "2026-10-23T00:00:00.000Z" --- Help # Connect AI crawler logs to Brantial See which AI crawlers read your site. Pick the method that fits your site, set it up in about ten minutes and test the connection from Brantial. Updated October 23, 2026 Brantial shows which AI crawlers read your site, which pages they read and where they hit errors. To do that, a short piece of code on your site reports each AI crawler request to Brantial. This guide helps you pick the right method for your site, set it up and check that it works. ## Before you start 1. In Brantial, open your project and go to **AI Traffic**. 2. In the **AI crawler visits** card, click **Start setup**. 3. Choose your platform. Brantial shows your own code with your project’s key already in it. Keep this page open; you will copy the code from there. If someone else manages your website, use **Email it** in the same card. It sends them the code and these instructions. ## Which method fits your site? > **Not technical?** You do not have to do this yourself. Find your site in the table below, then click **Email it** in Brantial. It sends the right code and a link to this guide to whoever runs your website. | Your site | Method | Time | Who can do it | | --- | --- | --- | --- | | Any site whose domain is on Cloudflare, including Wix, Shopify, ikas, IdeaSoft, T-Soft and ASP.NET sites | Cloudflare Worker (recommended) | About 10 minutes | Anyone with access to the Cloudflare account | | WordPress, not on Cloudflare | Must-use plugin | About 5 minutes | Someone with file access to the hosting | | Next.js on Vercel | Middleware | About 10 minutes | A developer | | Windows server with IIS / ASP.NET | Global.asax | About 15 minutes | A developer | | Wix, Shopify, Webflow, Squarespace, ikas, IdeaSoft or T-Soft, not on Cloudflare | Put the domain behind Cloudflare first | 30 minutes plus DNS time | Whoever manages the domain | | Any other server or a log tool | Send the visits as JSON | Depends on the setup | A developer | **Not sure whether your domain is on Cloudflare?** Look up your domain’s nameservers, for example with a WHOIS search. If they end in `ns.cloudflare.com`, it is. The Cloudflare Worker is the most complete method: it sees every request, including pages served from a cache, and needs no change on your server. ## Cloudflare 1. In Brantial, choose **Cloudflare** and click **Copy**. 2. In the Cloudflare dashboard, open **Workers & Pages**, click **Create application**, then **Start with Hello World!** ![Cloudflare's Create an app screen with Start with Hello World! highlighted](https://brantial.ai/images/help/ai-crawler-logs/1-create.webp) 3. Name the Worker, for example `brantial-ai-bots`, and click **Deploy**. The Hello World code is only a placeholder; you replace it in the next step. ![The Deploy Hello World form with the Deploy button highlighted](https://brantial.ai/images/help/ai-crawler-logs/2-deploy-hello-world.webp) 4. On the Worker’s page, click **Edit code**. ![The Worker's tabs with the Edit code button highlighted](https://brantial.ai/images/help/ai-crawler-logs/3-edit-code.webp) 5. Delete everything in the editor, paste the code you copied from Brantial and click **Deploy**. ![The Cloudflare code editor with the Brantial code pasted and the Deploy button highlighted](https://brantial.ai/images/help/ai-crawler-logs/4-paste-and-deploy.webp) The key in the first line is blurred here. Yours is in the code you copied. ![The Deploy button at the top right of the code editor](https://brantial.ai/images/help/ai-crawler-logs/4b-deploy.webp) 6. Go back to the Worker, open the **Domains** tab and click **Add Route**. ![The Worker's tabs with Domains highlighted](https://brantial.ai/images/help/ai-crawler-logs/5-domains-tab.webp) ![Custom Domains and Routes with the Add Route button highlighted and two example routes](https://brantial.ai/images/help/ai-crawler-logs/6-add-route.webp) 7. Choose your domain as the **Zone**, enter `example.com/*` (with your domain) as the **Route** and click **Add Route**. Add a second route for `www.example.com/*` if your site also answers on [www](http://www/). ![The Add Route form with the Zone and Route fields](https://brantial.ai/images/help/ai-crawler-logs/7-route-form.webp) 8. Back in Brantial, click **Test connection**. ### Things to check - Your site’s DNS records must be **Proxied** (the orange cloud in DNS settings). A record set to **DNS only** does not pass through Cloudflare, so the Worker never sees its requests. - The Worker runs on every request to your site. Cloudflare’s free Workers plan covers 100,000 requests a day. If your site gets more traffic than that, use the Workers Paid plan so the Worker does not run out. ## Site builders without Cloudflare Wix, Shopify, Webflow, Squarespace, ikas, IdeaSoft and T-Soft do not let you add server code or read server logs. You can still connect them by putting the domain behind Cloudflare: 1. Add your domain to a Cloudflare account and change the nameservers at your domain registrar to the ones Cloudflare gives you. 2. Make sure the records that point to your site builder are **Proxied**. 3. Follow the Cloudflare steps above. Check your site builder’s help pages on using Cloudflare in front of it before you switch, and leave enough time for DNS changes to spread. ## WordPress 1. In Brantial, choose **WordPress** and click **Copy**. 2. Using FTP or your hosting’s file manager, open `wp-content`. Create a folder named `mu-plugins` if it is not there. 3. Create a file named `brantial-ai-bots.php` in `wp-content/mu-plugins`, paste the code and save it. Must-use plugins run on their own; there is nothing to activate. 4. Back in Brantial, click **Test connection**. Pages that a caching plugin serves without running WordPress are not counted. If your site is also on Cloudflare, the Cloudflare Worker counts those too. ## Next.js on Vercel 1. In Brantial, choose **Next.js / Vercel** and click **Copy**. 2. Save the code as `middleware.ts` in the root of your project, next to the `app` or `pages` folder. If you already have a middleware file, add the Brantial part to your existing function instead. 3. Deploy, then click **Test connection** in Brantial. Middleware runs before the page is built, so the response code is not sent with these visits. ## IIS / ASP.NET 1. In Brantial, choose **IIS / ASP.NET** and click **Copy**. 2. Open `Global.asax.cs` and paste the code inside the `Global` class. If the class already has an `Application_EndRequest` method, add the Brantial lines to it instead of adding a second one. 3. The code needs .NET Framework 4.5 or later and a reference to `System.Net.Http`. 4. Publish the site, then click **Test connection** in Brantial. If your site is behind Cloudflare, the Cloudflare Worker does the same job without any change on the server. ## Other servers and log tools Any server, CDN or log tool can send visits to your Brantial address as JSON. Choose **Any server** in Brantial to see your address. - Send a `POST` request with `{"visits": [...]}`. Each visit takes `ua` (the user agent), `path`, `status` and `ts` (an ISO 8601 time). - Up to 500 visits fit in one request. - Brantial keeps only AI crawler requests and ignores everything else, so you can send complete logs without filtering them first. ## Test the connection Click **Test connection** in Brantial’s setup card. Brantial visits your homepage once as a test crawler called `BrantialVerify`. Your code reports that visit, and within a few seconds Brantial shows whether it arrived. The test visit is never counted in your statistics. You can also run the test yourself from a terminal: ``` curl -A "BrantialVerify/1.0" https://example.com/ ``` After that, **Last report** in Brantial changes to a few seconds ago. ## Troubleshooting The connection test tells you which of these happened. ### “Brantial could not open your site” - Check the website address in your Brantial project settings. It must be the address your site opens on. - Open the address in a browser to make sure the site is up. ### “Your site turned the test crawler away” Bot protection or a firewall rule stopped the test before your code ran. This is common with Cloudflare’s bot settings and with security plugins. - Add a rule that allows requests whose user agent contains `BrantialVerify`, then test again. - Real AI crawlers may be stopped by the same rule. Brantial’s Site Health shows which ones your site lets in. ### “Your site opened, but no report arrived” The site answered, but your code did not report the visit. - **Cloudflare:** check that the route covers the address you tested, with and without www, and that you clicked **Deploy** after pasting the code. Check that your DNS records are **Proxied**. - **WordPress:** the file must be in `wp-content/mu-plugins` and start with `