User Agents - Ineffective but Useful

Table of Contents

    Subscribe for Email Updates

    A user agent is a device (and more specifically, the software) that accesses a website. Web browsers like Chrome, Edge, and Safari are all user agents, and so is Google when it scans and indexes websites (i.e. “Googlebot”).

    The user agent identifies itself in every HTTP request using a written description in the “User-Agent” header. This HTTP header is primarily what we will be discussing today.

    User agents are awesome. They give you simple and useful information about the specific browsers, bots, and other services accessing your website. Web developers can use this information to optimize their content for the technology they were requested by. And, of course, they can be used to filter unwanted traffic from your servers.

    But user agents also have some major drawbacks.

    For starters, user agents are completely non-standardized. Some include version numbers, special characters, or domain names while others are simply brand names or even acronyms.

    Additionally, they are unbelievably easy to spoof. I know at least 3 easy ways for you to load this blog post with whatever user agent you want. In fact, you could even do it right there in your browser in about 30 seconds!

    Finally, user agents are not even required. It is perfectly valid to request a web page without any user agent at all. At least we can be glad that you can’t use multiple user agents for a single request.

    All of this adds up to two inescapable facts that do not appear compatible:

    1. User agents are extremely useful
    2. User agents are unbelievably unreliable

    Thankfully, most software - including all major browsers - reliably and accurately include a consistent user agent in every request. Even the majority of bots report some sort of relevant user agent. And if someone requests a page with a spoofed user agent, it is relatively safe to assume that they want the server to respond as if it were that user agent.

    So go ahead and customize your content for different browsers or user agents. Or even make up your own - there’s no rule against that. If you want to build a site that displays secret content when requested by “Programmers are Awesome”, then do it.

    And yes, it is good to block known “bad bots”. After all, there is no reason to subject your servers to “WebFu**” or “Kozmosbot” traffic unless you really want to.

    Just don’t be surprised when “malicious actors” pretend to be someone else. It’s just as easy for their spam factories to pretend to be “Google Chrome” or “Programmers are Awesome” as it is for you.

    Examples of User Agents

    Here are some sample user agents. Yeah - these don’t all make sense unless you understand the history behind them.

    For a more comprehensive list of common user agents, has a huge database of them.

    Here is a small sampling of some “bad” user agents you might want to block:

    • Aboundex
    • Acunetix
    • BlackWidow
    • BotALot
    • CATExplorador
    • Dispatch
    • Grafula
    • Heritrix
    • JetCar
    • Ltx71
    • NICErsPRO
    • Nmap
    • Nuclei
    • OutfoxBot
    • SBIder
    • SpankBot
    • Szukacz
    • Telesphoreo
    • TheNomad
    • Thumbor
    • TightTwatBot
    • Trendictionbot
    • VeriCiteCrawler
    • Virusdie
    • Wallpapers/3.0
    • Web Collage
    • check1.exe
    • ripz

    How to Use User Agents in Marketpath CMS

    Customizing content for a specific user in Marketpath CMS is incredibly easy.  In your template, simply reference {{ request.user_agent }}. You can even do this directly in HTML content areas using liquid markup!


    And if you want to block known bad bots that can be identified by their user agents, we make that as easy as enabling a site setting. Simply navigate to Site -> Site Settings. Scroll down to “Live Settings” and check the “Block Bad User Agents” checkbox.


    Not using Marketpath CMS yet?

    No worries - we make it painless to try out using our Free plan!

    Get Started Today

    About the Author

    Levi Carter

    Levi Carter is the senior developer of software products at Marketpath. While his primary responsibility is planning, developing, and maintaining Marketpath CMS, he has also been heavily involved in the full software lifecycle starting with strategy discussions and analysis through customer support and documentation.

    When Levi is not working on Marketpath software products he enjoys gardening, woodworking, and spending time with his wife and four children.

    Subscribe for Email Updates