# How to use `radiusd -X` (debug mode)

This page explains how to read the output of `radiusd -X`.

The first part of the debug output is the *startup* text.  Once the server is started, it prints `Ready to receive requests`.  The next part of the debug output is the *packet processing* text.  Both parts need to be read (and posted to the list!) in order to solve issues.

## If it doesn't start

If the server does not start up correctly, *the debug output will tell you why*.  It will produce long descriptive error messages, telling you exactly what's wrong.  In some cases, it will tell you how to fix it.

Look for messages like *Error* or *Warning*.  If you are running the server in a terminal console, those messages will appear in red or yellow.  It should be simple to scroll through the output, ignoring most of it, and looking for red *Error* text.

Fixing those errors will usually allow the server to start correctly.

## Startup text

We start the server with `-X`.  This option gives us the best combination of useful output, and readable output.  Adding more `-x` gives *more complex output*,  not *more useful output*.

    $ radiusd -X 2>&1 | tee debugfile

The server prints out the version number, copyright, and license information:

    FreeRADIUS Version 3.0.17
    Copyright (C) 1999-2017 The FreeRADIUS server project and contributors
    There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
    PARTICULAR PURPOSE
    You may redistribute copies of FreeRADIUS under the terms of the
    GNU General Public License
    For more information about these matters, see the file named COPYRIGHT

Next, it prints out messages relating to debuggers (e.g. gdb).  These messages are intended for developers, and can generally be ignored by everyone else.

    Getting debug state failed: ptrace capability not set.  If debugger detection is required run as root or: setcap cap_sys_ptrace+ep <path_to_radiusd>

### Reading the configuration files

The next block of messages is about the configuration files.  The server prints out the name of each configuration file it loads.

This output lets you check that the server is loading the files which you think it's loading.  One common issue is that people install multiple versions of the server, and then edit one file while the server is loading a different one.  Reading the debug output will let you verify that you're editing the right file.

    Starting - reading configuration files ...

It then reads the dictionaries.  It does not print out the name of every dictionary file it reads, as there are over 100 files, but they are never the less still loaded.

    including dictionary file /usr/share/freeradius/dictionary
    including dictionary file /usr/share/freeradius/dictionary.dhcp
    including dictionary file /usr/share/freeradius/dictionary.vqp
    including dictionary file /etc/raddb/dictionary

Next is loads the main `radiusd.conf` file, and some associated ones like `proxy.conf` and `clients.conf`

    including configuration file /etc/raddb/radiusd.conf
    including configuration file /etc/raddb/proxy.conf
    including configuration file /etc/raddb/clients.conf

It now loads all of the modules.  The module configuration is usually stored in `raddb/mods-enabled/`, with one module configuration per file in that directory.

Sometimes the module you're using does not seem to be loaded or used.  In that case, you should check this section to see that the file is loaded.

    including files in directory /etc/raddb/mods-enabled/
    including configuration file /etc/raddb/mods-enabled/pap
    including configuration file /etc/raddb/mods-enabled/expiration
    including configuration file /etc/raddb/mods-enabled/files
    including configuration file /etc/raddb/mods-enabled/linelog
    including configuration file /etc/raddb/mods-enabled/soh
    including configuration file /etc/raddb/mods-enabled/attr_filter
    including configuration file /etc/raddb/mods-enabled/ntlm_auth
    including configuration file /etc/raddb/mods-enabled/exec
    including configuration file /etc/raddb/mods-enabled/preprocess
    including configuration file /etc/raddb/mods-enabled/sradutmp
    including configuration file /etc/raddb/mods-enabled/chap
    including configuration file /etc/raddb/mods-enabled/digest
    including configuration file /etc/raddb/mods-enabled/expr
    including configuration file /etc/raddb/mods-enabled/echo
    including configuration file /etc/raddb/mods-enabled/unpack
    including configuration file /etc/raddb/mods-enabled/detail
    including configuration file /etc/raddb/mods-enabled/always
    including configuration file /etc/raddb/mods-enabled/eap
    including configuration file /etc/raddb/mods-enabled/mschap
    including configuration file /etc/raddb/mods-enabled/unix
    including configuration file /etc/raddb/mods-enabled/detail.log
    including configuration file /etc/raddb/mods-enabled/passwd
    including configuration file /etc/raddb/mods-enabled/date
    including configuration file /etc/raddb/mods-enabled/logintime
    including configuration file /etc/raddb/mods-enabled/utf8
    including configuration file /etc/raddb/mods-enabled/dynamic_clients
    including configuration file /etc/raddb/mods-enabled/radutmp
    including configuration file /etc/raddb/mods-enabled/realm
    including configuration file /etc/raddb/mods-enabled/cache_eap
    including configuration file /etc/raddb/mods-enabled/replicate

Next, some policies are loaded.  These policies are ones which the server uses (for example) to canonicalize user names, and to do things like attribute re-writes, sanity checks, etc.

    including files in directory /etc/raddb/policy.d/
    including configuration file /etc/raddb/policy.d/control
    including configuration file /etc/raddb/policy.d/cui
    including configuration file /etc/raddb/policy.d/debug
    including configuration file /etc/raddb/policy.d/moonshot-targeted-ids
    including configuration file /etc/raddb/policy.d/eap
    including configuration file /etc/raddb/policy.d/filter
    including configuration file /etc/raddb/policy.d/canonicalization
    including configuration file /etc/raddb/policy.d/abfab-tr
    including configuration file /etc/raddb/policy.d/operator-name
    including configuration file /etc/raddb/policy.d/dhcp
    including configuration file /etc/raddb/policy.d/accounting
    including files in directory /etc/raddb/sites-enabled/
    including configuration file /etc/raddb/sites-enabled/default
    including configuration file /etc/raddb/sites-enabled/inner-tunnel

### Server Configuration

Once all of the configuration files are loaded, the server prints out the configuration it is parsing.  Note that it *only prints out what it is using*.  I.e. you can put `foo = bar` into `radiusd.conf`, and the server will load it, but will not print it out in the debug output.

The result is that if you set a configuration item, *it will be printed out in the debug output*.  You should verify that this is the case, especially if it looks like the server is not using the setting you edited.  If the configuration item does not appear in the debug output, *then the server is not using it.*

This is the "main" server configuration:

    main {
    	name = "radiusd"

Which directories the server is using,

    	prefix = "/"
    	localstatedir = "/var"
    	sbindir = "/usr/sbin"
    	logdir = "/var/log/radius"
    	run_dir = "/var/run/radiusd"
    	libdir = "/usr/lib"
    	radacctdir = "/var/log/radius/radacct"

Next we have some generic configuration settings which don't belong anywhere else.

    	hostname_lookups = no
    	max_request_time = 30
    	cleanup_delay = 5
    	max_requests = 16384
    	pidfile = "/var/run/radiusd/radiusd.pid"
    	checkrad = "/sbin/checkrad"
    	debug_level = 0
    	proxy_requests = yes

And the logging configuration.

     log {
     	stripped_names = no
     	auth = no
     	auth_badpass = no
     	auth_goodpass = no
     	colourise = yes
     	msg_denied = "You are already logged in - access denied"
     }
     resources {
     }

The security settings.

     security {
     	max_attributes = 200
     	reject_delay = 1.000000
     	status_server = yes
     	allow_vulnerable_openssl = "yes"
     }
    }

At this point, the server is well on it's way to running.

### Proxy Configuration

The server now loads its proxy configuration, as was defined in `proxy.conf`:

    radiusd: #### Loading Realms and Home Servers ####

The over all proxy configuration is loaded.

     proxy server {
     	retry_delay = 5
     	retry_count = 3
     	default_fallback = no
     	dead_time = 120
     	wake_all_if_all_dead = no
     }

Followed by `home_server` configurations.

     home_server localhost {
     	ipaddr = 127.0.0.1
     	port = 1812
     	type = "auth"
     	secret = <<< secret >>>
     	response_window = 20.000000
     	response_timeouts = 1
     	max_outstanding = 65536
     	zombie_period = 40
     	status_check = "status-server"
     	ping_interval = 30
     	check_interval = 30
     	check_timeout = 4
     	num_answers_to_alive = 3
     	revive_interval = 120
      limit {
      	max_connections = 16
      	max_requests = 0
      	lifetime = 0
      	idle_timeout = 0
      }
      coa {
      	irt = 2
      	mrt = 16
      	mrc = 5
      	mrd = 30
      }
     }

Followed by `home_server_pool` configurations.

     home_server_pool my_auth_failover {
    	type = fail-over
    	home_server = localhost
     }

Followed by `realm` configurations.

     realm example.com {
    	auth_pool = my_auth_failover
     }
     realm LOCAL {
     }
     realm int {
    	virtual_server = inner-tunnel
     }

### Clients

It now loads individual clients:

    radiusd: #### Loading Clients ####
     client localhost {
     	ipaddr = 127.0.0.1
     	require_message_authenticator = no
     	secret = <<< secret >>>
     	nas_type = "other"
     	proto = "*"
      limit {
      	max_connections = 16
      	lifetime = 0
      	idle_timeout = 30
      }
     }
     client localhost_ipv6 {
     	ipv6addr = ::1
     	require_message_authenticator = no
     	secret = <<< secret >>>
      limit {
      	max_connections = 16
      	lifetime = 0
      	idle_timeout = 30
      }
     }

### Auth-Type

The server then creates `Auth-Type`, as set in the virtual servers.  These `Auth-Type` names and values are used by the modules, so they need to be created now.

     # Creating Auth-Type = mschap
     # Creating Auth-Type = digest
     # Creating Auth-Type = eap
     # Creating Auth-Type = PAP
     # Creating Auth-Type = CHAP
     # Creating Auth-Type = MS-CHAP

### Modules

When the server read the module configuration file above (from `raddb/mods-enabled`), that meant just reding the contents of the file.  It is now ready to load the dynamic library which reads that configuration, and which parses the configuration to determine what to do with it.

Each module will print out it's configuration, along with any errors or warnings it sees when it is loading itself.

    radiusd: #### Instantiating modules ####
     modules {

Here is loads the `pap` module.  There are multiple lines being printed at each stage of the process.

First, it discovers it has to load the `pap` module, and decides to do that.

      # Loaded module rlm_pap

Second, it tells you which files is used to read the `pap` module configuration.

      # Loading module "pap" from file /etc/raddb/mods-enabled/pap

Finally, it prints out the configuration that is used by the `pap` module.

      pap {
        normalise = yes
      }

When you are editing a module configuration, it is good to check the debug output for that module, to see:

* was the module loaded?
* is it using the file you edited?
* are the configuration changes you made showing up in the debug output?

Some modules like `expiration` do not have any configuration items, and therefore don't print out any configuration when they are loaded.

      # Loaded module rlm_expiration
      # Loading module "expiration" from file /etc/raddb/mods-enabled/expiration

This module loads the `users` file.

      # Loaded module rlm_files
      # Loading module "files" from file /etc/raddb/mods-enabled/files
      files {
      	filename = "/etc/raddb/mods-config/files/authorize"
      	acctusersfile = "/etc/raddb/mods-config/files/accounting"
      	preproxy_usersfile = "/etc/raddb/mods-config/files/pre-proxy"
      }

The server loads a large number of modules, so we will omit them here.

Finally, it is done loading all modules, and prints this:

     } # modules

### Virtual Servers

Next, it loads "virtual servers".

    radiusd: #### Loading Virtual Servers ####

It prints out the server, and name, along with which file it was loaded from.

    server default { # from file /etc/raddb/sites-enabled/default

It then double-checks the various processing sections.

     # Loading authenticate {...}
     # Loading authorize {...}
     # Loading preacct {...}
     # Loading accounting {...}

The `sql` module is in the default configuration, even if the `sql` module is not enabled.  This warning message is then printed out.

It can be ignored, or, in a production environment where you don't use `sql`, you can just remove references to `sql` from the virtual servers.

    Ignoring "sql" (see raddb/mods-available/README.rst)
     # Loading post-proxy {...}
     # Loading post-auth {...}
    } # server default
    server inner-tunnel { # from file /etc/raddb/sites-enabled/inner-tunnel
     # Loading authenticate {...}
     # Loading authorize {...}
     # Loading session {...}
     # Loading post-auth {...}

In some situations, it tells you where the configuration can be simplified.

     # Skipping contents of 'if' as it is always 'false' -- /etc/raddb/sites-enabled/inner-tunnel:335
    } # server inner-tunnel

### Listen Sections

After loading all of the virtual servers, it then opens the various network sockets used to read and write packets.

    radiusd: #### Opening IP addresses and Ports ####

First, it prints out the configuration for the listen sections.

    listen {
      	type = "auth"
      	ipaddr = *
      	port = 0
       limit {
       	max_connections = 16
       	lifetime = 0
       	idle_timeout = 30
       }
    }
    listen {
      	type = "acct"
      	ipaddr = *
      	port = 0
       limit {
       	max_connections = 16
       	lifetime = 0
       	idle_timeout = 30
       }
    }
    listen {
      	type = "auth"
      	ipv6addr = ::
      	port = 0
       limit {
       	max_connections = 16
       	lifetime = 0
       	idle_timeout = 30
       }
    }
    listen {
      	type = "acct"
      	ipv6addr = ::
      	port = 0
       limit {
       	max_connections = 16
       	lifetime = 0
       	idle_timeout = 30
       }
    }
    listen {
      	type = "auth"
      	ipaddr = 127.0.0.1
      	port = 18120
    }

After reading all of the configuration, it opens the ports, and prints out a list of IP addresses and ports which it is using.

    Listening on auth address * port 1812 bound to server default
    Listening on acct address * port 1813 bound to server default
    Listening on auth address :: port 1812 bound to server default
    Listening on acct address :: port 1813 bound to server default
    Listening on auth address 127.0.0.1 port 18120 bound to server inner-tunnel
    Listening on proxy address * port 39556
    Listening on proxy address :: port 52609

And we finally get to this line.

    Ready to process requests

Once this has been printed, the debug output changes to packet processing.

Note that if *nothing* is ever printed after this line, then the server is not receiving any packets.  No amount of changing the server configuration will solve that problem.  Instead, you must fix the firewall, SeLinux configuration, network routing, etc. so that the OS delivers RADIUS packets to the server.

## Packet Processing

Each line (or most of them) of the packet processing output starts with a request number, e.g. `(0)`.  This number lets you track debug messages which are all for one packet.  Messages from different packets may be inter-mingled, due to threading or proxying.  Prefixing the messages with a number allows the messages to be easily differentiated.

### Receiving a packet

The first message for a packet tells you what type of packet was received, what it's RADIUS ID is, and the source / destination IP / port information.

    (0) Received Access-Request Id 104 from 127.0.0.1:33278 to 127.0.0.1:1812 length 73

After that, it prints out all of the attributes which were received in the packet.

    (0)   User-Name = "bob"
    (0)   User-Password = "wrongpassword"
    (0)   NAS-IP-Address = 127.0.1.1
    (0)   NAS-Port = 0
    (0)   Message-Authenticator = 0x3d27116b37323e4f629b4e8217fc25c8

Note that if an attribute is not printed here, then *it does not exist*.  If you need the packet to contain an attribute, then you must **fix the NAS** so that the NAS sends the attribute.

Once the packet is received, it is run through the various processing sections of the server.  For Access-Request packets, these are `authorize`, `authenticate`, and `post-auth`.

    (0) # Executing section authorize from file /etc/raddb/sites-enabled/default
    (0)   authorize {

The `authorize` section contains a number of modules, along with unlang keywords like `if`, `update`, etc.

Each module prints out what it is doing, and why.  For example, the `suffix` module here is looking for `User-Name` attributes which contain `user@domain`.

    (0) suffix: Checking for suffix after "@"
    (0) suffix: No '@' in User-Name = "bob", looking up realm NULL
    (0) suffix: No such realm "NULL"

The server core then prints out the "return code" of the module.  See `man unlang` for a deeper explanation of return codes.

    (0)     [suffix] = noop
    (0)     [files] = noop

The `authorize` section also has a return code.

    (0)   } # authorize = noop

In this case, the user is unknown, so they will be rejected.

    (0) ERROR: No Auth-Type found: rejecting the user via Post-Auth-Type = Reject
    (0) Failed to authenticate the user

The packet is processed through the `Post-Auth-Type REJECT` section

    (0) Using Post-Auth-Type Reject
    (0) # Executing group from file /etc/raddb/sites-enabled/default
    (0)   Post-Auth-Type REJECT {

That section contains more modules and unlang statements.

Here, the `attr_filter` module is removing attributes which are forbidden from appearing in an Access-Reject packet.

    (0) attr_filter.access_reject: EXPAND %{User-Name}
    (0) attr_filter.access_reject:    --> bob
    (0) attr_filter.access_reject: Matched entry DEFAULT at line 11
    (0)     [attr_filter.access_reject] = updated
    (0)     [eap] = noop
    (0)     policy remove_reply_message_if_eap {
    (0)       if (&reply:EAP-Message && &reply:Reply-Message) {
    (0)       if (&reply:EAP-Message && &reply:Reply-Message)  -> FALSE
    (0)       else {
    (0)         [noop] = noop
    (0)       } # else = noop
    (0)     } # policy remove_reply_message_if_eap = noop

Finally, the `Post-Auth-Type REJECT` section is done.

    (0)   } # Post-Auth-Type REJECT = updated

Since this is an Access-Reject, it is delayed for one second to prevent password attacks.  The `reject_delay` configuration above control this delay.

    (0) Delaying response for 1.000000 seconds

The server then sleeps for a while, waking up periodically to deal with internal book-keeping.

    Waking up in 0.3 seconds.
    Waking up in 0.6 seconds.

Finally after a one-second delay, the server wakes up and sends the Access-Reject packet.

    (0) Sending delayed response

The IP / port printed here is the mirror of the one printed for the Access-Request, above.  This indicates that it's a reply to an earlier request.

    (0) Sent Access-Reject Id 104 from 127.0.0.1:1812 to 127.0.0.1:33278 length 20

If there are any attributes in the reply packet, they will be printed here.  In this case, there are non.

Note that you **should** check the reply attributes to see if the server is sending the reply you expect.  If it is not sending the correct reply. you will need to fix the configuration to do so.

If it is sending the correct reply, but the user is not getting the expected service, then the problem is the NAS.  No amount of changing the server configuration will make the NAS behave.

The server then sleeps for a while:

    Waking up in 3.9 seconds.

And then cleans up the request and it's associated response.  This delay is controlled by `cleanup_delay`, which was seen earlier in the debug output.

    (0) Cleaning up request packet ID 104 with timestamp +23

Since there's nothing more to do, we're back to this message.

    Ready to process requests

The debug log has another packet, which we will skip, as it's largely the same as the previous one.

    (1) Received Access-Request Id 146 from 127.0.0.1:40967 to 127.0.0.1:1812 length 73
    (1)   User-Name = "bob"
    (1)   User-Password = "wrongagain"
    (1)   NAS-IP-Address = 127.0.1.1
    (1)   NAS-Port = 0

The debug log then shows a successful authentication.

The input packet is largely the same, but has a different `User-Name` and `User-Password` attribute.

    (2) Received Access-Request Id 135 from 127.0.0.1:40344 to 127.0.0.1:1812 length 77
    (2)   User-Name = "bob@int"
    (2)   User-Password = "test"
    (2)   NAS-IP-Address = 127.0.1.1
    (2)   NAS-Port = 0

The `Message-Authenticator` attribute is a cryptographic signature of the packet, and has no other meaning.

    (2)   Message-Authenticator = 0x3b3f4cf11005dcccfe78bb4a5830dd52

We start the `authorize` section again/

    (2) # Executing section authorize from file /etc/raddb/sites-enabled/default
    (2)   authorize {

This time the `suffix` module does find a suffix.

    (2) suffix: Checking for suffix after "@"
    (2) suffix: Looking up realm "int" for User-Name = "bob@int"

The realm `int` is defined above in the `realm` configuration.

    (2) suffix: Found realm "int"

The `suffix` module splits the `User-Name` into a `Stripped-User-Name` and `Realm` attributes.  The `Stripped-User-Name` is used by subsequent modules to match the user's name.

    (2) suffix: Adding Stripped-User-Name = "bob"
    (2) suffix: Adding Realm = "int"
    (2) suffix: Proxying request from user bob to realm int
    (2) suffix: Preparing to proxy authentication request to realm "int" 
    (2)     [suffix] = updated
    (2)     [files] = noop
    (2)   } # authorize = updated
    (2) Starting proxy to home server (null) port 1812

This configuration has been changed from the default configuration to proxy packets to the `inner-tunnel` virtual server.  This change was done here strictly for demonstration purposes.  It is not necessary (and you should not do it!) in normal configurations.

    Proxying to virtual server inner-tunnel
    (2) # Executing section authorize from file /etc/raddb/sites-enabled/inner-tunnel
    (2)   authorize {

The `files` module matches the user name and realm, at `line 1` of the `users` file.  This output lets you know exactly which entry was matched.

If the server does not do what you expect it to do, you should read `line 1` of the `users` file (or whatever entry matched), to verify that the entry is what you expect it to be.

    (2) files: users: Matched entry bob@int at line 1
    (2)     [files] = ok

The `pap` module sees the `Cleartext-Password` which was set in the `users` file, along with the `User-Password` that came from the packet.  The module then sets `Auth-Type := PAP`.  This is so that the `authenticate` section will run the `pap` module, which will then authenticate the user.

    (2)     [pap] = updated
    (2)   } # authorize = updated

It now runs `Auth-Type PAP`

    (2) Found Auth-Type = PAP
    (2) # Executing group from file /etc/raddb/sites-enabled/inner-tunnel
    (2)   Auth-Type PAP {

The `pap` module tells you that everything is OK.

    (2) pap: Login attempt with password
    (2) pap: Comparing with "known good" Cleartext-Password
    (2) pap: User authenticated successfully
    (2)     [pap] = ok
    (2)   } # Auth-Type PAP = ok

It now runs the normal `post-auth` section

    (2) # Executing section post-auth from file /etc/raddb/sites-enabled/inner-tunnel
    (2)   post-auth {
    (2)     update reply {
    (2)       Reply-Message := "hello"
    (2)     } # update reply = noop
    (2)     if (0) {
    (2)     if (0)  -> FALSE
    (2)   } # post-auth = noop
    (2) Finished internally proxied request.
    (2) Clearing existing &reply: attributes

Because this is a proxied request, it now runs the `post-proxy` section of the `default` virtual server.

    (2) # Executing section post-proxy from file /etc/raddb/sites-enabled/default
    (2)   post-proxy {
    (2)     policy debug_reply {
    (2)       if ("%{debug_attr:reply:}" == '') {
    (2)       Attributes matching "reply:"
    (2)         EXPAND %{debug_attr:reply:}
    (2)            --> 
    (2)         if ("%{debug_attr:reply:}" == '')  -> TRUE
    (2)         if ("%{debug_attr:reply:}" == '')  {
    (2)           [noop] = noop
    (2)         } # if ("%{debug_attr:reply:}" == '')  = noop
    (2)       } # policy debug_reply = noop
    (2)     } # post-proxy = noop

This `Auth-Type = Accept` is added by the server code when proxying.  Since the proxied request returned an Access-Accept, the `default` virtual server treats that as a successful authentication.

    (2)   Found Auth-Type = Accept
    (2)   Auth-Type = Accept, accepting the user

It then runs the `post-auth` section from the `default` virtual server.

    (2)   # Executing section post-auth from file /etc/raddb/sites-enabled/default
    (2)     post-auth {
    (2)       update {
    (2)         No attributes updated
    (2)       } # update = noop
    (2)       [exec] = noop
    (2)       policy remove_reply_message_if_eap {
    (2)         if (&reply:EAP-Message && &reply:Reply-Message) {
    (2)         if (&reply:EAP-Message && &reply:Reply-Message)  -> FALSE
    (2)         else {
    (2)           [noop] = noop
    (2)         } # else = noop
    (2)       } # policy remove_reply_message_if_eap = noop
    (2)     } # post-auth = noop

And finally returns an Access-Accept to the client.  The `Reply-Message` here was take from `line 1` of the `users` file, when it matched above.

    (2)   Sent Access-Accept Id 135 from 127.0.0.1:1812 to 127.0.0.1:40344 length 0
    (2)     Reply-Message := "hello"
    (2)   Finished request
    Waking up in 4.9 seconds.
    (2)   Cleaning up request packet ID 135 with timestamp +74
    Ready to process requests
    ^C

That is a *lot* of information to go through.  We hope that this page has been useful.
