Authentication Hangs

Symptom

Intermittently, after submitting your credentials for a WebSEAL protected web page, you wait for a server response until browser reaches it's time-out limit and fails, or after many minutes eventually succeeds. The affects all users attempting to authenticate. Users who already have active sessions, continue on uninterrupted. Pages that WebSEAL allows anonymous access to, work as intended.

This happens most often first thing in the morning or when few users are accessing the system and your user registry is Active Directory.

Cause

Active Directory has reached it's Idle Timeout Limit, sent a "Notice of Disconnection" to WebSEAL,and terminated the incoming network connections from WebSEAL. However, WebSEAL has not honored the disconnect and continues to try and use its now unconnected sockets.

Diagnosis

On the WebSEAL side, running netstat -an | grep <your AD server> Reveals several connections to your registry server. Repeating the same thing on the Registry server shows no connections to WebSEAL. There is no firewall, router or other NAT in between that is silently terminating sessions.

Mitigation

You may wish to establish a cron job that performs at least one authentication per 15 min. 

If you have a firewall or load balancer or other means, have it send a reset packet to both sides after detecting 15 min of idle. Make sure this system is not set to indeterminate session length (that may silently terminate sessions). 



Background

AD

Here’s a relevant article: http://support.microsoft.com/kb/315071
MaxConnIdleTime - The maximum time in seconds that the client can be idle before the LDAP server closes the connection. If a connection is idle for more than this time, the LDAP server returns an LDAP disconnect notification 
 
Typical settings:
 
MaxPoolThreads 4 
MaxDatagramRecv 4096 
MaxReceiveBuffer 10485760 
InitRecvTimeout 120 
MaxConnections 5000 
MaxConnIdleTime 900 
MaxPageSize 1000 
MaxQueryDuration 120 
MaxTempTableSize 10000 
MaxResultSetSize 262144 
MinResultSets 0 
MaxResultSetsPerConn 0 
MaxNotificationPerConn 5 
MaxValRange 1500 
 

Registry Timeouts

the way to govern the length of time a TAM component will wait on a bind 
or search operation with AD is exclusive to the ldap-client-timeout     
setting in activedir_ldap.conf:                                         
                                                                 
[uraf-registry]                                                         
                                                                 
# LDAP Client time-out value (in seconds... default is 0)               
#   0 = synchronous operations; else asynchronous operations             
#   Applies to LDAP simple bind and LDAP search...                       
#   ldap-client-timeout = suggest: 60 seconds to 900 seconds             
ldap-client-timeout = 0                                                 
---------------------------------             

Sample Cron Script

Place this script somewhere handy
COOKIES=/var/webseal-cookies
curl https://webseal1.gattis.org/pkmslogin.form -d username=someAccount -d password=somePassword -d login-form-type=pwd -c $COOKIES
sleep 5
curl https://webseal1.gattis.org/pkmslogout -b $COOKIES

And put this file in your /etc/cron.d/
# simulate a login every 5 min to keep the ldap connection alive
*/5 * * * *     root /opt/webseal-login-keepalive >/var/log/webseal-login-keepalive 2>&1

Make sure and edit the locations in these to match where you wish to place them

Non-AD LDAP mechanisms

there is a mechanism to allow webseal to drop an inactive         
connection, thus preventing webseal from attempting to use a stale       
connection (and then having to wait on the applicable timeout condition 
before "discovering" that it is not going to get a response)...         
                                                                
in the webseald conf file (and same for pdmgrd.conf and pdacld.conf),   
there is the following attribute:                                       
                                                                
ldap-server-config                                                       
                                                                
which points to a conf file (typically called ldap.conf, but the name   
could be anything), which sets various LDAP parameters...               
                                                                
included amongst these is:                                               
                                                                
connection-inactivity                                                   
                                                                
which by default, is set to "0", which implies a persistent connection   
to LDAP...                                                               
                                                                
and, as you mention, if a connection is closed outside of webseal, such 
as by a firewall (usually done based on an inactivity setting at the     
firewall itself), then webseal would still think the connection is valid,     
until it tries it and eventually fails...                         
                                                                
for scenarios such as this, we recommend setting connection-inactivity   
to something less that the firewall's inactivity timeout... for example, 
if the firewall's inactivity timeout is 300 seconds, then you'd want to 
set the connection-inactivity in the TAM ldap.conf to something like 240 
seconds... so that webseal closes the connection before the firewall     
does...                                                                 
                                                                
as load increases, webseal will re-establish the connections to LDAP     
back up to the maximum of 16 if needed...                               
                                                                                                                             
                                                                
p.s. it is also recommended that you set enable and set non-zero values 
to the TAM LDAP timeouts, which are located in the TAM component conf   
file (such as the webseald conf file) in the [ldap] stanza... these are   
commented out by default and set to "0"... uncommenting them and     
setting them to positive values (such as 30 or 60) allows you to control 
how long the TAM component will wait for search / authentication         
requests to be returned by LDAP... as opposed to waiting on the system's 
TCP timeout settings...                                                 



Comments