Skip to main content

SAS 9.4 web services NOT up - PermGen issue

Issue:

We deployed a new SAS Software in SAS Web tier. While restarting the web services we were unable to bring up the services. We found out below error message from the web logs (/Lev1/Web/Logs/SASServer1_1):

com.atomikos.icatch.SysException: Error in init: Error in recover

We reached out for tech support and after the investigation he found out that it is PermGen issue.

Solution:

A restart of the SAS web application server instance would temporarily remedy this situation and get you back up, but it would likely re-occur

unless it was caused by some unusual usage scenario.

SAS TS took JVM memory settings from the log, and he discussed this with their performance expert to see if we may need to tweak some JVM arguments. He also suggested that it may also require some more in-depth GC logging in order to determine the root of the issue.

After futher investigation it is found out that it occured due to heavy OLAP/Cube usage and  PermGen issue again. This is a known issue if you have heavy OLAP/Cube usage.

Immediate action:

Tweak the JVM arguments defined in /Lev1/Web/WebAppServer/SASServer1_1/bin/setenv.sh. You should see these: -XX:PermSize=768m -

XX:MaxPermSize=1280m. To help with the PermGen issue I recommend updating these values to

-XX:PermSize=1024m -XX:MaxPermSize=1500m. Then you will need to restart the SASServer1_1 service when you can schedule time.

This may not permanently solve the issue. If it does not, we will have to enable GC logging and do some more in-depth performance analysis,

which could cause some further outage situations. I just want to make you aware of this, it may require a little more experimentation to get the

JVM arguments tweaked just right and avoid this issue. But, start with the recommendation above and it should help. Atlast we end up with following JVM arguments:

-Xmx6000m
-Xms2048m
-XX:PermSize=1024m
-XX:MaxPermSize=1500m

This gives you a bit more JVM memory overhead, which may prevent further outages. Worst case, it does no harm.

Enabling GC logging:

Capturing the GC logging for a week or so will

allow us to make more accurate JVM tuning suggestions. The following shell script code can be added to your setenv.sh file in order to enable

GC logging as well as some other diagnostics that we often use for this type of situation. It should create 2 new log files in the specified

LOG_DIR (you can change LOG_DIR if you choose).

#SAS web tier diagnostics
LOG_DIR="/Lev1/Web/Logs/SASServer1_1"
PROCESS_NAME="SASServer1_1"

# Output files
GC_LOG_FILE="$LOG_DIR/gc$PROCESS_NAME-`date +%d%b%Y-%H:%M:%S`.log"
ULIMIT_FILE="$LOG_DIR/ulimit$PROCESS_NAME-`date +%d%b%Y-%H:%M:%S`.log"
echo "GC data will be written to $GC_LOG_FILE"
echo "user limits written to $ULIMIT_FILE"

#Collect user limits for this process
echo "----------"  >  $ULIMIT_FILE
date               >> $ULIMIT_FILE
echo "----------"  >> $ULIMIT_FILE
echo "ulimit -a: "  >> $ULIMIT_FILE
ulimit -a          >> $ULIMIT_FILE
echo "----------"  >> $ULIMIT_FILE
echo "ulimit -Ha: " >> $ULIMIT_FILE
ulimit -Ha         >> $ULIMIT_FILE

#Enable verbose gc logging

JVM_OPTS="$JVM_OPTS -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"
JVM_OPTS="$JVM_OPTS -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC"
JVM_OPTS="$JVM_OPTS -verbose:gc -Xloggc:$GC_LOG_FILE"

Comments

Post a Comment

Popular posts from this blog

Insufficient authorization to access PIPE error in SAS EG

Issue: When I tried to run SAS code in SAS Enterprise Guide it throws following errors: ERROR: Insufficient authorization to access PIPE. ERROR: Error in the FILENAME statement. Screenshot of error: Solution: This error occurs when you try to run OS commands in SAS code. To run the OS commands in SAS code you need to enable XCMD option. You check it in SAS Management Console by following below steps.   Open SMC -> Expand Servers -> Expand   In SASApp , expand Logical Workspace Server -> right click on Workspace Server. Click properties -> option tab -> advanced options -> launch properties. Check whether Allow XCMD is checked. The issue arises if the Allow XCMD is not checked. In above image, Allow XCMD option is not checked. It should be checked to run OS commands from SAS code. In Unix /Linux machines, this XCMD option can be enabled by using system option XCMD in sasv9 config file or workspaceserver.sh script f...
2 comments
Read more

The authentication server is not SETUID ROOT error in SAS

Question: When validating the SAS Server from SAS Management Console, I received the following error: The authentication server is not SETUID ROOT.  So, I ran the setuid.sh utility and restarted the services many times. I just checked the elssrv sasauth sasperm setuid bit. There were no error in sasauth-debug.log, sasauth-access.log, sasauth-error.log.  Any suggestions? Answer: Please do the following:    1) Run /<SASConfig>/Lev<X>/ObjectSpawner/ObjectSpawner.sh stop  2) Edit /<SASConfig>/Lev<X>/ObjectSpawner/ObjectSpawner.sh and add the code shown below right after SCRIPT=`basename $0`:  if [ -n ""$TKPATH"" ]; then  unset TKPATH  fi   if [ -n ""$TK_PATHLIST"" ]; then  unset TK_PATHLIST  fi    3) Run /<SASConfig>/Lev<X>/ObjectSpawner/ObjectSpawner.sh start  The above code change in ObjectSpawner.sh should fix the issue.
1 comment
Read more

SAS - CLI error trying to establish connection

Issue: User asked me to make a database connectivity to SQL Server. They provided following details SQL server hostname and ip address Database/DSN name Username Password I made entry in ODBC.ini file. You know, SQL Server entries were made in ODBC.ini and Oracle entries were made in TNS.ora file. Everything went fine, took back up of odbc.ini, made entry and saved the file. So to test this connection I ran the libname statement in SAS Enterprise Guide 6.1. It throwed following error. Error Message: My DB team showed that they are able to login   14 GOPTIONS ACCESSIBLE; 15 LIBNAME test ODBC DATASRC=SGE_DS SCHEMA=VST USER=sales PASSWORD=XXXXXXXXX; ERROR: CLI error trying to establish connection: [SAS/ACCESS to SQL Server][ODBC SQL Server Legacy Driver][SQL Server]Login failed for user 'sales'. Solution: First I suspected that Login failed for user 'sales' meant the password provided by DB team was wrong. They responded that they were able to login wi...
3 comments
Read more