Skip to main content

SAS 9.4 web services NOT up - PermGen issue

Issue:

We deployed a new SAS Software in SAS Web tier. While restarting the web services we were unable to bring up the services. We found out below error message from the web logs (/Lev1/Web/Logs/SASServer1_1):

com.atomikos.icatch.SysException: Error in init: Error in recover

We reached out for tech support and after the investigation he found out that it is PermGen issue.

Solution:

A restart of the SAS web application server instance would temporarily remedy this situation and get you back up, but it would likely re-occur

unless it was caused by some unusual usage scenario.

SAS TS took JVM memory settings from the log, and he discussed this with their performance expert to see if we may need to tweak some JVM arguments. He also suggested that it may also require some more in-depth GC logging in order to determine the root of the issue.

After futher investigation it is found out that it occured due to heavy OLAP/Cube usage and  PermGen issue again. This is a known issue if you have heavy OLAP/Cube usage.

Immediate action:

Tweak the JVM arguments defined in /Lev1/Web/WebAppServer/SASServer1_1/bin/setenv.sh. You should see these: -XX:PermSize=768m -

XX:MaxPermSize=1280m. To help with the PermGen issue I recommend updating these values to

-XX:PermSize=1024m -XX:MaxPermSize=1500m. Then you will need to restart the SASServer1_1 service when you can schedule time.

This may not permanently solve the issue. If it does not, we will have to enable GC logging and do some more in-depth performance analysis,

which could cause some further outage situations. I just want to make you aware of this, it may require a little more experimentation to get the

JVM arguments tweaked just right and avoid this issue. But, start with the recommendation above and it should help. Atlast we end up with following JVM arguments:

-Xmx6000m
-Xms2048m
-XX:PermSize=1024m
-XX:MaxPermSize=1500m

This gives you a bit more JVM memory overhead, which may prevent further outages. Worst case, it does no harm.

Enabling GC logging:

Capturing the GC logging for a week or so will

allow us to make more accurate JVM tuning suggestions. The following shell script code can be added to your setenv.sh file in order to enable

GC logging as well as some other diagnostics that we often use for this type of situation. It should create 2 new log files in the specified

LOG_DIR (you can change LOG_DIR if you choose).

#SAS web tier diagnostics
LOG_DIR="/Lev1/Web/Logs/SASServer1_1"
PROCESS_NAME="SASServer1_1"

# Output files
GC_LOG_FILE="$LOG_DIR/gc$PROCESS_NAME-`date +%d%b%Y-%H:%M:%S`.log"
ULIMIT_FILE="$LOG_DIR/ulimit$PROCESS_NAME-`date +%d%b%Y-%H:%M:%S`.log"
echo "GC data will be written to $GC_LOG_FILE"
echo "user limits written to $ULIMIT_FILE"

#Collect user limits for this process
echo "----------"  >  $ULIMIT_FILE
date               >> $ULIMIT_FILE
echo "----------"  >> $ULIMIT_FILE
echo "ulimit -a: "  >> $ULIMIT_FILE
ulimit -a          >> $ULIMIT_FILE
echo "----------"  >> $ULIMIT_FILE
echo "ulimit -Ha: " >> $ULIMIT_FILE
ulimit -Ha         >> $ULIMIT_FILE

#Enable verbose gc logging

JVM_OPTS="$JVM_OPTS -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"
JVM_OPTS="$JVM_OPTS -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC"
JVM_OPTS="$JVM_OPTS -verbose:gc -Xloggc:$GC_LOG_FILE"

Comments

Post a Comment

Popular posts from this blog

How to create library for Oracle in SAS?

In this article, you will get details about creating a new library using SAS Management Console. Here, I have provided the procedure for creating library for Oracle database. However, the steps will be same for other databases like SQL server, Teradata etc.     Creating connection to Oracle database: To create library, you must first create a connection to Oracle database in your host machine. Host machine is the physical location where your SAS platform is installed. Details like Oracle database hostname/IP address and DSN/SID name is needed. Those details should be entered in tsnnames.ora file. For more details check this article Creating library in SAS Management Console: The steps for creating library for Oracle using SAS Management Console (SMC) can used for other database like SQL Server, My SQL etc. New library wizard: Expand Data Library Manager plug-in, right click on the Libraries and click New Library to start library wizard.  You will be asked
Post a Comment
Read more

SAS - CLI error trying to establish connection

Issue: User asked me to make a database connectivity to SQL Server. They provided following details SQL server hostname and ip address Database/DSN name Username Password I made entry in ODBC.ini file. You know, SQL Server entries were made in ODBC.ini and Oracle entries were made in TNS.ora file. Everything went fine, took back up of odbc.ini, made entry and saved the file. So to test this connection I ran the libname statement in SAS Enterprise Guide 6.1. It throwed following error. Error Message: My DB team showed that they are able to login   14 GOPTIONS ACCESSIBLE; 15 LIBNAME test ODBC DATASRC=SGE_DS SCHEMA=VST USER=sales PASSWORD=XXXXXXXXX; ERROR: CLI error trying to establish connection: [SAS/ACCESS to SQL Server][ODBC SQL Server Legacy Driver][SQL Server]Login failed for user 'sales'. Solution: First I suspected that Login failed for user 'sales' meant the password provided by DB team was wrong. They responded that they were able to login wi
2 comments
Read more

Insufficient authorization to access PIPE error in SAS EG

Issue: When I tried to run SAS code in SAS Enterprise Guide it throws following errors: ERROR: Insufficient authorization to access PIPE. ERROR: Error in the FILENAME statement. Screenshot of error: Solution: This error occurs when you try to run OS commands in SAS code. To run the OS commands in SAS code you need to enable XCMD option. You check it in SAS Management Console by following below steps.   Open SMC -> Expand Servers -> Expand   In SASApp , expand Logical Workspace Server -> right click on Workspace Server. Click properties -> option tab -> advanced options -> launch properties. Check whether Allow XCMD is checked. The issue arises if the Allow XCMD is not checked. In above image, Allow XCMD option is not checked. It should be checked to run OS commands from SAS code. In Unix /Linux machines, this XCMD option can be enabled by using system option XCMD in sasv9 config file or workspaceserver.sh script file. Us
2 comments
Read more