SAS Platform Admin Blog

We got a weird issue in our SAS environment. All of the sudden users were not able to connect the SAS Servers. SAS services such as ObjectSpawner and Metadata Server were running fine. It is obvious that there is no issue with metadata as the user was login to the application. To make this issue more complex we didn’t find any error in the log file. When the user try to connect to the server (by clicking on the + button in SAS Enterprise Guide) all they can see is Windows hour glass. It is not throwing any error except for few where the dialog box showed a error pointing to IOM. However we can’t make any sense of them. Following are the investigation we made during the issue: 1.        Modification in configuration file by checking the date 2.        Permission change to workspaceserver.sh 3.        File system mounted properly 4.        Whether files are creating in SASWORk while connecting to SAS EG 5.        Firewall problems by pinging from local to host (server) 6.

Below is the message you get in metadata log when some user remove a role from group or add a role to a group. 2016-01-15T13:51:56,249 INFO  [10777807] 4216288:venkat.heart - Removed Member IdentityType=IdentityGroup  Name=Group Developers, ObjId=XXXXXXXXX.XXXXXXXX from Role  Name=Management Console: Scheduler Only, ObjId=XXXXXXXXX.XXXXXXXX. 2016-01-15T13:52:34,522 INFO  [10779877] 4216288:venkat.heart - Added Member IdentityType=IdentityGroup  Name=Group Admin, ObjId=XXXXXXXXX.XXXXXXXX to Role  Name=Management Console: Scheduler Only, ObjId=XXXXXXXXX.XXXXXXXX. If this is made by some unknown user then search with following term. [sas@fghg9 Logs]$ grep Removed SASMeta_MetadataServer_XXXXXX_fghg9_XXXXX.log or [sas@fghg9 Logs]$ grep Removed SASMeta_MetadataServer_XXXXXX_fghg9_XXXXX.log | grep Role or [sas@fghg9 Logs]$ grep Removed SASMeta_MetadataServer_XXXXXX_fghg9_XXXXX.log | grep Role| grep Management 

Here I have given SAS license renewal steps. This is the output directly taken from putty while renewing license in our environment. While renewing license I didn't restart the SAS services. Licence updated successfully without any issues. Green color text are system generated. Blue color text is the input. [sas94@fghg99 9.4]$ cd /sas/software/SASFoundation/9.4 [sas94@fghg99 9.4]$ ./sassetup Welcome to SAS Setup, the program used to renew your SAS software. Some conventions used throughout SAS Setup are:        *   indicates which menu selection is the default choice      ( )   indicates the default response to a prompt        !   starts a sub-shell on your system        h   displays help for a menu selection or prompt        g   goes back to a previous menu selection or prompt        q   quits SAS Setup at any point Setup Utilities Menu --------------------  * 1.  Renew SAS Software    -------------------------------    g: Go back   q: Quit    h: Help    

Problem: Users were not able to connect using DIS or EG due to access denied to SAS platform. Error: Application could not log on to the server. The user id or password is incorrect. Troubleshooting 1.       Verify all server 2.       Connect to all nodes through ssh using AD id. I was not able to connect using my AD id but I was able to connect using my local id (/etc/passwd). Solution: Restart Centrify client. If this error reoccurs we have to restart Centrify on affected node. 

lsb.users is the configuration file for Platform LSF 9.1.x. SAS Admin will modify this file if they need to configure user groups and to limit job slots. File location - LSB_CONFDIR/cluster_name/configdir Steps to make change in this file: Logon to lsf admin account. Take backup of lsb.users and make required changes in the file. Execute  badmin reconfig  Verify lsf is running fine by using  bhost command.  It is NOT required to restart LSF to make the changes effective. The changes will effective once badmin reconfig command is done. To verify if the user has been added correctly you must type bugroup -l It will show you the groups and users who belong to those groups. You can verify if it was set correctly.

Here are some interesting SAS papers about automation and configuration. These papers really improved our user experience. Paper SAS1887-2015 - Automating a SAS® 9.4 Installation without Using Provisioning Software: A Case Study Involving the Setup of Machines for SAS Regional Users Groups Paper 2060-2015 - Automating Your Metadata Reporting Paper SAS1857-2015 - Hands-Off SAS® Administration — Using Batch Tools to Make Your Life Easier I referred this paper to automate the migration from SAS 9.2 to SAS 9.4. Paper SAS1904-2015 Your Top Ten SAS® Middle-Tier Questions Paper 2883-2015 - How to Implement SAS 9.4 on an Amazon Web Services Cloud Server Instance Paper 373-2011 - Troubleshooting Tips and Techniques for a SAS® 9.2 Business Intelligence Environment To get the above papers visist  http://support.sas.com/resources/papers/ and search with the ID in the search bar. You can also search with the heading mentioned in the above blog.

Here are some tools you should be familiar with if you are a SAS Admin. Tools monitor CPU: First thing to check when your SAS programs performs poor is CPU.Following tools can be used. glance (multi-purpose) sar (multi-purpose) ps xload top Memory: SAS uses WORK as temporary storage, however PDV and input buffer created in memory. It has to be free for your program to perform fast. Tools: glance (multi-purpose) sar (multi-purpose) size vmstat ipcs Disk: Disk involves WORK, home directory and SAS intallation and configuration directory. There should be enough space in all the directory where SAS lookup. Tools glance (multi-purpose) sar (multi-purpose) bdf df iostat Network: It is the last component we check when we have some issues in SAS Application. Remember, SAS is client-server model and network plays an important role. Many SAS components depends on network resouce. SAS metadata and OLAP server demands robust network for qucik response rate. SAN

Issue: SAS Management Console job scheduling plugin file event shows greyed out browse button.  This is a Linux server tier with SAS 9.4 M2 and Platform Process Manager 9.1 as scheduler. SMC Job Scheduling plugin does not show 'Browse' button enabled on New File Event dialog as attached in screenshot. Screenshot: Solution: The Platform Process Manager was not associated to the same machine as the WorkSpace server (the SAS Application Server).  Using metabrowse edit mode metadata was updated for PM so it would have same machine name as WS.      The SAS Application Server is what is used to browse the file system.  If the AppServer cannot be connected to then it will not be able to use the remote browse dialog. There should be a workspace server running on the machine where PM is defined and running.  You can check this in metadata by opening an interactive SAS session and running the METABROWSE command in the command box.  If your SAS session does not have

Few days back I posted article about running Deployment registry in SAS Server running in Linux based OS. This tutorial is about running it in Windows OS. Deployment registry information is useful if you need to find whether any hotfix has been installed and the hotfix number. It also give information about the all software components installed in your system. Running deployment registry in Windows OS (client machine) is bit trickier because it is not managed by experienced admins. To run deployment registry open command prompt and use below command: "C:\Program Files\SASHome2\SASPrivateJavaRuntimeEnvironment\9.4\jre\bin\java" -jar "C:\Program Files\SASHome2\deploymntreg\sas.tools.viewregistry.jar" sas.tools.viewregistry.jar is the Java binary file which on execution create a text and html output on the same folder. I have SAS 93 and SAS 94 client installed in my machine, so my SAS 9.4 clients are under SASHome2. SAS home folder depends on your system.

Before any new software installation, it is recommended to backup the software and config directory in SAS. We usually do this using rsync command in our environment. Below is the command I used to backup the software and config files. Compute Tier Backup: rsync -arA /sas/software /backup   rsync -arA /sas/conifg  /backup Run the above command using ROOT user. You may get below error if you run this command using non-root user: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6] I am using rsync utility which is available in many Unix/Linux flavors.The a option will preserve the permissions, A option will preserve ACL and r option will recursively backup all the sub-directories. While doing backup for sas-config-directory, metadata server must be STOPPED. For sas-software-directory it is not necessary to stop metadata server. So backup the sas-software-directory first and stop the metadata server and

[root@localhost john.snow]# gunzip centrify-samba-4.5.7-rhel3-x86_64.tgz [root@localhost john.snow]# ls adinfo.txt       centrify-samba-4.5.7-rhel3-x86_64.tar   id_rsa.pub    oradiag_john.snow  smb.conf authorized_keys  DFX81_9B5GGH_70111643_LINUX_X86-64.txt  network.pcap  sasuser.v94            test.txt [root@localhost john.snow]# ls -lrt total 74344 -rwx------ 1 john.snow admin     6021 Oct  3  2012 authorized_keys drwxr-xr-x 2 john.snow admin     3864 Oct 27 10:44 sasuser.v94 drwxr-xr-x 3 john.snow admin     3864 Oct 27 10:44 oradiag_john.snow -rw-r--r-- 1 root          root      11212 Nov 28 12:17 smb.conf -rw-r--r-- 1 john.snow admin      396 Nov 30 16:40 id_rsa.pub -rw-r--r-- 1 john.snow admin     2104 Dec  1 14:03 DFX81_9B5GGH_70111643_LINUX_X86-64.txt -rw-r--r-- 1 root          root         33 Dec  8 11:23 test.txt -rw-r--r-- 1 root          root    1069081 Dec 10 14:25 network.pcap -rw-r--r-- 1 root          root      22970 Dec 10 14:43 adinfo.txt -rw-r--r-- 1

Download SAS Binary File from SAS FTP server Copy the File to production environment using Winscp client .   Export the path where hotfix is placed Path->/export/home/sas/Hot_Fix Change the ownership and permission. chown sas B25259s6.bin chmod 755 B25259s6.bin B25259s6.bin is the hotfix file Set Display environment Variable  Export DISPLAY=<node>:0 Run the Binary Files. ./B25259s6.bin –silent –sashome “/apps/sas/”

1. From the root directory of the dfexec.cfg installation, run: ./bin/dfdbconf. 2. Select A. to add a data source. You can also use dfdbconf to delete a data source if it is no longer needed. 3. Select a template for the new data source by choosing a number from the list of available drivers. 4. You are prompted to set the appropriate parameters for that driver. The new data source is then added to your odbc.ini file. Once you have added all of your data sources, the interactive ODBC Viewer (dfdbview) application can be used to test your connection. For example, if you added a data source called my_oracle, run: ./bin/dfdbview my_oracle (from the installation root) to test the connection. You may be prompted for a user name and password. If the connection succeeds, you will see a prompt from which you can enter SQL commands and query the database. If the connection fails, DIS displays error messages describing one or more reasons for the failure. Note: When configuring t

Below steps are specific for metadata User folder migration, but can be used for other object too. I used MobaXterm, as DISPLAY option is not enable in Putty. Step 1) Insert the correct user metadata folder location in user.txt at source environment. $cat > user.txt /Users/White, Walter /Users/White, Skyler /Users/White, Walter Jr.   Step 2) export DISPLAY.  $export DISPLAY=ip.address   If we don’t  export DISPLAY, we will get below error: No X11 DISPLAY variable was set, but this program performed an operation which requires it. For more information, view the export log file: /home/lannister/export/log/Export_150523062423.log The export process has failed. Step 3) Run below script in source environment. for i in {1..3}   /*depends on number of user */ do /* will cut the user  location one by one from file user.txt */ j=`sed -n "$i"p user.txt`    /* this is batch export script, and is used to create package on required objects

To find the deployment registry information run below command: /sas-home-directory/java/jre1.8.0_45/bin/java -jar sas.tools.viewregistry.jar The o/p of the command will generate DeploymentRegistry.html and DeploymentRegistry.txt file. Just make sure the user has the write access at the location. You can also change the location using below command. /sas-home-directory/java/jre1.8.0_45/bin/java -jar sas.tools.viewregistry.jar -out /home/piper/ Please refer to the link http://support.sas.com/kb/35/968.html

Issue: In SAS 9.4, you notice that Metadata Server is up and running but backup is in hung state.  Solution: At present there is no working solution but to have a graceful restart of Metadata. After restart you can see the backup works normally.  We contacted SAS TS for this issue but at present there is no working solution or patch.

Issue: I see below error message in SAS Enterprise Miner log: ERROR: SSL Error: Missing CA trust list Solution: This issue occurs if SSL certificate is not set correctly. To rectify this problem, you need to mention the SSL certificate location in SSLCALISTLOC option. SSLCALISTLOC option can be invoked in many ways. Here I will explain invoking this option using Workspace Server.  Stop the Object spawner. Take backup of   /sas-config-directory/Lev1/SASApp/WorkspaceServer/sasv9_usermods.cfg and opne the file   in your favorite text editor.  Add the below line to the file SSLCALISTLOC="/etc/pki/tls/cert.pem" Save the file. Start the Object spawner.  Run below code in SAS Enterprise Guide to check the settings change: proc options option=SSLCALISTLOC value; run;

SAS Eminer 13.2 shows error message while creating project. Below is the error message I got while creating project. This server location is already registered for use by either another project or SAS application, or the user does not have permission to create a project at this location. Try using a different name for your project or enter a different path. Solution: This issue occurs if the user doesn't have write access to metadata. Add the user to any group that has write metadata access. 

We had an user who gets this error message whenever user opens the portal: Solution: The problem is the user does not have SAS folder created. To check if the user has SAS folder or not: Open SAS Management Console -> Authorization Manager -> Resource Management -> By Application -> SAS Folders -> User Folders Check under User Folder for the directory. We found out that the user doesn't have folder created. To create folder: Add the user to the group which has write metadata access. Then ask the user to login to the portal Then remove the group which you have added from the user. Adding the user to group with write metadata is temporary which allows user to create SAS home folder. 

Recently I faced issue with activemq, because of which the web services were not starting in SAS 9.4. I followed below mentioned steps as workaround to bring up the services for the error I faced. Follow these steps if only the web server don’t come up, and you notice the error mentioned below: Error message: 2015-12-03 23:49:54,066 | INFO  | Using Persistence Adapter: KahaDBPersistenceAdapter | org.apache.activemq.broker.BrokerService | main 2015-12-03 23:49:54,187 | INFO  | KahaDB is version 4 | org.apache.activemq.store.kahadb.MessageDatabase | main 2015-12-03 23:49:54,229 | INFO  | Recovering from the journal ... | org.apache.activemq.store.kahadb.MessageDatabase | main 2015-12-03 23:49:54,230 | INFO  | Recovery replayed 1 operations from the journal in 0.026 seconds. | org.apache.activemq.store.kahadb.MessageDatabase | main 2015-12-03 23:49:54,384 | INFO  | Apache ActiveMQ 5.7.0 (localhost, ID:*******-29526-1449204594252-0:1) is starting | org.apache.activemq.broker.Brok

Recently we got an scenario where user need to transfer file from production to test. His condition is to transfer data without login to production environment. So below is the code we did to achieve his condition. Option comamid=tcp  Remote=hostname.7552; Filename rlink "/sas/software/SASFoundation/9.4/misc/connect/tcpunix.scr"; signon user=<user_id> password="<password>";      Libname localdev  "<local library where file needs to be copied>"; Rsubmit; libname test "<remote library from where we need to copy>"; proc download inlib=test              outlib=localdev; endrsubmit; signoff; More references on this topic: Connecting Simply with SAS/CONNECT Michael G. Sadof, MGS Associates, Inc., Bethesda, MD. Structuring Base SAS for Easy Maintenance Gary E. Schlegelmilch, U.S. Dept. of Commerce, Bureau of the Census, Suitland MD

If you are a SAS Admin and want to test some admin related feature in SAS code like strace, symbolgen fullstimer then you can do it directly from batch. Generally, SAS developers use tools like SAS Enterprise Guide or SAS DI studio to make changes to their program. Changing in client application is a tedious task. You need first make modification and then re-deploy it. The whole process will take atleast an hour. In batch mode, in OS like *nix, Linux you can use the following command. /sas/config/Lev1/SASApp/BatchServer/sasbatch.sh -log /sas/qat/Lev1/SASApp/BatchServer/Logs/job_for_batch_#Y.#m.#d_#H.#M.#s.log -batch -noterminal -logparm "rollover=session"  -sysin /home/timothy/sales.sas Command and its purpose: /sas/config/Lev1/SASApp/BatchServer/sasbatch.sh - This will invoke the batch server. The location of the batchserver changes as per your environment. However everything after config will be same in all environment. -log /sas/qat/Lev1/SASApp/BatchServer/Logs/jo

Below is the shell script to find out top 10 users utilizing work directory. This script tested in Solaris and may or may not work in *.nix versions. ## Script to find Top users in Work Directory (/work) echo  " \n Top 10 Work directory and Memory in GB " > Temp_Mail.txt echo  " \n Size  Directory_Name \n" >> Temp_Mail.txt du -sh /work/SAS*|/usr/xpg4/bin/grep -E 'T'|grep -v 'K'|grep -v 'M'|sort -nr|head -10 >> Temp_Mail.txt du -sh /work/SAS*|/usr/xpg4/bin/grep -E 'G'|sort -nr|head -10 >> Temp_Mail.txt du -sh /work/SAS*|/usr/xpg4/bin/grep -E 'T'|grep -v 'K'|grep -v 'M'|sort -nr|head -10|cut -c 13-39 > Temp_User.txt du -sh /work/SAS*|/usr/xpg4/bin/grep -E 'G'|sort -nr|head -10|cut -c 13-39 >> Temp_User.txt var1=`sed -n '1p' Temp_User.txt` var2=`sed -n '2p' Temp_User.txt` var3=`sed -n '3p' Temp_User.txt` var4=`sed -n '4p' Temp_Us