Troubleshoot - VMC and ICM dowm / IPC basics

It happened that after restart of SAP and underlying OS, I have VMC down. Not just Java part like usually (in that case just restart VMC in prod client - I could write some some very short article about this - on two-three lines), but whole VMC.
As newbie I was confused till I found out that I have down ICM on same application server and in log of it, I found out some more information. I missed this from start, because I forgot that ICM has to be running and checked after each restart on all SAP instances, not just central one  (yeah, I'm still junior).
And reason? Another of my wrong assumption. Even if you restart OS, IPC shared memory is not cleared. It actually saved to some meta file. So in my case, thanks to not cleared IPC, ICM does not have enough memory, fell down on start and VMC was unable to start.

Lessons learned?

- first part:
SMC are important on all application servers - not only central one (my bad memory from basic education).
VMC and ICM processes are using shared memory separately from rest of SAP AS, so they can fall or did not start because of full shared memory on OS.

- now second, more interesting part:
IPC (Inter-Program Communication) is way, how shared memory of OS is managed (I will not get to details now, this sentence is not completely correct, right). Important info is, that memory allocation is stored in /dev/mem and it's not cleared after restart of OS so during restart of SAP you need to free it always manually by submitting usual command:
cleanipc XX remove
(where XX is number of instance/s you need to clear)

if you want to be sure, all is cleared, you could check it by command:
ipcs -mS
or
ipcs -mS|grep -i <SID>adm

Not so precise information in blog - intro of #notsure hastag

Sometimes you will overlook some easy issue, because you will forgot or you are not so skilled. Like me. For that and for my reminder there will be this renewed blog and part of it. It will not be so accurate, because sometimes I can be one, who thinks "I get it", but opposite can be true. In that case i will correct that entry once I found, I'm wrong.

You can find such blog entries with hastag #notsure .
In such blog posts, some theories can be wrong, based on my experience, but not true. I will be more then glad if you will correct any "lie" in them in comments, so I can adjust that post.

But yes, you could argue even about other blog post and their missing info.

OS utilities - SAPPFPAR

OK, this one is really short (mainly I forgot to publish it 3 weeks back :-/ )

"The program sappfpar lets you check the minimum and maximum (worst case) swap space requirements for an SAP application server. It also checks the shared memory requirements and whether or not the parameters em/initial_size_MB and abap/heap_area_total are set correctly." (cite from SAP pages itself)
You will find it in this path:

/usr/sap/<SID>/SYS/exe/run

and those are some simple usage you could use these commands:
sappfpar check pf=<path_to_instance_profile> - to check given profile for errors and evaluate it

sappfpar check pf=<path_to_instance_profile> <parameter_name> - to show parameter value

I don't know what more to write. I nearly did not used it, but yes, it can be handy if you need to check, why is your SAP crashing or during some installations or upgrades.

If you want, you can read this and something little more on original page:
http://help.sap.com/saphelp_nw70/helpdata/en/66/380fb7d43d11d188bd0000e83539c3/content.htm

OS utilities - SAPOSCOL / SAPHOSTEXEC / SAPHOSTAGENT

So today just quick look on two not so needed utilities (I'm party joking now).
SAPOSCOL and SAPHOSTEXEC are utilities for collecting of OS information and relaying that info to SAP application (in case of SAPHOSTEXEC not only this). 

Lets start with SAPOSCOL at it's older. You can find this application in path:
/usr/sap/hostctrl/exe/saposcol

As it was told, app is used to relay information about OS to SAP and there is always one running SAPOSCOL per SAP instance and SAPOSCOL is independent on SAP program (and vice versa)´. All colected data are sent to SAP through share memory, where on SAP is running job SAP_COLLECTOR_FOR_PERFORMANCE (every 24 hours) which writes this data to MONI DB. SAPOSCOL program have several parameters (like -o for showing collected data), witch u can use for his administration.

From SAP NetWeaver components as of release 7.0 EhP2 there should be program SAPHOSTEXEC. It's used for control of SAPHOSTAGENT and also for control of SAPOSCOL program. You can found his executables in:
/usr/sap/hostcontrol/exe/saphostexec

What SAPOSCOL is doing we already know, so what is this SAPHOSTAGENT good for? From SAP kernel 7.20 is this application installed on system to discover and monitor SAP instance, monitor OS through SAPOSCOL and to monitor and manage database on host. There are several other function like instance provisioning.
SAPHOSTAGENT is controled through SAPHOSTEXEC program/commands.


you can found full info on addresses:
SAPOSCOL:
https://help.sap.com/saphelp_nw04/helpdata/en/c4/3a6bff505211d189550000e829fbbd/content.htm
SAPHOSTEXEC:
https://help.sap.com/saphelp_nw73ehp1/helpdata/en/f5/6737c5de81497590d2fbb4fa055dd6/content.htm
SAPHOSTAGENT:
http://help.sap.com/saphelp_nw73ehp1/helpdata/en/48/c6f9627a004da5e10000000a421937/frameset.htm

OS utilities - DPMON / JSMON / JCMON

So for today, we will encounter a few similar tools for OS checking and administration of ABAP and Java part of SAP. Lets start with DPMON.
DPMON is "Dispatcher monitor" This tool is similar to transactions DSM50 or SM66 and it is used for monitoring and some basic task on your ABAP work processess. It is not much, but can be handy in case you are not able to access system via SAPGUI, but you do not want to shut it down (or you have closed ports or whatever).

DPMON is using menu structure with text screens, where you have to go through choices to get to desired function. You can see load statistic, queues or even kill WP.

This tool can be usually found in path:
usr\sap\<SID>\SYS\exe\run

and is used by command:
dpmon pf=<path_to_instance_profile>


When you need to check your Java stack, you have 2 tools now - JCMON and newer JSMON. Both tools are quite similar to DPMON. If you have AS Java older than 7.1 you will have only JCMON on your system.
Both can be found in path:
/usr/sap/<SID>/DVEBMGS<#no>/exe

and are ussualy started by commands:
jcmon pf=<path_to_instance_profile>
jsmon pf=<path_to_instance_profile>
(where <path_to_instance_profile> is path to your DVEBMGS<#no> profile)

Like DPMON, JCMON is using text screens and list for choice of next step / menu. But these menus are numbered with unique numbers, so if you know proper number, you can switch between menus, even if you cannot see number in list. (so you can type any time 20 to see process list)

JSMON is simmilar to JCMON, but it is using color version of "text screens" and instead of numbers, it is using internal commands (like process, display port or help command)

OS utilities - SAPCAR

From today, I'l be posting short stream of more or less short blogs about OS utilities useful in life of SAP administrator.
It's little know-how from one of my colleagues and little add-on from me and internet. But it can be useful.

First in line is SAPCAR.

SAPCAR utility is application for packing/compressing and unpacking of files. It's mainly used for SAP installation packages, but if you want, you can use it on other files to. SAPCAR has 90-95% compress ratio for text files and lover compress ratio for binary files (I did not find later one).
On unix/AIX/linux you will find this utility in directory:
 usr\sap\<SID>\SYS\exe\run
and you should call it with upper-case.On windows, search for sapcar.exe


usage:
SAPCAR -xvf "<filter_for_files_to_extract>" - to extract files
SAPCAR -cvf <name_of_archuve> <files_to pack> - to pack and compress files
SAPCAR -xvf <file> -R <path>  - to give path where to unpack files (relative or absolute)

used parameters explanation:

-x	- extract data
-c	- compress data
-v	- verbose
-f <FILE>	- use archive file FILE (default DEFAULT.SAR)
-R <dir>	- use dir instead of current directory/td>

Note:
because Java packages are compressed in .zip format, do not use SAPCAR on them. Just unzip them (7zip or Winzip).

If you would like to know more about this utility, there is one quite good link:
http://www.easymarketplace.de/SAPCAR.php#OnlineHelp

SAP Open academy ...

Some news could be short.
Did you know about #MOOC? That aberration means Massive Open On-line Course. (You can read more about therm on Wiki) and one of places where you can get them is Open SAP .
This page is managed by SAP company. It is quite good start point for some of SAP educations and you can bound your SAP ID with your Open SAP account if you want.
Disadvantages I found so far: courses are here only for some period and there are not much of them. But who knows what future holds ...