Tuesday 27 December 2011

Performance Monitor Implementation

Performance Monitor Implementation

Peoplesoft have a delivered tool called performance monitor using which we can monitor performance of different Peoplesoft Environment. The setup needs a Monitoring system and a monitored system. In my example I am assuming PMN is the monitoring system and PMN1 is the env which needs to be monitored.

Configurations
Monitoring System
1. Login PMN using the id say PSADM and navigate to User security. Add the role PeopleTools Perfmon Client to PSADM user. (the user used to configure the Perf Monitor needs this role)
2. Select PeopleTools, Performance Monitor, Administration, Global Administration.
3. On the Global Administration page, enter the appropriate value in the PPMI URL edit box.
http://PMNwebserver:port/ppmi/PMNsitename/
4. Specify the PPMI user ID and password (PSADM)
5. Click Save.
6. Click Ping Test to ensure that the URL that you entered is valid and the URL is available.
7. On the Appserver side enable the Perf Collator option (process will be PSPPMSRV). Make the Min Instance and Max Instance equal as this process is not Tuxedo does not handle load balancing between PSPPMSRVs
8. Enter the gateway value as and
http:// PMNwebserver:port//PSIGW/PeopleSoftListeningConnector.
9. Schedule the following three AEs (optional, but schedule these in order to avoid data issues) from Peopletools->performance monitor->administration.
Maintain the current PMU table (the PSPM_REAPER program).-every 15 mins
Archive or delete historical performance data (the PSPM_ARCHIVE program).-daily
Manage the lookup tables in the Performance Monitor interface (the PSPM_LOOKUP program). Monday to Friday daily
10. System defaults set to the following (PeopleTools, Performance Monitor, Administration, System defaults)
Archive mode-Delete after 2 days (example)-Rest is set to default

Monitored System

Do the configurations in all systems which needs to be monitored

1. Select PeopleTools, Performance Monitor, Administration, Specify Monitor.
2. On the Specify Monitor page, enter the appropriate value in the Monitor URL edit box.
http://PPMN1webserver:port/monitor/PMN1sitename/
3. Click Ping Test to ensure that you entered a valid URL and that the URL is available.
4. Click Save and Ping PPM to save the URL into the URL catalog, and to ensure that the associated agents are registered and able to share data with PPMI URL.
5. Change the parameter EnablePPM to 1 in Appserver and Process scheduler configuration files and reboot the servers

Monitoring the targets.
Go to Peopletools, Performance monitor, System monitor, system performance
Select the DB to monitor
Alternatively go to Peopletools, Performance monitor, System monitor and select the target DB to monitor. There are separate menus for process scheduler, Appserver and webserver here.

Issues and troubleshooting
Warning Stale Agent Data Detected

This is a normal warning which comes in Perf Monitor, and if you have this issue following are the steps that you can take.

Scenario 1
Reason for all the agents showed stale was due to the tables had exceeded the delivered default maximum row settings on the Global administration page. To access this page,navigate to PeopleTools > Performance Monitor > Administration > Global Administration, within the "Collator Row Limit" section, either increase the value of "Maximum Rows" or set it to 0, where 0 = unlimited.

What is Collator Row Limit?
Limits the amount of data that can be inserted into the PeopleSoft Performance Monitor tables. Collators (PSPPMSRVs) periodically count the number of rows in each of the performance tables. If the total count of rows in PSPMTRANSHIST, PSPMTRANSCURR, and PSPMEVENTHIST is higher than this value, the PSPPMSRV does not insert any more rows into these tables until the number of rows deleted (by the archive program and/or reaper program and/or manual intervention) brings the row count below this limit. If you set this value to 0, the PSPPMSRVs do not check for a row limit.

Scenario#2:
Delete the performance history for the database/system from which you want to remove the message: run the perfmonpurgeall.dms Data Mover script. This enables you to purge all PeopleSoft Performance Monitor tables in the monitoring database. This should be run in monitoring DB (PMN). This scripts deletes all the performance data. You should shut down the monitoring system before doing this


Scenario#3
Another scenario that would cause the "Stale Agent Data Detected" is when no data is being inserted into these tables, even though all the Performance Monitor settings appear to be correctly configured. PSPMTRANSCURR; PSPMTRANSHIST; PSPMTRANSARCH;SPMTRANSHISTCL; PSPMEVENTHIST; PSPMEVENTARCH; PSPMEVENTHISTCL. In this case, the problem is due to the Agent Filters set to "01-Standby". 1. Go to PeopleTools > Performance Monitor > Administration > Agent Filters page. 2. Select 04-Standard from the Reset All Filters drop-down. 3. Click Apply, then Save and Notify Agents (this will require the IB gateway setup). The agents will be active in a few minutes (about 5 minutes). In PT8.45, do make sure that Agent Filter Level field in the System Defaults page (PeopleTools >Performance Monitor >Administration) is set to "04-Standard" instead of default value of "01-Standby".

Scenario#4:
The Agents and PPMI Client were registered successfully as seen in both monitored and monitoring Application Server logs. The agent buffer size on the System definition page is correct. However, in the diagnostic URL, the Show Clients page shows these columns (Item Processed, Estimated Bytes Processed, Max Size, Running Avg Size, and Limit) with value of zero, which shouldn't be the case. This means that no data is being inserted into the Performance tables (PSPMTRANSCURR, PSPMTRANSHIST and PSPMEVENTHIST) despite the fact that the PSPPMSRV process (Performance Collator) is up running. In this case, the issue is due to the "PPM Monitor Buffer Size" on the Web Profile configuration page. It should be set to 51,200 KB (50 MB) rather than 0. This means no buffer was available to be used by the monitor servlet for the incoming performance data. For PT8.45 and higher: If the "PPM Monitor Buffer Size" is grayed out, then the "Enable PPM Agent" is most likely not selected. In this case, check this option to enable the "PPM Monitor Buffer Size" field, then type in 51,200 and uncheck the "Enable PPM Agent" ONLY if you decide to disable the web server agent to be started on a monitored web server. For PT8.44, the "PPMMonitorBufferSize" needs to be added into the Custom Properties tab of the Web Profile Configuration if it's not already there.


Data Type Changes for Oracle/Relscripts/Database_OPTIONS

Data Type Changes for Oracle/Relscripts/Database_OPTIONSThere are few Data types changed in Peoplesoft for Oracle Platform, which plays a crucial role in upgrades, as well as installation.
From App9 on wards Peoplesoft started supporting CLOB and BLOB. CLOBS replaced LONG datatype and BLOBS replaced LONG ROW in Peoplesoft. CLS (Character Length semantic) was another feature introduced in Apps9.







What is CLS?
CLS is a feature introduced in Apps9 and above and is supported only for Unicode Oracle DBs. Before Apps9 to support complex character sets such as Chinese, Peoplesoft used a method called tripling feature. This means a field say FIELD1 in appdesigner which is of type CHARACTER and have a length of n, when this record is built in a Pre 9.0 Unicode DB, the field length automatically gets converted to 3n(bytes). That means FIELD will be of type VARCHAR(3n) in the backend. This feature is used to support huge characters like Chinese, which may come in an unicode DB. One Chinese Character is almost equal to 3 bytes, hence if we build the record with the same length as in appdesigner (as we do for normal non-Unicode DB), it will be difficult to accommodate the Unicode characters. In non Unicode DB tripling is not needed as most of the characters are one byte.
In Pre 9.0 DBs the value of the NLS_LENGTH_SEMANTICS init parameter is set to BYTE (by default NLS_LENGTH_SEMANTICS value is BYTE). That means filed length in DB is interpreted as bytes. This is the reason why Peoplesoft triples the field length to accommodate enough bytes coming from Unicode character sets. NLS_LENGTH_SEMANTICS can also be set to CHAR in oracle, and this allows the length to be treated in Characters rather than bytes. Peoplesoft started adopting this feature from apps 9.0 onwards. That means from 9.0 onwards Unicode DBs should have init parameter NLS_LENGTH_SEMANTICS set to CHAR and there will not be any tripling of field lengths as length is treated in characters.
In a pre App9 DB a field of length 1 in appdesigner will show as VARCHAR(3) in backend and in 9.0 onwards it will show as VARCHAR(1 CHAR) in backend.
LOB and CLS come together with App9 and higher. For Unicode DBs both features will be there and for non-Unicode DBs only LOBS will be enabled


From 8.50
From Peoplesoft 8.50 onwards (tools related) the DATETIME and TIME data types are changed to TIMESTAMP. This is a feature introduced from PT850 (Irrespective of Apps Version)

All Together-Very ImportantIn short CLS and LOBS are an App9 feature (not dependent on tools) and TIMESTAMP is a PT850 feature (not dependent on Apps_)
DATABASE_OPTIONS
DATABASE_OPTIONS is a critical column where the data type changes are recorded. This column is introduced from 847 onwards. Let’s see what all values this column can take. This column value changes during upgrade process, and is a critical column for the peoplesoft tools functionality.
· If the value is zero =neither Apps9 data types nor TIMESTAMP is enabled on the DB. This means Peoplesoft is using old data type.
· If the value is 2 then PeopleSoft is using the new ORA data types (CLS and LOBS).No timestamp is enabled
· If the value is 32, then PeopleSoft is using old data types, (Longs and Legacy Unicode implementation) with TIMESTAMP post PT8.50 conversion enabled
· If the value is 34 then PeopleSoft is using the new Oracle data types supported with PT8.48 (Longs and Legacy Unicode implementation) with TIMESTAMP post PT8.50 conversion enabled


The build process using appdesigner or Data mover have a great impact on this value. This value is checked before any build process and the datatype that should be used during build is decided based on this value. For example if the value is zero VARCHAR fields are built as LONG and if the value is 32 they are built as CLOBs or BLOBs. So if there is any issue with this value the system may go to an inconsistent state.


Install
There is not going to be any bigger impact if we are doing a fresh install, as the datatypes and database_options columns are set automatically.
For PT847,PT848,PT849 System DBs DATABASE_OPTIONS will be zero- as no new datatypes are supported
For PT850, 851 system DBs (PTSYS) DATABASE_OPTIONS will be 32- as only TIMESTAMP is enabled
For Apps9 DB on PT848 and 849 DATABASE_OPTIONS will be 2-Only Apps 9 data type change is enabled
For Apps9 or 91 DBs on PT850 and PT851 DATABASE_OPTIONS will be 34-Both TIMESTAMP and APPs9 change are enabled
For below Apps9 DBs on PT847, PT848, PT849 DATABASE_OPTIONS will be zero- No new Datatypes are enabled
For below Apps9 DBs on PT850, 851 DATABASE_OPTIONS will be 32- Only timestamp is enabled
Relscripts
Relscripts as the name indicates are used to rel up the tools versions. Relscripts contains all the managed tools tables used by Peoplesoft (These are core tables and is needed for basic functionality). Note that relscript contains only core tables and will not contain all system tables. The remaining system tables will be built during the PPLTLS84CUR build process in upgrades.
Normally relscripts comes as the first step in tools upgrade process. Without these tables built normal Appdesigner features, DM and other tools basic functionalities won’t work. This is the reason that relscripts are executed as an initial step so that the essential tools changes are built before we copy PPLTLS84CUR with appdesigner which contains all the new tools objects. If we don’t execute rel scripts we won’t be able to operate even the new tools Appdesigner and DM, by which we complete remaining tools upgrade steps..
In oracle there are four different types of relscripts available from 848. Let’s see why we have four different relscripts
Rel84x.sql- This is the normal relscripts used for non-Unicode pre Apps9 DBs. Pre App9 DBs have the old datatypes and when we upgrade them to new tools we want to use the same data type in which the DB is built. Rel84x.sql contains the old datatypes.
Rel84xn.sql- This is used for all upgrades where target DB is on Apps9 or above. These DBs already uses the new datatypes hence this script is needed as rel84xn contains new datatypes. If we use rel84x.sql instead, this will create the tools tables with old datatypes and will cause issues later.
Rel84xu.sql and rel84xun.sql are the Unicode version of the above scripts

Upgrades
• In Upgrade process the data types are changed using
• Relscripts
• Converting Oracle Time Data types: PSORADataTypesConversion.bat- This is the step where all the Database tables with LONG and LONG ROWS are converted to CLOBS or BLOBS.
This step implements the CLS feature also if the DB is Unicode. This is a crucial step in Unicode DB upgrades. Right before this step you should make the NLS_LENGTH_SEMANTICS parameter to CHAR and execute this step. This step will decrease the column lengths which were tripled earlier. If the parameter is set incorrectly you will face issues. (Step is needed only if you are Upgrading a Pre App9 DB to App9 or above).
• ConvertingDatabasDatatypes: PSORATimestampConversion.bat- This step converts all DATETIME and TIME columns in the DB to TIMESTAMP. (step is need only if you are upgrading a pre-850 release to 850 or above)
The column DATABASE_OPTIONS is changed in the following Upgrade steps.• Updating the Database for Timestamp- This updates database_options to indicate that new timestamp data type is now enabled. The script name is UPGDBOPTIONS_ENABLETIMESTAMP.SQL. This step normally comes after executing the relscripts. The relscript for 850 comes with TIMESTAMP datatypes. So onceyouu rel up the tools to 850 this step is executed to indicate that any further builds will utilize the 850 feature and DB is now in 850.
• Updating Database Options- This updates database_options to indicate that new data types (LOBS and CLS) is now enabled. The script name is UPGDBOPTIONS_ENABLE.SQL. This step is executed right after the new datatypes are enabled. Means after executing all the steps generated by “ConvertingDatabasDatatypes: SORATimestampConversion.bat” step. After this step any further build process will build the tables with LOBs and CLS (for Unicode)





Thursday 22 December 2011

PS Report posting issues

This is a quite frequent issue with reports remaining in posting status , or going to not posted state. Even thopugh the reports gets posted sometimes when we open the same it will prompt for userid/pwd or will show us a blank page.

When a process runs from process scheduler the output files, trace files and other related files gets generated in /appserv/prcs/PS_CFG_HOME/domain/log_output direcotry. When the program is completed these files gets transferred to psreports directory which you specified while creating the PIA.

Points to check when we have issues with posting
1. Check the disk space in the machinewhere psreports dirctory resides.
2. Make sure the report node is configured correctly. Peopletools->process scheduler->report nodes. For HTTP check the URL format (http://webserver:port/psreports/),OS,Transfer protocol,URL Host,URL Port and URL resource (SchedulerTransfer/). For FTP check Home Directory(psreports path),URL,OS,FYP Address (machine where psreports is created) and FTPID,pwd (username/pwd to access the machine where psreports is created)
3. Make sure that one working report node is assigned to the process scheduler. Peopletools->process scheduler->servers, select the server and make sure that report node details are filled in distribution tab.
4. Make sure the id by which you are booting the process scheduler have Report Dist Admin role assigned.
5. Make sure that the default local node have an authentication option of password
(Peopletools->Integration Broker->Integration setup->nodes).
6. Check the psreports directory in Web Profile is correct. Goto Peopletools->webprofile->webprofile configuration and select the WEB profile you are using. Check the value of "Report Repository Path:"- This normally makes an issue when you have a cloned DB.
7. Sometimes the Distribution server process in process scheduler gets Hung. Restart the scheduler in such cases.
8. Check webserver configuration file configuration.properties and make sure that the path is correct in this file
9. Make sure the owner and group have the write permission to write on psreports directory
10. Your local message node should be a trusted node. Peopletools->security->Security Objects->Single Sign on. Make sure you default local nod is there
11. Be consistent with you domain names.

Process Scheduler Issues

Issue 1: Generating Integration Broker .trc file with messages like (PublicationManager::Publish()), eventhough IB services are down Resolution: I find this as a bug with Peoplesoft. This wont do any harm ,but will result in unnecesssary generation of trace files. To avoid this change the parameter LogFence=0 in psprcs,cfg file and reboot the process scheduler
Issue2: Peoplesoft Process scheduler-Processes getting queued.
Resolution There are three tables in Peoplesoft which should be in sync always PSPRCSRQST, PSPRCSQUE and PSPRCSPARMS
Execute the below query to check the tables are in synch
select count(*) from PSPRCSRQSTunion allselect count(*) from PSPRCSQUEunion allselect count(*) from PSPRCSPARMS;
If the counts are different execute the delete queries to synch them and restart the scheduler. This will resolve the issue
DELETE FROM PSPRCSQUE QUE WHERE NOT EXISTS (SELECT 'X' FROM PSPRCSRQST RQST WHERE RQST.PRCSINSTANCE = QUE.PRCSINSTANCE);
DELETE FROM PSPRCSPARMS PARMS WHERE NOT EXISTS (SELECT 'X' FROM PSPRCSQUE QUE WHERE QUE.PRCSINSTANCE = PARMS.PRCSINSTANCE);




You can also check when the process is scheduled to run if it remains queued after doing the above synch up. Also check whether there is any timezone difference between your DB,App and PRcs server machines. This can also cause the processes to get queued.



Issue:3

Database error encountered: SELECT DAYSBEFOREPURGE FROM PS_PRCSPURGELIST WHERE RUNSTATUS = '9'.

When starting the Process Scheduler, this error message appears in the SCHDLR_mmdd.log. This is preventing the purge process from running

Resolution

This is a potential issue for customers upgrading to 8.4 or new customers. The applications upgrade path or installation instructions should include a step to run the prcspurgelist.dms. Some products like CRM and EPM don't include it. Most customers reporting this issue are using CRM or EPM. The prcspurgelist.dms populates the PS_PRCSPURGELIST with the information seen in the System Settings System Purge Options. The PS_PRCSPURGELIST table must contain this information before the purge process can run.

Issue4

The Process Scheduler purge process is not running. The following messages were found in the scheduler log file, schdlr_mmdd.log. "PSMSTPRC.1278194 (0) [03/31/05 01:00:29 PSAPP@machinename](0) Purging expiredrequest(s)...PSMSTPRC.1278194 (0) [03/31/05 01:00:30 PSAPP@machinename](-1) Debuggingdisabled. Comm unication failure during connect(): 79. (143,9)PSMSTPRC.1278194 (0) [03/31/05 01:00:30 PSAPP@machinename](1) Debugging has beenforcibly disabled do to a Configuration or Communication problem.PSMSTPRC.1278194 [03/31/05 01:00:32 PSAPP@machinename](1) (Handler):PSMSTPRC.1278194 (0) [03/31/05 01:00:32 PSAPP@machinename](3) Unable toinitiate purge due to unsuccessful schedule ofthe request using PT_PRCS Application Class"

Resolution The PTPT1200 and PTPT1000 permission lists are required for the Process Scheduler user. These can be made available to the user by adding the roles PeopleTools and PeopleSoft User.For additional information on the security needed for the purge, refer to metalink article: E-PRCS: What Are The Permissions For The Purge Process, Prcsyspurge? (Doc ID 1110701.1).
Issue 5
Peoplesoft nvision programs continues to be in processing status
Resolution
This normally happens when Excel process in windows Hangs. Check the task manager and restart excel if needd.

Issue5

AE programs stays in initiated state
If you check PSAESRV log you see an error message like
'RunAeAsync1 parameter data validation failed'

Resolution:
Check the trace settings associated with the program
setting trace options on an application engine process but using override instead of append in the parameter list can be the issue 



ISSUE6:
Process Scheduler(s) are running fine, then processes remain in a queued state.


Resolution
Things to check:
1) MAX API Aware setting <--- Are all active slots filled?
This setting controls the total amount of concurrently running processes that the Process Scheduler can handle at any time. This value controls all Process Category Types and Process Types. Once the maximum value has been reached, the Process Scheduler cannot start processing new requests.

Note: PSJobs are not considered in this value. PSJobs are a container for other processes.
2) MAX Concurrent setting <--- Are all active slots filled for problem process type?
This setting controls the amount of processes assigned to the corresponding process category that the Process Scheduler can handle at any time. For example, if the Process Category is defined as 5, then a total of 5 processes assigned to that Process Category can be run concurrently. This is limited by the value defined in the Max Api Aware. Once the maximum value has been reached, the Process Scheduler cannot start processing new requests.
3) Is the Process Scheduler successfully updating it's timestamp and heartbeat?
Shutdown the process scheduler, delete it's CACHE directory to clear any possibly corrupt cache,
set TraceSQL=31and LogFence=5 in psprcs.cfg, reconfigure the process scheduler server via the psadmin
utility and then boot up the process scheduler. Gather all log files from the process scheduler LOGS
directory with the most current date for review to see the behavior of the process scheduler on startup for
further clues as to what the problem may be.
4) Check the system Time Zone settings. For UNIX, this may be in different places, for example AIX, the /etc/environment file. Make sure the parameter is correctly stated, e.g. PST8PDT, EST5EDT. The middle figure is the UTC offset. Consult UNIX man pages for details on the particular OS.
CASE 2: Recurrence causes the Process Scheduler to hang
Process Schedulers hang on a recurrence and stop initiating all processes.
SOLUTION:
In this case after reviewing the trace file it was found that the Process Scheduler could not move past a certain process instance and it was found that the instance was a bad recurrence. Once the problem rows were removed from the PSPRCSRQST and PSPRCSQUE tables, the Process Scheduler was able to start processing all other processes. This issue ended up being that the recurrence had overlapping date ranges in the "Run On Specific Dates" section vs the "Exception Dates".
CASE 3: Processes submitted with a blank servername remain queued
When submitting a process online without specifying a Process Scheduler, the process will often stay in queued status in the Process Monitor even though there is a Process Scheduler active. Should you hardcode the process definition to a particular Process Scheduler or select a Process Scheduler from the process request submission page, then the process does run immediately.
SOLUTION:
There are two possible solutions for this:

1) Decrease the heartbeat on the affected Process Scheduler.

OR

2) Configure a Master Scheduler to take over the job of routing the processes.
CASE 4: All processes are queued
All Process requests remain queued.
SOLUTION:
The Process Scheduler tables: PSPRCSRQST and PSPRCSQUE were out of sync. After syncing up the tables to ensure the same process instance numbers were in both tables and recycling the Process Schedulers, the processes started running.
CASE 5: Appengine processes are queued, but other processes are running normally
Appengine processes remain queued, however other processes like Sqrs and Crystals run.

This issue was caused by a code failure as outlined in Bug:11326740 - "psft 1381611000-master scheduler: ae processes stay queued as sqr to e-mail fail".
SOLUTION:
1) A resolution is available in the following releases:
  • PeopleTools 8.46.18 Update ID [701898] / Bug ID [11407475].
  • PeopleTools 8.47.11 Update ID [686615] / Bug ID [11401062].
  • PeopleTools 8.48.05 Update ID [686170] / Bug ID [11401063].
Please review Knowledge Document “How to Download PeopleSoft Bundles and Fixes from My Oracle Support” (Doc ID 1432368.1) for further details.

OR

2) Manually remove the offending Sqr processes.
a) Locate the offending processes by querying the database:
- SELECT PRCSINSTANCE, JOBINSTANCE, RUNDTTM, PRCSNAME FROM PSPRCSRQST WHERE RUNSTATUS=3 AND PRCSTYPE IN ('SQR Report', 'SQR Process')
- SELECT PRCSINSTANCE, JOBINSTANCE, RUNDTTM, PRCSNAME FROM PSPRCSQUE WHERE RUNSTATUS=3 AND PRCSTYPE IN ('SQR Report', 'SQR Process')

b) Of the returned values, run the following statements:
- UPDATE PSPRCSRQST SET RUNSTATUS = 2 WHERE RUNSTATUS=3 AND PRCSTYPE IN ('SQR Report', 'SQR Process')
- UPDATE PSPRCSQUE SET RUNSTATUS = 2 WHERE RUNSTATUS=3 AND PRCSTYPE IN ('SQR Report', 'SQR Process')

c) Commit the changes and bounce all Schedulers in the environment to force the Master Scheduler to pull the updated list from the database.


CASE 6: Processes are queued on a Solaris Process Scheduler
On Solaris Process Schedulers, all Processes intermittenly remain queued.
SOLUTION:
This issue was caused because the Solaris server time kept drifting to the past, which is a known issue on Sun Fire 3800/4800/4810/6800, V1280, and Netra 1280 domains.

To resolve this issue, apply one of the following workarounds:

Setting the variable "tod_broken" to 1 in the domain kernel (see below), or
Reboot the SCs before 528 days of SC continuous uptime (recommended at 500 days), or
Install Patch 112884-04 (ScApp 5.15.3)


CASE 7: Processes remain queued if a non-Master Scheduler is brought down
In a multi-Process Scheduler environment running with a Master Scheduler, if one of the non-Master Schedulers is brought down, processes will remain queued.
SOLUTION:
For exampe, there are two Schedulers: PSNT1 and PSNT2 - this one is configured with "Do Not Redistribute". PSNT1 can run 3 processes, while PSNT2 can run 5. This gives 8 total processes that can be run at any time. 5 processes are submitted to PSNT2 and 3 processes are submitted to PSNT1. Then, PSNT2 is brought down.

The Master Scheduler will try to run the first 5 submitted to PSNT2, but it cannot because it is down and it cannot reassign the processes because of the "Do Not Redistribute" option. As a result, all subsequent processes will remain queued until PSNT2 comes back online and takes care of the first 5 processes.



CASE 8: Processes defined to run on one OS are run on another, leading to queuing behavior
In an environment containing different OS types - ie NT, Unix, certain processes that are defined to run only on a specific OS type, such as NT, are assigned to run on Unix. This results in those processes remaining queued.
SOLUTION:
This issue was caused by a code failure in the Master Scheduler code that caused it to ignore the "Redistribute Workload" Option defined on NT to keep its processes on NT type OS only. This issue was fixed in PeopleTools 8.50.







Wednesday 21 December 2011

Peoplesoft Process scheduler Load balancing

Peoplesoft Process scheduler Load balancing.

I am trying to explain how we can do a load balancing with Peoplesoft process scheduler. I am assuming that we have two process scheduler running on UNIX as PSUNX and PSUNX1, and one scheduler on Windows as PSNT. Lets see how we configure the load balancing.

For load balancing to take place master scheduler process in atleast one of the process schedulers should be active. A Master Scheduler can be activated in any of the Process Scheduler servers in Microsoft Windows and UNIX. This option is enabled by default when you are configuring a new Process Scheduler server in Windows and UNIX. However, this option is not available in the IBM UNIX System Services (USS). Each Process Scheduler domain on Windows or UNIX (except for USS) can be set up to have a Master Scheduler started. However, only one Master Scheduler is active to control the workload at any time. The other Master Schedulers remain in a state of idle. If the active Master Scheduler goes down, then one of the idle Master Scheduler servers take control. If a Master Scheduler is not available, then the PSPRCSRV servers, currently in remote server mode, switch back to standalone mode and query the Process Request table to find work.


1. As an initial step got peopletools->process scheduler->system settings. Make sure that you have a primary OS defined. System load balancing option should be Assign To Server in Any O/S. This will help us to use all available servers in all the operating systems to be used for load balancing. If this is not selected, the process schedulers in other OS apart from your Primary OS will not be used for load balancing. In this case on PSUNX and PSUNX2 will be used.




2. Now goto Peopletools->Process scheduler-> Servers and select all the servers one by one and do the following configurations. Lets take PSUNX as an example
Following are the options we need to monitor




a. Make sure Operating system is correct
b. Server Load Balancing option: Make sure this is set to Use for load balancing. Otherwise this server will not be used for load balancing. When you select Do Not Use Option, the server takes a request only when the name of the server has been specified on the Process Request page or in any of the definition tables.
c. Redistribute Workload Option: Select Do Not Redistribute, Redistribute to any O/S, or Redistribute to Same O/S. When you select the option to redistribute, another active agent can take a request that is originally assigned to this server. When this agent detects that one of the other active agents is no longer active and the server is set up to allow work to be distributed, it takes any queued request that was assigned to the inactive agent and redistributes it, based on the available active agents.

Once these settings are done you can test the load balancing on your system

Peoplesoft CA cleanup procedure.

In case CA fails to detect your environment or you want to start a new search, following is the procedoure to clean PSEMAGENT. Make sure you stop the Agents, CA, Viewer, HUB before you delete these files or directories. Its better to do a cleanup once you completes an MP patch or bundle.

1. Stop PSEMAGENT from all the servers in your environment.
2. Clean the following directories
For Agents, Viewer and Change Assistant: Delete the following directories, under PS_HOME\PSEMAgent for Agents, under PS_HOME\PSEMViewer for the Viewer and under the Change Assistant directory for Change Assistant.
envmetadata/PersistentStorageenvmetadata/scratchpad
envmetadata/data/idsenvmetadata/transactions (if it exists)
For Agents and Change Assistant delete the following file:
envmetadata/data/search-results.xml
For HUB Delete the following directories under PS_HOME\webserv\\applications\peoplesoft\PSEMHUB PSEMHUB\envmetadata\ScratchpadPSEMHUB\envmetadata\PersistentStoragePSEMHUB\envmetadata\transactions (if it exists)
Also delete all files and directories within:
PSEMHUB\envmetadata\data\*.* (files and directories.don't delete the data directory itself)

Note: Make sure that you dont delete matchers.xml. Even if you do, you can copy the same from anothe PS_HOME with same settings

Peoplesoft CA setup for MPs and Bundles

Peoplesoft CA setup

PSEMAGENT (agent) should be running on your machines where the following servers are hosted


Note: If this is a cloned environment make sure you set the GUID field in PSOPTIONS to blank and restart your application servers

Application Server, Web Server, File Server, Batch Server (Process Scheduler).
1. For each agent, edit the configuration.properties, located in /PSEMAgent/envmetadata/config.
a. Set the hubURL=http://< PSEMHUB:8O>/PSEMUB/hub- is the HUB URL. In single server configuration this is same as your webserver port and host, but in multiserver installation give the proper port used by PSEMHUB
b. Set the windows(unix) drivestocrawl=etc.windowsdrivestocrawl=d:/pt851c:/hr91unixdrivestocrawl=/usr/home/findev88/usr/home/finprod88
Make sure you include PS_CFG_HOME path starting with PT850 in addition to PS_HOME
2. On the Change Assistant workstation, set the parameter Drives to Crawl, found in CA > Options > Environment Management Hub tab > to the connectivity directory for your database platform. Give the PS_HOME/PS_CFG_HOME directories also

example:C:\app\oracle\product\11.1.0\client_1\bin;c:\pt851;c:\pscfg
Go to Change Assistant, Options, Environment Management Hub, Server section and enter the machine name in the Server Host area and PSEMHUB port in the Server Port area (which same as PIA port in a single server installation).
3. Enter an environment description by logging into each database via the browser and go to PeopleTools, Utilities, Administration, PeopleTools Options. At the top of the page enter Environment name, short name and System type. The system type is another descriptive field and does not affect operation of Change Assistant.
4. Start PIA if not started already (and the hub if this is a multiserver install) and the agents.Agent is started using PS_HOME\PSEMAgent\StartAgent
Open a browser and paste the hub url into the address section. (The Hub Url is also found in your Configuration.properties.) You should see a status page, with a status of Running for each agent. The Change Assistant machine will also be listed, but will show a status of Not running - this is expected.
5. From webserver machine $PS_HOME\PSEMViewer\ execute StartAgent
If the web server is on Unix, you will have to transfer the resulting files in /PSEMViewer/data to a Windows machine. On the Windows machine, double click on viewer.html in the \PSEMViewer\data directory to review what Change Assistant has discovered about your environment. This is what will be uploaded to the Update Gateway. Click on each heading and verify that the information is present and correct.-This is mainly importany when we use CA to upload enviroment.
7. Goto CA->Tools->validate to check the CA settings are perfect
8. Goto Tools->Options and ensure that in CA tab, update environment is clicked. Then got Environment management tab and click on PING and VIEW button and make sure that all the servers are listed and the ping works properly, 9. If there is any issues in configuration before you retry clean up the CA, using standard CA cleanup process.

Peoplesoft DBA - CA Issues

Issue 1
Unable to connect to your Proxy Server. Is Your Proxy Server (host = , port = 0) defined correctly?- When we tried to upload Peoplesoft Enviroment through CA

Resolution:
This is a firewall issue. As per Oracle Metalink The Network or System Administrator needs to change the IP Sec filter to include the new subnet for Peoplesoft 141.146.54.0 in addition to 192.203.43.0. Next, add IP 141.146.8.119 to your IPsec or firewall Rules in order to connect to Update Gateway through Change Assistant.Once these changes were in place, you should be able to connect. If the server is restricting websites for the web you will need to add the following to your trusted web sites

Also try whether there is any proxy in your organization which allows internet conn ection. If the proxy server is available give the proxy server name. ideally you should be able to ping oracle sites (not just connect) in order for CA to work properly
https://login.oracle.com/
https://metalink3.oracle.com/
https://psft-updates.oracle.com/
http://www.peoplesoft.com/
https://update.peoplesoft.com/

-------------------------------------------------------------------------------------------------
Issue2


When Validating Change Assistant,Discovering Hub Environments - Failed. No environments were found in the Environment Management Hub.
ERROR Message

No Environments were found in the Environment Management Hub.
Note: All agents should be running and sending pulses for all Application,
Batch, File and Web Servers for the environment before Change Assistant is started. 

Solution

1. View the  <PSHOME> 
\PSEMAgent\envmetadata\config\configuration.properties file for each agent.
2. Compare the hubURL line to the one in
C:\Program Files\PeopleSoft\Change Assistant\envmetadata\config\configuration.properties 
3. If they are different edit them to be the same.
4. Start the agents on all servers after following the cleanup procedure (http://peoplesoftdbafriend.blogspot.in/2011/12/peoplesoft-ca-cleanup-procedure.html)

For 8.50 and above versions due to the introduction of PS_CFG_HOME concept, make sure that the drives to crawl includes this location in addition to PS_HOME, and make sure that the peopletools.properties in both PS_HOME and PS_CFG_HOME points to correct location of PS_HOME, if the environment is a cloned one.
 -----------------------------------------------------------------------------------------------------------
Issue3

While trying to apply MP or when clicking View button on tools->option->Environment Management old environments are displayed. 

Follow clean up procedure. Make sure webserver Hub is cleared. 
(http://peoplesoftdbafriend.blogspot.in/2011/12/peoplesoft-ca-cleanup-procedure.html)

Make suere CA version is same as your PS_HOME versions. If you apply any tools patches to PS_HOME make sure CA is re installed.