Process Scheduler Issues

Issue 1: Generating Integration Broker .trc file with messages like (PublicationManager::Publish()), eventhough IB services are down Resolution: I find this as a bug with Peoplesoft. This wont do any harm ,but will result in unnecesssary generation of trace files. To avoid this change the parameter LogFence=0 in psprcs,cfg file and reboot the process scheduler
Issue2: Peoplesoft Process scheduler-Processes getting queued.
Resolution There are three tables in Peoplesoft which should be in sync always PSPRCSRQST, PSPRCSQUE and PSPRCSPARMS
Execute the below query to check the tables are in synch
select count(*) from PSPRCSRQSTunion allselect count(*) from PSPRCSQUEunion allselect count(*) from PSPRCSPARMS;
If the counts are different execute the delete queries to synch them and restart the scheduler. This will resolve the issue
DELETE FROM PSPRCSQUE QUE WHERE NOT EXISTS (SELECT 'X' FROM PSPRCSRQST RQST WHERE RQST.PRCSINSTANCE = QUE.PRCSINSTANCE);
DELETE FROM PSPRCSPARMS PARMS WHERE NOT EXISTS (SELECT 'X' FROM PSPRCSQUE QUE WHERE QUE.PRCSINSTANCE = PARMS.PRCSINSTANCE);



You can also check when the process is scheduled to run if it remains queued after doing the above synch up. Also check whether there is any timezone difference between your DB,App and PRcs server machines. This can also cause the processes to get queued.



Issue:3

Database error encountered: SELECT DAYSBEFOREPURGE FROM PS_PRCSPURGELIST WHERE RUNSTATUS = '9'.

When starting the Process Scheduler, this error message appears in the SCHDLR_mmdd.log. This is preventing the purge process from running

Resolution

This is a potential issue for customers upgrading to 8.4 or new customers. The applications upgrade path or installation instructions should include a step to run the prcspurgelist.dms. Some products like CRM and EPM don't include it. Most customers reporting this issue are using CRM or EPM. The prcspurgelist.dms populates the PS_PRCSPURGELIST with the information seen in the System Settings System Purge Options. The PS_PRCSPURGELIST table must contain this information before the purge process can run.

Issue4

The Process Scheduler purge process is not running. The following messages were found in the scheduler log file, schdlr_mmdd.log. "PSMSTPRC.1278194 (0) [03/31/05 01:00:29 PSAPP@machinename](0) Purging expiredrequest(s)...PSMSTPRC.1278194 (0) [03/31/05 01:00:30 PSAPP@machinename](-1) Debuggingdisabled. Comm unication failure during connect(): 79. (143,9)PSMSTPRC.1278194 (0) [03/31/05 01:00:30 PSAPP@machinename](1) Debugging has beenforcibly disabled do to a Configuration or Communication problem.PSMSTPRC.1278194 [03/31/05 01:00:32 PSAPP@machinename](1) (Handler):PSMSTPRC.1278194 (0) [03/31/05 01:00:32 PSAPP@machinename](3) Unable toinitiate purge due to unsuccessful schedule ofthe request using PT_PRCS Application Class"

Resolution The PTPT1200 and PTPT1000 permission lists are required for the Process Scheduler user. These can be made available to the user by adding the roles PeopleTools and PeopleSoft User.For additional information on the security needed for the purge, refer to metalink article: E-PRCS: What Are The Permissions For The Purge Process, Prcsyspurge? (Doc ID 1110701.1).
Issue 5
Peoplesoft nvision programs continues to be in processing status
Resolution
This normally happens when Excel process in windows Hangs. Check the task manager and restart excel if needd.

Issue5

AE programs stays in initiated state
If you check PSAESRV log you see an error message like
'RunAeAsync1 parameter data validation failed'

Resolution:
Check the trace settings associated with the program
setting trace options on an application engine process but using override instead of append in the parameter list can be the issue 

ISSUE6:
Process Scheduler(s) are running fine, then processes remain in a queued state.

Resolution

Things to check:

1) MAX API Aware setting <--- Are all active slots filled?
This setting controls the total amount of concurrently running processes that the Process Scheduler can handle at any time. This value controls all Process Category Types and Process Types. Once the maximum value has been reached, the Process Scheduler cannot start processing new requests.

Note: PSJobs are not considered in this value. PSJobs are a container for other processes.
2) MAX Concurrent setting <--- Are all active slots filled for problem process type?
This setting controls the amount of processes assigned to the corresponding process category that the Process Scheduler can handle at any time. For example, if the Process Category is defined as 5, then a total of 5 processes assigned to that Process Category can be run concurrently. This is limited by the value defined in the Max Api Aware. Once the maximum value has been reached, the Process Scheduler cannot start processing new requests.
3) Is the Process Scheduler successfully updating it's timestamp and heartbeat?
Shutdown the process scheduler, delete it's CACHE directory to clear any possibly corrupt cache,
set TraceSQL=31and LogFence=5 in psprcs.cfg, reconfigure the process scheduler server via the psadmin
utility and then boot up the process scheduler. Gather all log files from the process scheduler LOGS
directory with the most current date for review to see the behavior of the process scheduler on startup for
further clues as to what the problem may be.
4) Check the system Time Zone settings. For UNIX, this may be in different places, for example AIX, the /etc/environment file. Make sure the parameter is correctly stated, e.g. PST8PDT, EST5EDT. The middle figure is the UTC offset. Consult UNIX man pages for details on the particular OS.

CASE 2: Recurrence causes the Process Scheduler to hang

Process Schedulers hang on a recurrence and stop initiating all processes.

SOLUTION:

In this case after reviewing the trace file it was found that the Process Scheduler could not move past a certain process instance and it was found that the instance was a bad recurrence. Once the problem rows were removed from the PSPRCSRQST and PSPRCSQUE tables, the Process Scheduler was able to start processing all other processes. This issue ended up being that the recurrence had overlapping date ranges in the "Run On Specific Dates" section vs the "Exception Dates".

CASE 3: Processes submitted with a blank servername remain queued

When submitting a process online without specifying a Process Scheduler, the process will often stay in queued status in the Process Monitor even though there is a Process Scheduler active. Should you hardcode the process definition to a particular Process Scheduler or select a Process Scheduler from the process request submission page, then the process does run immediately.

SOLUTION:

There are two possible solutions for this:

1) Decrease the heartbeat on the affected Process Scheduler.

OR

2) Configure a Master Scheduler to take over the job of routing the processes.

CASE 4: All processes are queued

All Process requests remain queued.

SOLUTION:

The Process Scheduler tables: PSPRCSRQST and PSPRCSQUE were out of sync. After syncing up the tables to ensure the same process instance numbers were in both tables and recycling the Process Schedulers, the processes started running.

CASE 5: Appengine processes are queued, but other processes are running normally

Appengine processes remain queued, however other processes like Sqrs and Crystals run.

This issue was caused by a code failure as outlined in Bug:11326740 - "psft 1381611000-master scheduler: ae processes stay queued as sqr to e-mail fail".

SOLUTION:

1) A resolution is available in the following releases:

• PeopleTools 8.46.18 Update ID [701898] / Bug ID [11407475].
• PeopleTools 8.47.11 Update ID [686615] / Bug ID [11401062].
• PeopleTools 8.48.05 Update ID [686170] / Bug ID [11401063].

Please review Knowledge Document “How to Download PeopleSoft Bundles and Fixes from My Oracle Support” (Doc ID 1432368.1) for further details.

OR

2) Manually remove the offending Sqr processes.
a) Locate the offending processes by querying the database:
- SELECT PRCSINSTANCE, JOBINSTANCE, RUNDTTM, PRCSNAME FROM PSPRCSRQST WHERE RUNSTATUS=3 AND PRCSTYPE IN ('SQR Report', 'SQR Process')
- SELECT PRCSINSTANCE, JOBINSTANCE, RUNDTTM, PRCSNAME FROM PSPRCSQUE WHERE RUNSTATUS=3 AND PRCSTYPE IN ('SQR Report', 'SQR Process')

b) Of the returned values, run the following statements:
- UPDATE PSPRCSRQST SET RUNSTATUS = 2 WHERE RUNSTATUS=3 AND PRCSTYPE IN ('SQR Report', 'SQR Process')
- UPDATE PSPRCSQUE SET RUNSTATUS = 2 WHERE RUNSTATUS=3 AND PRCSTYPE IN ('SQR Report', 'SQR Process')

c) Commit the changes and bounce all Schedulers in the environment to force the Master Scheduler to pull the updated list from the database.

CASE 6: Processes are queued on a Solaris Process Scheduler

On Solaris Process Schedulers, all Processes intermittenly remain queued.

SOLUTION:

This issue was caused because the Solaris server time kept drifting to the past, which is a known issue on Sun Fire 3800/4800/4810/6800, V1280, and Netra 1280 domains.

To resolve this issue, apply one of the following workarounds:

Setting the variable "tod_broken" to 1 in the domain kernel (see below), or
Reboot the SCs before 528 days of SC continuous uptime (recommended at 500 days), or
Install Patch 112884-04 (ScApp 5.15.3)

CASE 7: Processes remain queued if a non-Master Scheduler is brought down

In a multi-Process Scheduler environment running with a Master Scheduler, if one of the non-Master Schedulers is brought down, processes will remain queued.

SOLUTION:

For exampe, there are two Schedulers: PSNT1 and PSNT2 - this one is configured with "Do Not Redistribute". PSNT1 can run 3 processes, while PSNT2 can run 5. This gives 8 total processes that can be run at any time. 5 processes are submitted to PSNT2 and 3 processes are submitted to PSNT1. Then, PSNT2 is brought down.

The Master Scheduler will try to run the first 5 submitted to PSNT2, but it cannot because it is down and it cannot reassign the processes because of the "Do Not Redistribute" option. As a result, all subsequent processes will remain queued until PSNT2 comes back online and takes care of the first 5 processes.



CASE 8: Processes defined to run on one OS are run on another, leading to queuing behavior

In an environment containing different OS types - ie NT, Unix, certain processes that are defined to run only on a specific OS type, such as NT, are assigned to run on Unix. This results in those processes remaining queued.

SOLUTION:

This issue was caused by a code failure in the Master Scheduler code that caused it to ignore the "Redistribute Workload" Option defined on NT to keep its processes on NT type OS only. This issue was fixed in PeopleTools 8.50.