Hi buddies,
In last few years, i faced issue with SAP PO, it took me one month to identify what is root cause of issue. That is very hard time for me. It happened in my customer production system and i have to check data duplication every day. Now i would like to share with you what is really happened in system.
Related with previous topic https://dangvinhcuong.com/2019/06/03/sap-po-understanding-sap-po-system-logs/ To see how important of reading system logs and investigate issue from technical information.
Issue detail
My SAP PO is handling more than 200 interfaces with million inbound and outbound messages per day including files and synchronous messages. In some days, inbound messages here is sales transaction from POS to SAP POS DM suddenly duplicate randomly. In thousand of message go to SAP PO in the same time, only one or two messages ware duplicated. Sometime outbound IDoc from SAP to PO and PO to files (SFTP Adapter) also generated multiple.
Investigating
Because that issue happened randomly and in the production system then it very hard to re-produce and monitoring. Just imaging, thousand of messages go in and go out SAP PO in the same time, number of messages log is very huge. The monitoring and checking detail message became big challenge.
Case 01, i was with sender and receiver communication channels to make sure that no two channels point to the same folder at the same time. But the question is, if channels pointed to same folders so all messages had to send duplicate. Issue here that happened randomly. it isn’t possibility.
Case 02, check inbound payload and compare with none-duplication messages… There are no difference between both of them.
Case 03, check duplication sending from source system. I used SFTP adapter, many files came to SFTP folders near same time. SAP PO picked files, process and archive in the archive path. In the SFTP application logs, i checked all files from sources was different name. So no duplication sending from source system.
Case 04, System log check, yes just imaging how many line of system logs we have each day. As far as my understood about SAP PO application, SAP PO can have multi app(s) or nodes to consume system resources and process messages faster with parallel mechanism.
Case 05, check UDF (User defined Function). SAP PO provided to you many graphic mapping function but in some scenarios you need to build your own function by Java language. As you know, Java run on JRE (Java Runtime Environment) so all resources of Java Application, In-line program was managed by Java engine. Resources are allocated by setting in Java engine setting. You need to understand behavior of Java language (scope of variables) and recursive functions. In some reason variables located in Java memory and can’t release properly so heap area will be full.
Fixing issue
SAP PO can use Sybase or Oracle DBMS (Database Management System) to store messages log and configuration. Two kind of databases were designed for high performance processing.
Both Sybase and Oracle are using syslogs tablespace to help improve transaction logs during IN/OUT processing.
Check document for more detail
In some reason, the tablespace is not enough space then transaction log can’t be located. Database will raise exception.
Can’t allocate space for object ‘syslogs’ in database ‘DatabaseName‘ because ‘logsegment’ segment is full/has no free extents. If you ran out of space in syslogs, dump the transaction log. Otherwise, use ALTER DATABASE to increase the size of the segment.
Root cause is, SAP is using Sybase or Oracle as back-end database, when messages from SAP ERP or third party system, messages will be processed by SAP PO Apps level. The apps write consuming files as messages into log area then release lock after processing completely to avoid multi-threading consume same files at the same time. Unfortunately, the database’s tablespace is full and PO apps can not write lock messages. Another thread still consume files then send to destination system (here is ERP system) so messages will duplicated.
When you download system log and search log files content you may see detail of database log
00:23:11:172#+0700#Error#/System/Database/sql/jdbc/direct#
com.sap.sql_0003#BC-JAS-PER-SQL#opensqlkernel#C0000A646C28066C00022B1F00007E37#8823653000055442##com.sap.sql.jdbc.direct.DirectPreparedStatement#J2EE_GST_YourDB#0##57C34EA2172E03B6E10000000A646C1B#57C34EA2172E03B6E10000000A646C1B##0#Thread[Managed_Application_Thread_30,5,Managed_Application_Thread]#Plain#com.sap.sql.log.OpenSQLResourceBundle#
SQL error occurred on connection YourDB:YourDB:SAPSR3DB: code={0,number,integer}, state=”ZZZZZ”, message=”Can’t allocate space for object ‘syslogs’ in database ‘YourDB’ because ‘logsegment’ segment is full/has no free extents. If you ran out of space in syslogs, dump the transaction log. Otherwise, use ALTER DATABASE to increase the size of the segment.
“;
SQL statement is “INSERT INTO “BC_MID_CON_TRFC” (“ID”,”STATUS”,”LAST_CHANGE”) VALUES (?,?,?)”.#
Conclusion, when you work with SAP PI/PO you need to take care about Database transaction logs tablespace. Any reason with live system you need to check in SAP PO System logs and JRE Heap to prevent “Out of memory” errors. Knowledge about programming and database architecture also very useful for you to identify issue through error message of SAP PO logs.
Thank you and let me know if any question by comment my topic below.
Cuong Dang
SAP Zero to Hero 