SOA Domain failed during the AdminServer start up

INFO: Configuring the Service Table DataSource...

INFO: Getting Database Defaults...

Error: getDatabaseDefaults() failed. Do dumpStack() to see details.

Error: runCmd() failed. Do dumpStack() to see details.

Problem invoking WLST - Traceback (innermost last):

  File "/u01/oracle/container-scripts/createDomain.py", line 624, in ?

  File "/u01/oracle/container-scripts/createDomain.py", line 157, in createDomain

  File "/u01/oracle/container-scripts/createDomain.py", line 374, in extendSoaDomain

  File "/u01/oracle/container-scripts/createDomain.py", line 311, in configureJDBCTemplates

  File "/tmp/WLSTOfflineIni7524882329358641379.py", line 263, in getDatabaseDefaults

  File "/tmp/WLSTOfflineIni7524882329358641379.py", line 19, in command

Failed to build JDBC Connection object:

        at com.oracle.cie.domain.script.jython.CommandExceptionHandler.handleException(CommandExceptionHandler.java:69)

        at com.oracle.cie.domain.script.jython.WLScriptContext.handleException(WLScriptContext.java:2983)

        at com.oracle.cie.domain.script.jython.WLScriptContext.runCmd(WLScriptContext.java:735)

        at sun.reflect.GeneratedMethodAccessor155.invoke(Unknown Source)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

com.oracle.cie.domain.script.jython.WLSTException: com.oracle.cie.domain.script.jython.WLSTException: Got exception when auto configuring the schema component(s) with data obtained from shadow table:

Failed to build JDBC Connection object:

----------------------

$ pwd

/opt/oracle/product/12.2.0.1/dbhome_1/network/admin

$ cat sqlnet.ora

NAME.DIRECTORY_PATH= (TNSNAMES, EZCONNECT, HOSTNAME)

SQLNET.INBOUND_CONNECT_TIMEOUT=300

SOA Docker AdminServer failed to Start

SUPP_ATTRS = {severity-value=8, rid=0, partition-id=0, partition-name=DOMAIN} SERVER = AdminServer TIMESTAMP = 1591359709126 USERID = <WLS Kernel> MACHINE = f80e78590414 MSGID = BEA-101020 DATE = Jun 5, 2020 12:21:49,126 PM UTC SUBSYSTEM = HTTP CONTEXTID = 6781aeea-b679-439d-afb3-336ef966f685-0000006e TXID =  THREAD = [ACTIVE] ExecuteThread: '52' for queue: 'weblogic.kernel.Default (self-tuning)' SEVERITY = Error

WatchAlarmType: AutomaticReset

WatchAlarmResetPeriod: 30000

> 

<Jun 5, 2020 12:21:49,157 PM UTC> <Error> <Kernel> <BEA-000802> <ExecuteRequest failed

 java.lang.NullPointerException.

java.lang.NullPointerException

        at weblogic.servlet.internal.VirtualConnection.getLocalAddr(VirtualConnection.java:89)

        at weblogic.servlet.internal.ServletRequestImpl.getLocalAddr(ServletRequestImpl.java:1117)

        at com.bea.console.utils.ConsoleRequestListener.requestInitialized(ConsoleRequestListener.java:30)

        at weblogic.servlet.internal.EventsManager.notifyRequestLifetimeEvent(EventsManager.java:438)

        at weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:28)

        Truncated. see log file for complete stacktrace

> 

com.oracle.bpel.client.BPELFault: faultName: {{http://schemas.oracle.com/bpel/extension}remoteFault} messageType:

{{http://schemas.oracle.com/bpel/extension}RuntimeFaultMessage} parts: {{ summary=<summary>oracle.fabric.common.FabricInvocationException: Unable to

invoke endpoint URI "http://soaapdevlb1.corppvt.com:7877/osb/QueryExpenditureService" successfully due to: javax.xml.soap.SOAPException:

javax.xml.soap.SOAPException: Bad response: 503 Service Unavailable from url http://soaapdevlb1.corppvt.com:7877/osb/QueryExpenditureService</summary>

,detail=<detail>Unable to invoke endpoint URI "http://soaapdevlb1.corppvt.com:7877/osb/QueryExpenditureService" successfully due to:

javax.xml.soap.SOAPException: javax.xml.soap.SOAPException: Bad response: 503 Service Unavailable from url

http://soaapdevlb1.corppvt.com:7877/osb/QueryExpenditureService</detail> ,code=<code>503</code>}

Solution –

Added  Server start parameter in config.xml and removed external DNS entry from config.xml

   </web-server-log>

    </web-server>

    <listen-address></listen-address>

        <server-start>

      <arguments>-Dweblogic.security.SSL.ignoreHostnameVerification=true -Dweblogic.configuration.schemaValidationEnabled=false -Xms2g -Xmx2g</arguments>

    </server-start>

    <server-diagnostic-config>

      <name>AdminServer</name>

    </server-diagnostic-config>

  </server>

Different timezone in WebLogic Logs and on OS

Check the time using the below commands,

date
hwclock
If this is as expected then check below

hwclock --localtime


If this is incorrect then synchronise using below command, (with root)

 hwclock --systohc --localtime



 https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/sect-configuring_the_date_and_time-hwclock



 or you add the below startup parameter in weblogic,

 -Duser.timezone=GMT

 but I think it is OS related issue so better you resolve it on OS. 

SOA Database Purge Steps

SOA 12c database purge scripts

To check the purgeable instances,

select state, count(*) from cube_instance group by state;

Purge script,

execute soa.delete_instances ( to_timestamp('01-01-2016','MM-DD-YYYY'),to_timestamp('01-01-2020','MM-DD-YYYY'),20000,60,to_timestamp('01-03-2020','MM-DD-YYYY'),true,null,null,null,true);

Link for reference - 

https://docs.oracle.com/cloud/latest/soa121300/SOAAG/GUID-DBCFFB8F-3B67-4000-9F48-404D16D503DD.htm#SOAAG98235

Parameter details starting from (
1. min_creation_date Minimum creation period in days for the business flow instances.
2. max_creation_date Maximum creation period in days for the business flow instances.
3. batch_size Maximum number of flows selected for deletion. The default value is 20000.
4. max_runtime Expiration time at which the purge script exits the loop
5. retention_period  Specify the time interval in days for which to retain data.
6. purge_partitioned_component  You can invoke the same purge to delete partitioned data.
7. 7. Null,null, null   are place holder.
8. ignore_state hen set to true, purges all open and closed instances within the specified date range

Execute the script using SOAINFRA.

Note- Without null parameter we were getting the place holder issue. 

Restore SOA Production Database on Development environment

Use the below steps to backup and restore SOA schema from production to development environment. Please make sure you have a proper backup to restore if anything goes wrong. 

1. stop the server 
2. Take a backup of the existing database and restore it from backup
3. Take a backup of the SOA domain. 
4. Change the schema username of SOA instance as per prod schema.
5. change the password as per dev schema.
6. Execute the below steps,

Go to below and execute the wlst.sh

< MiddlewareHome>/oracle_common/bin/wlst.sh


modifyBootStrapCredential(jpsConfigFile='./jps-config.xml', username='PREFIX_OPSS', password='PASSWORD')


modifyBootStrapCredential(jpsConfigFile="/user_project/config/fmwconfig/jps-config.xml",username="PREFIX_OPSS",password="PASSWORD")


Reference -

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=423438322466796&id=2427426.1&displayIndex=4&_afrWindowMode=0&_adf.ctrl-state=g9djwouzc_769#FIX

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=418949888681059&id=2267189.1&displayIndex=7&_afrWindowMode=0&_adf.ctrl-state=g9djwouzc_159#SYMPTOM

Docker Commands

Docker useful commands -   

• docker –version
• docker pull <image name>
• docker run -it -d <image name>  -- This command is used to create a container from an image
• docker ps  -- This command is used to list the running containers.
• docker ps -a -- This command is used to show all the running and exited containers
• docker exec -it <container id> bash  -- This command is used to access the running container
• docker stop <container id> -  This command stops a running container
• docker kill <container id> - This command kills the container by stopping its execution immediately. The difference between ‘docker kill’ and ‘docker stop’ is that ‘docker stop’ gives the container time to shutdown gracefully, in situations when it is taking too much time for getting the container to stop, one can opt to kill it
• docker commit <conatainer id> <username/imagename>  -- This command creates a new image of an edited container on the local system
• docker login  -- This command is used to login to the docker hub repositors
• sync && sysctl -w vm.drop_caches=3   -- clear the cache memory on unbutu
• docker logs --tail 300  containerID
• docker logs --since 10 containerID  (10 Minutes)
• docker inspect --format={{.LogPath}} containerID
• tail -f `docker inspect --format={{.LogPath}} containerID`
• grep error `docker inspect --format={{.LogPath}} containerID`