Tuesday, 27 September 2016

wms - GeoServer 2.9.2 database connection pooling/thread locking issue


I am mocking GetMap requests. Scenario is that 50 users use only "database layers". Each user "sends" 8 GetMap requests every ~22s for ~300s which sums up to 5200 GetMap requests for vector data. Problem is that with these parameters I see that thread count starts to grow almost immediately and usually reaches ~2k threads. When inspecting with VisualVM I saw that 1.5k threads where BLOCKED: ~50% by org.geoserver.catalog.ResourcePool.getCacheableFeatureType(ResourcePool.java:943) and other half by org.geotools.jdbc.JDBCDataStore.getPrimaryKey(JDBCDataStore.java:1062). Thread count tends to grow when Java heap reaches max point but not always - it also grows rapidly even when heap is at ~50%.


When inspecting Tomcat logs I see:


org.geoserver.platform.ServiceException: Rendering process failed
...

Caused by: java.lang.RuntimeException: Unable to obtain connection: Cannot get a connection, pool error Timeout waiting for idle object
...

Increasing max connections from 20 to 100 in GeoServer makes threads accumulate even faster.


There are 6 CPU's and 5GB RAM 2 of which are given to GeoServer's Tomcat.


Any ideas on what is the root of such problem?


SOLUTION: DB server had not enough CPU power so sessions were waiting to be processed. Increasing CPU count from 2 to 8 seem to solve extremely high tread count issue.




No comments:

Post a Comment

arcpy - Changing output name when exporting data driven pages to JPG?

Is there a way to save the output JPG, changing the output file name to the page name, instead of page number? I mean changing the script fo...