We have a master-slave Solr set up and run live queries only against the slave. Full import (with optimize) happens on master every day at 2 a.m. Delta imports happen every 5 min for one entity and every hour for another entity i.e. our master crontab is like:
# full-import Lists entity
00 02 * * * curl "http://localhost:8080/solr/List/dataimport?command=full-import&clean=true&optimize=true&entity=Lists" > /dev/null
# delta-import Lists entity
*/5 0-1,4-23 * * * curl "http://localhost:8080/solr/List/dataimport?command=delta-import&entity=Lists" > /dev/null
# delta-import ViewedLists entity
23 0-1,4-23 * * * curl "http://localhost:8080/solr/List/dataimport?command=delta-import&entity=ViewedLists" > /dev/null
The following exceptions occur a few times every day in our app logs:
...
org.apache.solr.client.solrj.SolrServerException: Error executing query
at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:95)
at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:118)
...
and ones like these for arbitrary queries:
08:26:39,430 ERROR SolrUtil:107 - SOLR QUERY (LIST): q=*:*&fq=-list_type:CONTRIBUTING AND is_source_list:false AND is_blocked_on_hpage:false&start=0&rows=8&sort=first_publish_date desc
org.apache.solr.client.solrj.SolrServerException: java.net.SocketException: Connection reset
at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:478)
at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:244)
at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:89)
at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:118)
...
If I inspect the localhost.log file in tomcat on the solr slave server, the following exception occurs at the same time:
Apr 06, 2013 7:16:33 AM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet default threw exception
java.lang.IllegalStateException
at org.apache.catalina.connector.ResponseFacade.sendError(ResponseFacade.java:407)
at org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:389)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:291)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:722)
The replication set-up is as follows:
<requestHandler name="/replication" class="solr.ReplicationHandler" >
<lst name="master">
<str name="enable">${enable.master:false}</str>
<str name="replicateAfter">startup</str>
<str name="replicateAfter">commit</str>
<str name="replicateAfter">optimize</str>
<str name="confFiles">solrconfig.xml,data-config.xml,schema.xml,stopwords.txt,synonyms.txt,elevate.xml</str>
</lst>
<lst name="slave">
<str name="enable">${enable.slave:false}</str>
<str name="masterUrl">http://${master.ip}:${master.port}/solr/${solr.core.name}/replication</str>
<str name="pollInterval">00:00:10</str>
</lst>
</requestHandler>
Aside from the full and delta imports, we also have external file fields which are loaded every 15 min. with reloadCache on both master and slave.
What is causing this exception and how to fix it?
Update: Increased the pollInterval
from 10 secs to 1 minute. Still seeing the same issue. Also, forgot to mention that the error is happening on different queries in a short span of time, so it is not query-dependent.
Update 2: Decreased the delta import frequency to once every 10 min and decreased external file field loading and reloadCache to once every hour. Still seeing the exceptions, though the frequency of exceptions seems to have reduced.