Lucene plugin not working

pir · February 15, 2016, 10:54am

Hi,
I’ve installed a fresh 2.4.8.0 ojs with a 2.3.7.0 database (upgraded to 2.4.8.0 of course).
The ojs is running good.
I want to enable de lucene search plugin, I have followed the steps here https://github.com/pkp/ojs/tree/ojs-stable-2_4_8/plugins/generic/lucene and I am attempting to make the embedded solr installation.

All seemed to install good and when I start the process I can see a solr process running but when I execute the rebuildSearchIndex.php I get this:

LucenePlugin: Clearing index … done
Indexing “” error: Unfortunately the OJS search service is currently offline.

And of course searches doesn’t work.

The only “strange” I have done is to change the solr password as instructions says, but I also put it on the lucene settings via web.

I can’t understand why says that the search service is offline (suppose that is referring to solr), in the same environment I can execute this wget http://localhost:8983/solr/ --proxy=off which downloads an index.html with this:

Welcome to Solr

Welcome to Solr!

Admin ojs

</html

So solr is really running, and I redirected the solr port of the server to my desktop and I can query solr without problem, but it’s empty.

Of course, checked the solr-java.log but no there’s no error.

What’s happening?

bozana · February 16, 2016, 11:58am

Hi pir,

Hmm… Yes, your Solr seems to be running… This error means that something was wrong with the web service call, but out of my mind I don’t know what this could be Do you maybe use proxy i.e. have the proxy part in the file config.inc.php set up?
The error is displayed here in the code: ojs/SolrWebService.inc.php at ojs-stable-2_4_8 · pkp/ojs · GitHub
Thus, the web service response was null and it would be great if you could somehow debug those web server calls happening here. I.e.:
pkp-lib/XmlWebService.inc.php at ojs-stable-2_4_8 · pkp/pkp-lib · GitHub
and/or:
pkp-lib/WebService.inc.php at ojs-stable-2_4_8 · pkp/pkp-lib · GitHub

Thanks a lot!
Bozana

pir · February 17, 2016, 1:38pm

Hi Bozana,
I have printed the variable xmlResult from XmlWebService.inc.php and it shows an html file from squid software saying unauthorized access here:

http://localhost.mysubdomain.cat:8983/solr/ojs/update?

so I can’t understand why is trying to go localhost.mysubdomain instead of localhost.

If I print webServiceRequest->getUrl() in the same code I get that’s trying to go to:

http://localhost:8983/solr/ojs/update?commit=true

and

http://localhost:8983/solr/ojs/dih?command=full-import&clean=false

so seems that ojs is trying a correct url but anything else redirects them outside the machine.

The machine is behind a proxy to access internet, is set in the config.inc.php.

pir · February 17, 2016, 4:22pm

One step closer.
I think I got the problem, something related to proxy, php curl and embedded solr.

Let me explain further tomorrow.

bozana · February 17, 2016, 4:55pm

Hi @pir,

Great! :-))) I thought it has something to do with the proxy, but didn’t know further…

pir · February 18, 2016, 10:06am

Hi @bozana,
Continuing with this issue, let me say before this, I set the variables http_host and http_port in config.inc.php because the server runs through a proxy for getting content from internet, nothing special, during the database upgrade I saw that the script which does this process failed because needs to go outside to check something about new versions, after setting the proxy the script runs perfect so seems that at least, proxy variables are setted correctly.

Now, as I previously explained, seems that the requests to solr where going through the proxy to internet searching for ‘localhost’, which of course failed.
So, looking at the code you can see that the requests to solr are done using a simple curl in php, and before doing the request itself the php curl is setted with some options, between them, one is the proxy config, and is setted with the config you write in config.php.ini, this can be seen here https://github.com/pkp/pkp-lib/blob/6246b313b461146de2c472a8bcd1d337643cdbf4/classes/webservice/WebService.inc.php#L132-L133

This is very strange for me because if you are in an embedded solr installation I think that all the requests to solr are localhost so they don’t have to pass through any proxy, I don’t know if i’m right because my knowledge of networking is not very high.

So, I modified the code and hardcoded the proxy to localhost and the port to 8983 and the reindex worked!

LucenePlugin: Clearing index … done
Indexing “” … 829 articles indexed
Indexing “” … 1799 articles indexed

Anyway, if I try to make a search in ojs I still get:

Unfortunately the OJS search service is currently offline. The technical
administrator of this journal has been informed about the problem.

So, I suppose that somewhere in the code i’m having the same problem about curl proxy.

What I am missing here? is this a bug for embedded solr installations or what?

Thanks!

bozana · February 19, 2016, 10:17pm

Hi @pir

Thanks a lot! The curl proxy consideration has been added recently for all web services calls and we didn’t take into account that it could make problems for the Lucene/Solr plugin. I will take a look i.e. try to find a solution and will then come back to you again as soon as possible.
I think you could try to comment out all those ‘if’ statements where the curl proxy options are set in the file WebService.inc.php i.e. also in other functions there, to see if that works for you at the moment. I just hope that this will not cause another problem for you…

Best
Bozana

pir · February 22, 2016, 10:59am

Thanks @bozana, waiting for a commit fixing this.
Anyway, where is the function that makes the request to solr when you search something in the website? should have the same proxy problem but I can’t see it in the code.

bozana · February 22, 2016, 6:34pm

Hi @pir,

I think either _callGetWebService or _callPostWebService is used, thus if you comment out those ‘if’ statements from both functions, it could work for you for now:

github.com

pkp/pkp-lib/blob/ojs-stable-2_4_8/classes/webservice/WebService.inc.php#L129-L135


		$useProxySettings = $webServiceRequest->getUseProxySettings();
		if ($useProxySettings && $httpProxyHost = Config::getVar('proxy', 'http_host')) {
			curl_setopt($ch, CURLOPT_PROXY, $httpProxyHost);
			curl_setopt($ch, CURLOPT_PROXYPORT, Config::getVar('proxy', 'http_port', '80'));
			if ($username = Config::getVar('proxy', 'username')) {
				curl_setopt($ch, CURLOPT_PROXYUSERPWD, $username . ':' . Config::getVar('proxy', 'password'));
			}

and

github.com

pkp/pkp-lib/blob/ojs-stable-2_4_8/classes/webservice/WebService.inc.php#L202-L208


		curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
		$useProxySettings = $webServiceRequest->getUseProxySettings();
		if ($useProxySettings && $httpProxyHost = Config::getVar('proxy', 'http_host')) {
			curl_setopt($ch, CURLOPT_PROXY, $httpProxyHost);
			curl_setopt($ch, CURLOPT_PROXYPORT, Config::getVar('proxy', 'http_port', '80'));
			if ($username = Config::getVar('proxy', 'username')) {
				curl_setopt($ch, CURLOPT_PROXYUSERPWD, $username . ':' . Config::getVar('proxy', 'password'));

pir · February 23, 2016, 10:48am

Thanks @bozana, I don’t commented those if, I added the proxy and port hardcoded and now the lucene plugin started to work.

I have found the same proxy problem in external feed plugin, it tries to resolve and external rss channel without setting the proxy so it fails… the concrete php class is this https://github.com/pkp/ojs/blob/master/plugins/generic/externalFeed/simplepie/SimplePie.inc.php#L7679-L7686

bozana · February 23, 2016, 9:11pm

Hi @pir,

Thanks for reporting!

I made a patch for the Lucene plugin, s:

and

Is there maybe a possibility for you to test those patches? – You will have to apply them on the original OJS 2.4.8 version i.e. without your changes.

I will also take a look at external feed plugin…

Best!
Bozana

pir · February 24, 2016, 2:56pm

thanks @bozana,
I’m going to test the fix now!

bozana · February 24, 2016, 3:52pm

Hi @pir,

Here is also a patch for the external feed plugin – could you maybe test it too?

Ah, I forgot to insert the link :-\ Here it is: pkp/pkp-lib#1187 consider proxy settings for external feed plugin · bozana/ojs@9adb178 · GitHub

Thanks!!!
Bozana

pir · February 25, 2016, 1:44pm

Hi @bozana,
Thanks for the last edit, It was driving me mad searching the fix of the external feed plugin in the ojs github.

The fix of the lucene works good, I also tested a reindex and all goes well.

The fix of the external feed is not working, I think ther’s a silly mistake about the variable $ch, it should be $fp right?

THANKS!

bozana · February 25, 2016, 4:54pm

Ah, so sorry @pir Me crazy! Here the new commit: pkp/pkp-lib#1187 consider proxy settings for external feed plugin · bozana/ojs@118030d · GitHub
THANKS A LOT!!!

pir · February 26, 2016, 9:41am

Hi @bozana,
yes, the new commit works good.
So seems all fixed now!
I’m having an issue reindexing solr in one of my environments, seems proxy related another time but with the same code in other environment works good so seems my fault. Will tell you.

bozana · February 26, 2016, 11:03am

Thanks a lot @pir!!!

pir · May 18, 2016, 2:13pm

hi, seems that it isn’t fixed already, petitions are going to port 80 instead of 8983

pir · May 19, 2016, 10:03am

@bozana any help please?

bozana · May 19, 2016, 9:38pm

Hi @pir

Hmmm… When is this happening? Always (for example when you search for something and when you index a document)? Hmmm… Are you sure you have all those changes from above – could you double check it?
What is in your lucene plugin settings – what is the option for proxy settings there?
Could you maybe also check what is in your database, in the plugin_settings table – the useProxySettings seem to have there “string” type instead of “bool”. Could you maybe change it to bool and see if it then works?
Hmmm…

Best
Bozana