SERP Hacker support forum - serposcope

Support forum for open-source softwares built by serphacker.com : serposcope the rank checker.

You are not logged in.

#1 2016-08-03 07:31:20

WhatNoWebsite
Member
Registered: 2016-06-01
Posts: 21

Increase In Errors Since Upgrade

Hi

I've noticed that each day (with the exception of 1) since I upgrdaded I get errors and have to rescan. Just wondering if you have any suggestions...?

Offline

#2 2016-08-03 08:30:06

Davide
Member
From: CH
Registered: 2016-03-24
Posts: 80
Website

Re: Increase In Errors Since Upgrade

WhatNoWebsite wrote:

Hi

I've noticed that each day (with the exception of 1) since I upgrdaded I get errors and have to rescan. Just wondering if you have any suggestions...?

I have the same issu. Hope he got it asap solved big_smile


easy-develope.ch ¦ kontakt@easy-develope.ch

Offline

#3 2016-08-05 13:38:13

serphacker
Administrator
Registered: 2014-06-13
Posts: 411

Re: Increase In Errors Since Upgrade

Hi, not enough information, I need error logs.

Offline

#4 2016-08-05 14:49:18

WhatNoWebsite
Member
Registered: 2016-06-01
Posts: 21

Re: Increase In Errors Since Upgrade

OK, will post those for you later. Thanks.

Offline

#5 2016-08-06 08:30:22

WhatNoWebsite
Member
Registered: 2016-06-01
Posts: 21

Re: Increase In Errors Since Upgrade

I'm unable to post the log, string is too large.

How might I get this to you...?

Offline

#6 2016-08-06 10:49:38

serphacker
Administrator
Registered: 2014-06-13
Posts: 411

Re: Increase In Errors Since Upgrade

use something like pastebin.com or gist.github.com

Offline

#7 2016-08-08 05:35:23

Davide
Member
From: CH
Registered: 2016-03-24
Posts: 80
Website

Re: Increase In Errors Since Upgrade

So i put for you all my logs, sysconfig and stackdump in a zip file. you can get it here. Download

Hope this will help you to solve the issue big_smile

Last edited by Davide (2016-08-08 05:36:26)


easy-develope.ch ¦ kontakt@easy-develope.ch

Offline

#8 2016-08-08 15:17:42

serphacker
Administrator
Registered: 2014-06-13
Posts: 411

Re: Increase In Errors Since Upgrade

Hi Davide, just checked your logs, so sometimes the checking fail because your captcha provider is overloaded. By example if you check 2016-08-07.log you will see :

[2016-08-07 18:12:09,408] [google-6] INFO  c.s.s.s.c.s.FailoverCaptchaSolver - all captcha solver failed
[2016-08-07 18:12:09,408] [google-6] ERROR c.s.s.s.g.s.GoogleScraper - solver can't resolve captcha (overload ?) error = SERVICE_OVERLOADED

My advice :

_ You are using anticaptcha, you can increase your bid on their control panel, higher bid cost more but you are prioritary in case of overload (higher bid is only apply in case of overload, you will always pay the lower price if their server is not overloaded, check their documentation).
_ Since 2.5 you can configure multiple providers, try to also register an account on deathbycaptcha and decaptcher.

Offline

#9 2016-08-09 10:42:19

Davide
Member
From: CH
Registered: 2016-03-24
Posts: 80
Website

Re: Increase In Errors Since Upgrade

serphacker wrote:

Hi Davide, just checked your logs, so sometimes the checking fail because your captcha provider is overloaded. By example if you check 2016-08-07.log you will see :

[2016-08-07 18:12:09,408] [google-6] INFO  c.s.s.s.c.s.FailoverCaptchaSolver - all captcha solver failed
[2016-08-07 18:12:09,408] [google-6] ERROR c.s.s.s.g.s.GoogleScraper - solver can't resolve captcha (overload ?) error = SERVICE_OVERLOADED

My advice :

_ You are using anticaptcha, you can increase your bid on their control panel, higher bid cost more but you are prioritary in case of overload (higher bid is only apply in case of overload, you will always pay the lower price if their server is not overloaded, check their documentation).
_ Since 2.5 you can configure multiple providers, try to also register an account on deathbycaptcha and decaptcher.


Okay Thank you very much i will try it smile


easy-develope.ch ¦ kontakt@easy-develope.ch

Offline

#10 2016-08-12 12:57:04

Davide
Member
From: CH
Registered: 2016-03-24
Posts: 80
Website

Re: Increase In Errors Since Upgrade

Nothin will solve the error issue.

My config:

Database: mySql

General:
Cron time: 7am (wont work to on another time same problem)
History limit: 0
Captcha: Anticaptcha & DeathByCaptcha

Google:
Number of pages:1
Results per page:100
Pause: min:1, max:3 (same problem with bigger valuse to min:10, max:20)
Maximum threads:20 (same problem with smaller value: 3)
Fetch retries:3 (same problem with bigger value:10)

Proxies:
Try it with 50 payd proxy (wont work)
Try it wth 5k working free proxy (wont work)

Total number of unique keywords in the groups : 2169

Server info:
CPU     Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz (8 core(s))
Version     Plesk v12.5.30_build1205150826.19 os_CentOS 7
OS     CentOS Linux 7.2.1511 (Core)
Ram: 16Gb
Swap: 16Gb
Free disk space: 1TB

Error Log:

[2016-08-12 12:23:05,733] [google-17] INFO  c.s.s.t.g.GoogleTaskRunnable - search "Fahrzeugausbau Bülach" | try 1 | total search done : 153/2169
[2016-08-12 12:23:05,733] [google-17] DEBUG c.s.s.s.g.s.GoogleScraper - GET https://www.google.ch/search?q=Fahrzeugausbau+B%C3%BClach&uule=w+CAIQICISWnVyaWNoLFN3aXR6ZXJsYW5k&num=100 via proxy:http://201.55.46.6:80/ try 1
[2016-08-12 12:23:51,842] [google-17] INFO  c.s.s.s.g.s.GoogleScraper - GOT status=[200] exception=[none]
[2016-08-12 12:23:51,975] [google-17] INFO  c.s.s.t.g.GoogleTaskRunnable - search "huawei news" | try 1 | total search done : 154/2169
[2016-08-12 12:23:51,975] [google-17] DEBUG c.s.s.s.g.s.GoogleScraper - GET https://www.google.com/search?q=huawei+news&uule=w+CAIQICIDUFJU&num=100 via proxy:http://201.55.46.6:80/ try 1
[2016-08-12 12:23:55,981] [google-17] INFO  c.s.s.s.g.s.GoogleScraper - GOT status=[302] exception=[none]
[2016-08-12 12:23:55,981] [google-17] DEBUG c.s.s.s.g.s.GoogleScraper - captcha form detected via proxy:http://201.55.46.6:80/
[2016-08-12 12:23:59,392] [google-17] INFO  c.s.s.s.c.s.FailoverCaptchaSolver - all captcha solver failed
[2016-08-12 12:23:59,392] [google-17] ERROR c.s.s.s.g.s.GoogleScraper - solver can't resolve captcha (overload ?) error = SERVICE_OVERLOADED
[2016-08-12 12:23:59,392] [google-17] DEBUG c.s.s.s.g.s.GoogleScraper - GET https://www.google.com/search?q=huawei+news&uule=w+CAIQICIDUFJU&num=100 via proxy:http://201.55.46.6:80/ try 2
[2016-08-12 12:24:02,564] [google-17] INFO  c.s.s.s.g.s.GoogleScraper - GOT status=[302] exception=[none]
[2016-08-12 12:24:02,564] [google-17] DEBUG c.s.s.s.g.s.GoogleScraper - captcha form detected via proxy:http://201.55.46.6:80/
[2016-08-12 12:24:08,927] [google-17] INFO  c.s.s.s.c.s.FailoverCaptchaSolver - all captcha solver failed
[2016-08-12 12:24:08,927] [google-17] ERROR c.s.s.s.g.s.GoogleScraper - solver can't resolve captcha (overload ?) error = SERVICE_OVERLOADED
[2016-08-12 12:24:08,927] [google-17] DEBUG c.s.s.s.g.s.GoogleScraper - GET https://www.google.com/search?q=huawei+news&uule=w+CAIQICIDUFJU&num=100 via proxy:http://201.55.46.6:80/ try 3
[2016-08-12 12:24:10,560] [google-17] INFO  c.s.s.s.g.s.GoogleScraper - GOT status=[302] exception=[none]
[2016-08-12 12:24:10,560] [google-17] DEBUG c.s.s.s.g.s.GoogleScraper - captcha form detected via proxy:http://201.55.46.6:80/
[2016-08-12 12:24:21,828] [google-17] INFO  c.s.s.s.c.s.FailoverCaptchaSolver - all captcha solver failed
[2016-08-12 12:24:21,828] [google-17] ERROR c.s.s.s.g.s.GoogleScraper - solver can't resolve captcha (overload ?) error = SERVICE_OVERLOADED
[2016-08-12 12:24:21,850] [google-17] WARN  c.s.s.t.g.GoogleTaskRunnable - scrap failed for huawei news because of ERROR_CAPTCHA_INCORRECT
[2016-08-12 12:24:21,850] [google-17] WARN  c.s.s.t.g.GoogleTaskRunnable - no more proxy, stopping the thread
[2016-08-12 12:24:21,850] [google-17] INFO  c.s.s.t.g.GoogleTaskRunnable - google thread stopped
[2016-08-12 12:24:22,116] [Thread-53] WARN  c.s.s.t.g.GoogleTask - 219 proxies failed during the task
[2016-08-12 12:24:22,116] [Thread-53] WARN  c.s.s.t.g.GoogleTask - 2015 searches have not been checked
[2016-08-12 12:24:22,133] [Thread-53] INFO  c.s.s.t.AbstractTask - task done for module GOOGLE

i run serposcope service like this:

 (java -jar -Dserposcope.conf=/root/serposcope/serposcope.conf serposcope.jar &) 

my java version:

java version "1.8.0_60"
Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)

i really dont know what i should do. every day i have to rescan, sometimes the rescan wont work to.


easy-develope.ch ¦ kontakt@easy-develope.ch

Offline

#11 2016-08-12 14:20:38

serphacker
Administrator
Registered: 2014-06-13
Posts: 411

Re: Increase In Errors Since Upgrade

how many captcha service did you configure ?

Offline

#12 2016-08-12 14:24:53

Davide
Member
From: CH
Registered: 2016-03-24
Posts: 80
Website

Re: Increase In Errors Since Upgrade

serphacker wrote:

how many captcha service did you configure ?

this two: but i never had problems before with just one service.

Davide wrote:

Captcha: Anticaptcha & DeathByCaptcha

Edit: but why the thread stops until no proxy is there. it will be better he try again and start at the top of the proxy list.

Last edited by Davide (2016-08-12 14:26:23)


easy-develope.ch ¦ kontakt@easy-develope.ch

Offline

#13 2016-08-12 14:26:56

serphacker
Administrator
Registered: 2014-06-13
Posts: 411

Re: Increase In Errors Since Upgrade

I need the full logs to examine. (But in the previous log you sent, you had only one service configured, and already have the same problem.)

Offline

#14 2016-08-12 14:29:32

Davide
Member
From: CH
Registered: 2016-03-24
Posts: 80
Website

Re: Increase In Errors Since Upgrade

yes i gave you the full log, and there was just one service, now i have two and nothing changed. if you want i can give you full log since i have 2 services running

Last edited by Davide (2016-08-12 14:33:48)


easy-develope.ch ¦ kontakt@easy-develope.ch

Offline

#15 2016-08-12 14:51:01

serphacker
Administrator
Registered: 2014-06-13
Posts: 411

Re: Increase In Errors Since Upgrade

yes paste it on gist.github.com and paste the link here

Offline

#16 2016-08-12 14:56:35

Davide
Member
From: CH
Registered: 2016-03-24
Posts: 80
Website

Re: Increase In Errors Since Upgrade

https://gist.github.com/anonymous/46615 … 511ffd1712 10.8.16

https://gist.github.com/anonymous/043fd … 3e0cbc20cb 11.8.16

https://gist.github.com/anonymous/e3b58 … 4d964882c3 12.8.16

Here are the last three days, im not shure but i start yesterday evening with the second service


easy-develope.ch ¦ kontakt@easy-develope.ch

Offline

#17 2016-08-12 15:14:06

serphacker
Administrator
Registered: 2014-06-13
Posts: 411

Re: Increase In Errors Since Upgrade

So it failed for the same reason (captcha service overloaded). Your captcha service configuration was ok : I see there was 2 services used, I will try to improve that again with more retry but really, it doesn't depends on me but more on the external service which are unavailable.

Also your proxy are VERY bad and it contribute a lot to your inability to performs checks. Please don't use public proxy, 1 private proxy is better than 100 public proxy. By using public proxy you have a lot of network error, and it increase a lot the number of required captcha.

Offline

#18 2016-08-15 06:24:20

Davide
Member
From: CH
Registered: 2016-03-24
Posts: 80
Website

Re: Increase In Errors Since Upgrade

I forgot something, i hve three ip adresses registered on my server. can i bind dem by using bind#my.ip.adress ??

if this will work i will buy mor ip addresses.

Edit:
It works to bind ip addresses. but i have allredy the same problem. It works better but not perfect, now i can scan to  65% and then it crash.

Is there a way to add "auto recheck failed keywords" and to setup how much to auto recheck. i think this will be a nice feature.

Last edited by Davide (2016-08-15 10:28:47)


easy-develope.ch ¦ kontakt@easy-develope.ch

Offline

#19 2016-08-16 07:28:01

Davide
Member
From: CH
Registered: 2016-03-24
Posts: 80
Website

Re: Increase In Errors Since Upgrade

so i got it running without any errors, now i try to improve the best settings, this takes some days, i will report all things for you to understand. big_smile


easy-develope.ch ¦ kontakt@easy-develope.ch

Offline

#20 2016-08-17 07:42:00

Davide
Member
From: CH
Registered: 2016-03-24
Posts: 80
Website

Re: Increase In Errors Since Upgrade

Date/Time: 2016-08-16 07:00:21
Total number of unique keywords in the groups : 2206
Captchas: 0
Errors: 0
Duration: 02:51:34	
Status: DONE_SUCCESS

Configuration:
Captcha-services: "Anticaptcha and DeathByCaptcha"
Proxy: 10 100% privat Proxies! (Bind to different local IP)

Number of pages: 1
Results per page: 100
Pause: 20-30
Maximum threads: 5
Fetch retries: 7
Date/Time: 2016-08-17 07:00:21	
Total number of unique keywords in the groups : 2206
Captchas: 10
Errors: 0
Duration: 02:23:10	
Status: DONE_SUCCESS

Configuration:
Captcha-services: "Anticaptcha and DeathByCaptcha"
Proxy: "10" 100% privat Proxies! (Bind to different local IP)

Number of pages: 1
Results per page: 100
Pause: 15-25
Maximum threads: 5
Fetch retries: 7

Last edited by Davide (2016-08-17 07:48:55)


easy-develope.ch ¦ kontakt@easy-develope.ch

Offline

#21 2016-08-18 06:13:41

Davide
Member
From: CH
Registered: 2016-03-24
Posts: 80
Website

Re: Increase In Errors Since Upgrade

Date/Time: 2016-08-18 07:00:21	
Total number of unique keywords in the groups : 2206
Captchas: 71
Errors: 971
Duration: 01:04:40
Status: DONE_WITH_ERROR

Configuration:
Captcha-services: "Anticaptcha and DeathByCaptcha"
Proxy: "10" 100% privat Proxies! (Bind to different local IP)

Number of pages: 1
Results per page: 100
Pause: 10-20
Maximum threads: 5
Fetch retries: 7

Really strange big_smile allways the same reason:

[2016-08-18 08:05:01,269] [google-4] ERROR c.s.s.s.g.s.GoogleScraper - solver can't resolve captcha (overload ?) error = SERVICE_OVERLOADED
[2016-08-18 08:05:01,295] [google-4] WARN  c.s.s.t.g.GoogleTaskRunnable - scrap failed for Domain Schweiz because of ERROR_CAPTCHA_INCORRECT
[2016-08-18 08:05:01,296] [google-4] WARN  c.s.s.t.g.GoogleTaskRunnable - no more proxy, stopping the thread
[2016-08-18 08:05:01,296] [google-4] INFO  c.s.s.t.g.GoogleTaskRunnable - google thread stopped
[2016-08-18 08:05:01,369] [google-0] INFO  c.s.s.s.c.s.FailoverCaptchaSolver - all captcha solver failed
[2016-08-18 08:05:01,369] [google-0] ERROR c.s.s.s.g.s.GoogleScraper - solver can't resolve captcha (overload ?) error = SERVICE_OVERLOADED
[2016-08-18 08:05:01,387] [google-0] WARN  c.s.s.t.g.GoogleTaskRunnable - scrap failed for Sushi Richterwil because of ERROR_CAPTCHA_INCORRECT
[2016-08-18 08:05:01,387] [google-0] WARN  c.s.s.t.g.GoogleTaskRunnable - no more proxy, stopping the thread
[2016-08-18 08:05:01,387] [google-0] INFO  c.s.s.t.g.GoogleTaskRunnable - google thread stopped
[2016-08-18 08:05:01,696] [Thread-433] WARN  c.s.s.t.g.GoogleTask - 10 proxies failed during the task
[2016-08-18 08:05:01,696] [Thread-433] WARN  c.s.s.t.g.GoogleTask - 971 searches have not been checked
[2016-08-18 08:05:01,712] [Thread-433] INFO  c.s.s.t.AbstractTask - task done for module GOOGLE
[2016-08-18 08:05:01,712] [pool-2-thread-1] INFO  s.s.CronService - history pruning is disabled

easy-develope.ch ¦ kontakt@easy-develope.ch

Offline

#22 2016-08-18 06:22:24

serphacker
Administrator
Registered: 2014-06-13
Posts: 411

Re: Increase In Errors Since Upgrade

did you increase your minimum bid on anticaptcha ?

Offline

#23 2016-08-18 06:22:58

serphacker
Administrator
Registered: 2014-06-13
Posts: 411

Re: Increase In Errors Since Upgrade

also you should reduce the number of thread, set it to 3, or better 1.

Offline

#24 2016-08-18 06:25:56

Davide
Member
From: CH
Registered: 2016-03-24
Posts: 80
Website

Re: Increase In Errors Since Upgrade

Yes i increase my minimum bid. And if i use just 3 threads you think it will work? i take a try tomorrow morning on next cronjob.


easy-develope.ch ¦ kontakt@easy-develope.ch

Offline

#25 2016-08-19 11:01:07

Davide
Member
From: CH
Registered: 2016-03-24
Posts: 80
Website

Re: Increase In Errors Since Upgrade

Date/Time: 2016-08-19 07:00:21	
Total number of unique keywords in the groups : 2251
Captchas: 0
Errors: 0
Duration: 04:01:39	
Status: DONE_SUCCESS

Configuration:
Captcha-services: "Anticaptcha and DeathByCaptcha"
Proxy: "10" 100% privat Proxies! (Bind to different local IP)

Number of pages: 1
Results per page: 100
Pause: 10-30
Maximum threads: 3
Fetch retries: 7

by using 3 threads at same time it takes to much time to scan. i try to play a little bti around and report it. big_smile


easy-develope.ch ¦ kontakt@easy-develope.ch

Offline

Board footer