Rocksolid Light

Welcome to RetroBBS

mail  files  register  newsreader  groups  login

Message-ID:  

"The wages of sin are death; but after they're done taking out taxes, it's just a tired feeling:"


rocksolid / Rocksolid Nodes / Re: rate limiting in rslight

SubjectAuthor
* rate limiting in rslightRetro Guy
`* rate limiting in rslightAnonymous
 `* rate limiting in rslightRetro Guy
  `* rate limiting in rslighttrw
   `- rate limiting in rslightRetro Guy

1
rate limiting in rslight

<cca9cd3838da484861b2129d7d32bb89$1@news.novabbs.org>

 copy mid   Newsgroups: rocksolid.nodes
Path: i2pn2.org!rocksolid2!.POSTED.localhost!not-for-mail
From: retro.guy@rocksolidbbs.com (Retro Guy)
Newsgroups: rocksolid.nodes
Subject: rate limiting in rslight
Date: Wed, 6 Jan 2021 10:06:19 +0000
Organization: Rocksolid Light
Message-ID: <cca9cd3838da484861b2129d7d32bb89$1@news.novabbs.org>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Info: novabbs.org; posting-account="retrobbs1"; posting-host="localhost:127.0.0.1";
logging-data="26704"; mail-complaints-to="usenet@novabbs.org"
User-Agent: Rocksolid Light (news.novabbs.com/getrslight)
X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on novabbs.org
X-Rslight-Site: $2y$10$YxEGNIzfuNIiwhxjk84LYOX.htRsaQn6pOGb45NZ6XvQxx9hxAtX6
 by: Retro Guy - Wed, 6 Jan 2021 10:06 UTC

I've finally set up rate limiting in rslight so that it's not necessary to do so with web server settings.

The server works great for this as long as each client has it's own ip address, which is fine in clearnet and i2p, but does not work well in tor. tor users all have the same ip address, so it ends up blocking all tor users if one is abusing the server.

I'm now limiting per session, and so far it seems to be working fine. It should be fine for clearnet and tor also, but it's really for tor. Most of my tor traffic is badly configured crawlers that just hit the server as fast as they possibly can, so you get many hits per second. Now the logs show the crawlers getting a '429' and regular users are still able to load the site.

I'll keep testing this and see how it goes.

Retro Guy

--
Posted on Rocksolid Light
news.novabbs.org

Re: rate limiting in rslight

<n.1283.3r3v9b@anon.com>

 copy mid   Newsgroups: rocksolid.nodes
Path: i2pn2.org!rocksolid2!def5!.POSTED.bogusentry!not-for-mail
From: poster@anon.com (Anonymous)
Newsgroups: rocksolid.nodes
Subject: Re: rate limiting in rslight
Date: Wed, 06 Jan 2021 04:27:53 -0800
Organization: def5
Message-ID: <n.1283.3r3v9b@anon.com>
References: <cca9cd3838da484861b2129d7d32bb89$1@news.novabbs.org>
Content-Type: text/plain; charset=UTF-8
Injection-Info: def5.org; posting-host="bogusentry:192.168.1.189";
logging-data="27087"; mail-complaints-to="usenet@def5.org"
 by: Anonymous - Wed, 6 Jan 2021 12:27 UTC

>I'm now limiting per session,

I was not aware that rslight uses sessions. How did you do that, cookies ?

Good that it works against the crawlers, though. I just realized that cookies will not work against crawlers that do not take them, so probably you have found another way (really curious here).

--
Posted on def2

Re: rate limiting in rslight

<20210106155913.279b35b5@desktop14.dt>

 copy mid   Newsgroups: rocksolid.nodes
Path: i2pn2.org!rocksolid2!.POSTED.localhost!not-for-mail
From: retro_guy@retrobbs.rocksolidbbs.com (Retro Guy)
Newsgroups: rocksolid.nodes
Subject: Re: rate limiting in rslight
Date: Wed, 6 Jan 2021 15:59:13 -0700
Organization: RetroBBS
Message-ID: <20210106155913.279b35b5@desktop14.dt>
References: <cca9cd3838da484861b2129d7d32bb89$1@news.novabbs.org>
<n.1283.3r3v9b@anon.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Injection-Info: novabbs.org; posting-account="retrobbs1"; posting-host="localhost:127.0.0.1";
logging-data="18418"; mail-complaints-to="usenet@novabbs.org"
X-Newsreader: Claws Mail 3.11.1 (GTK+ 2.24.25; x86_64-pc-linux-gnu)
 by: Retro Guy - Wed, 6 Jan 2021 22:59 UTC

On Wed, 06 Jan 2021 04:27:53 -0800
Anonymous <poster@anon.com> wrote:

> >I'm now limiting per session,
>
> I was not aware that rslight uses sessions. How did you do that,
> cookies ?
>
> Good that it works against the crawlers, though. I just realized that
> cookies will not work against crawlers that do not take them, so
> probably you have found another way (really curious here).
>

What I do is connect to the session in specific files, not every file:
session_start();

Then verify a start time or create a start time (of the session), count
hits (views), then I can monitor hit rates:
$rate = ($_SESSION['views'] / (time() - $_SESSION['starttime']));

If there is a big gap since start time, it is reset to avoid spoofing
rate. Then, reading $rate it's decided whether it's too fast or not and
if it is, it sends a 429 "Too Many Requests" and exits the script
(sending no content).

In my logs, all the crawlers are receiving 429, and the non crawlers
(users) are able to load the page with no issues.

Cookies are not involved, just sessions.

Retro Guy

Re: rate limiting in rslight

<bde362c847086d7194b65713c57ce9cc@def4>

 copy mid   Newsgroups: rocksolid.nodes
Path: i2pn2.org!rocksolid2!.POSTED.127.117.190.215!POSTED.localhost!not-for-mail
From: trw@anon.org (trw)
Newsgroups: rocksolid.nodes
Subject: Re: rate limiting in rslight
Date: Sat, 09 Jan 2021 18:17:04+0000
Organization: def5
Message-ID: <bde362c847086d7194b65713c57ce9cc@def4>
References: <20210106155913.279b35b5@desktop14.dt>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Info: novabbs.org; posting-account="def.i2p"; posting-host="127.117.190.215";
logging-data="31908"; mail-complaints-to="usenet@novabbs.org"
In-Reply-To: <20210106155913.279b35b5@desktop14.dt>
 by: trw - Sat, 9 Jan 2021 18:17 UTC

thx for the explanation. I was just wondering how a specific request is mapped to a session, but I can read it up in the php documentation then.
if there are no cookies involved it is probably done with a session token ? I will check it out.
I have not used sessions in any of my own php code, but fudforum does of course (in the adapted version the session token is embedded in the links, so that the resulting GET string contains the token).

--
Posted on def4

Re: rate limiting in rslight

<20210204190049.0f3eff6a@desktop14.dt>

 copy mid   Newsgroups: rocksolid.nodes
Path: i2pn2.org!.POSTED!not-for-mail
From: retro_guy@novabbs.com (Retro Guy)
Newsgroups: rocksolid.nodes
Subject: Re: rate limiting in rslight
Date: Thu, 4 Feb 2021 19:00:49 -0700
Organization: novaBBS
Message-ID: <20210204190049.0f3eff6a@desktop14.dt>
References: <20210106155913.279b35b5@desktop14.dt>
<bde362c847086d7194b65713c57ce9cc@def4>
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Injection-Info: i2pn2.org; posting-account="retrobbs1";
logging-data="20961"; mail-complaints-to="usenet@i2pn2.org"
X-Newsreader: Claws Mail 3.11.1 (GTK+ 2.24.25; x86_64-pc-linux-gnu)
 by: Retro Guy - Fri, 5 Feb 2021 02:00 UTC

On Sat, 09 Jan 2021 18:17:04+0000
trw <trw@anon.org> wrote:

>
> thx for the explanation. I was just wondering how a specific request
> is mapped to a session, but I can read it up in the php documentation
> then. if there are no cookies involved it is probably done with a
> session token ? I will check it out. I have not used sessions in any
> of my own php code, but fudforum does of course (in the adapted
> version the session token is embedded in the links, so that the
> resulting GET string contains the token).
>

That's interesting. I was wondering how you resolved the sessions in
tor issue :)

My reasons in rslight is that tor users all have the same ip address
from the web server's view, so fail2ban, throttling in the web server,
etc. won't work properly without just blocking all tor users.

I can see in the logs clients being throttled, while other tor users
are not affected. It's much better than when a spider could just shut
the site down by getting itself blocked (so all tor users blocked). The
spiders seem to hit around 5 request per second and get blocked very
quickly.

I wish the operators of the crawlers would check their logs and see all
they are getting is 429 and nothing useful, but they'll probably just
keep it up forever. To get actual stuff to use in your search engine,
or whatever, slow the f down and don't get blocked.

1
server_pubkey.txt

rocksolid light 0.9.7
clearnet tor