I have now a functional full text search working for rslight. It's
written in a separate script right now for testing. It's just easier to
mess with the code that way without breaking anything.

I can now run a full search on a group and it's very fast. It accepts
AND/OR, sorts by relevance and can highlight the terms in results.
Also, it does not do things like find "cat" in "cathy" like the current
search does.

I'm cleaning it up now and plan to get it online on rslight.i2p in a
week or two for further testing.

Retro Guy

Thats really cool. Did you finally put the messages into a db as well, or did you find some other way ?

cheers

trw

--
Posted on def2

On Sun, 28 Feb 2021 09:20:18 -0800
Anonymous <poster@anon.com> wrote:

> Thats really cool. Did you finally put the messages into a db as
> well, or did you find some other way ?

Yes, there is an option to use tradspool or put articles in a db
(sqlite). Search will only be available if using a db.

Really the only reason to use tradspool is if you need it for some
particular application on your system, or just like it that way. Moving
forward, rslight will favor storing in a db for multiple reasons.

Retro Guy

On Mon, 1 Mar 2021 01:43:02 -0700
Retro Guy <retro_guy@novabbs.com> wrote:

> On Sun, 28 Feb 2021 09:20:18 -0800
> Anonymous <poster@anon.com> wrote:
>
> > Thats really cool. Did you finally put the messages into a db as
> > well, or did you find some other way ?
>
> Yes, there is an option to use tradspool or put articles in a db
> (sqlite). Search will only be available if using a db.

I'm doing something a bit different here than I see in other forum
searches. To avoid a ton of results simply because the terms are in the
quoted text, I am only searching the reply to the quote. This may be
awesome, or it may suck, we'll see. I can change it to search the whole
body if I choose to.

I hate searching a forum and getting a while bunch of results inside
the same thread due to quoting. I'm hoping this might be a nice way to
handle it in rslight. The only results in any thread should be if the
respondent used the terms. Then once you click on the article, you are
in the thread and can look around as necessary.

Any comments are appreciated as I'm still working on it. It is working
nicely so far :)

Retro Guy

This is a simple result from a search. Still doing this in a separate script.

I searched 'vichan AND fudforum' in 100 articles in rocksolid.nodes, which took 4.7 milliseconds on my slow vm. The specific results in each snippet are highlighted when results are returned <b></b>. Then below this, I searched 'vichan OR fudforum':

Message-ID: <d0f272eee501dba7d282ed580fe0da6c$1@www.novabbs.com>
Subject: vichan and rslight
-----

Now that def is close to having <b>vichan</b> working independent of rslight (if desired), I'm really looking forward to seeing both interfaces implemented.

Once ready, I'd like to deploy a <b>vichan</b> site on my end also, maybe in tor. Since I have now completely dropped <b>fudforum</b>, I'd...
--------------------------------
Message-ID: <n.1268.460pfw@anon.com>
Subject: sqllite, huh =?UTF-8?B?Pw==?=
-----

that sounds just lovely. Correct database setup is one more unpleasant parts, although I have to say it was very easy with <b>vichan</b> (not so much with <b>fudforum</b>). But if it is all handled inside your code, that should reduce this problem to zero.

cheers

trw

--------------------------------
Message-ID: <n.1272.3j5yok@anon.com>
Subject: Re: sqlite, huh =?UTF-8?B?Pw==?=
-----
...For both <b>fudforum</b> and <b>vichan</b> I did some research also on this, but could not find anything that would suggest that unsafe procedures were used.

So if the code is very similar between rslight and <b>vichan</b>, maybe this could be interesting to look at (I mean, maybe I can switch...
--------------------------------

Search for 'vichan OR fudforum':

Message-ID: <d0f272eee501dba7d282ed580fe0da6c$1@www.novabbs.com>
Subject: vichan and rslight
-----

Now that def is close to having <b>vichan</b> working independent of rslight (if desired), I'm really looking forward to seeing both interfaces implemented.

Once ready, I'd like to deploy a <b>vichan</b> site on my end also, maybe in tor. Since I have now completely dropped <b>fudforum</b>, I'd...
--------------------------------
Message-ID: <n.1268.460pfw@anon.com>
Subject: sqllite, huh =?UTF-8?B?Pw==?=
-----

that sounds just lovely. Correct database setup is one more unpleasant parts, although I have to say it was very easy with <b>vichan</b> (not so much with <b>fudforum</b>). But if it is all handled inside your code, that should reduce this problem to zero.

cheers

trw

--------------------------------
Message-ID: <20201216161452.5f1fb78a@desktop14.dt>
Subject: Re: sqllite, huh ?
-----
...It's the same in <b>vichan</b>, that's where you see statements like:

$query = prepare("INSERT INTO ``search_queries`` VALUES
(:ip, :time, :query)");

Then the variables are inserted into the placeholders with bindvalue.
data should be sanitized for use using this step.

Both <b>vichan</b> and rslight use PDO so the...
--------------------------------
Message-ID: <af47f5c0e519077aa6b86daf82701cd1@def4>
Subject: Re: vichan and rslight
-----
...if the <b>vichan</b> surface is accepted well, i could also think about dropping it. on the other hand, def3 is the node with the most visitors by far on my end, so people really seem to like this forum style. let's see.

btw if you roll out <b>vichan</b> as...
--------------------------------
Message-ID: <bde362c847086d7194b65713c57ce9cc@def4>
Subject: Re: rate limiting in rslight
-----
...I have not used sessions in any of my own php code, but <b>fudforum</b> does of course (in the adapted version the session token is embedded in the links, so that the resulting GET string contains the token).

--------------------------------
Message-ID: <n.245.21rwrv@anon.com>
Subject: loop for new messages
-----

Well, i can call the script that does the import to <b>vichan</b> as often as need be, and it will import one message at a time.
I could not loop in this script itself, this throws an error because of missing imports (guess this was really just designed to import...
--------------------------------
Message-ID: <n.1272.3j5yok@anon.com>
Subject: Re: sqlite, huh =?UTF-8?B?Pw==?=
-----
...For both <b>fudforum</b> and <b>vichan</b> I did some research also on this, but could not find anything that would suggest that unsafe procedures were used.

So if the code is very similar between rslight and <b>vichan</b>, maybe this could be interesting to look at (I mean, maybe I can switch...
--------------------------------
--
Posted on Rocksolid Light
news.novabbs.org

On Fri, 5 Mar 2021 09:22:10 +0000
retro.guy@rocksolidbbs.com (Retro Guy) wrote:

> This is a simple result from a search. Still doing this in a separate
> script.

I have it working fine creating snippets. I can use these for overboard
also, which should speed that up significantly.

Searching only the reply portion of messages seems to produce a nice
result. There is not every message in the thread in the result due to
quoting. Only the message with the terms, then you're in the thread and
can look around if you wish.

The snippet below (from <s24mcf$eja$1@dont-email.me>) is from a message
that is about 85% quote, so only the reply text (below) is what is
searched:

--------------------
Your sentiment is absolutely right. Which is why I use the open source
digikam tool. It's been around for well over a decade and is still being
supported. It is also cross platform working on windows, macOS and
linux. Being open source means that it'll never be pulled.
--------------------

I now need to modify the search script to offer searching the body, and
decide whether it should be per group, section or everything on the
site. The hard part is done, and is working well.

Retro Guy

Retro Guy wrote:

> On Fri, 5 Mar 2021 09:22:10 +0000
> retro.guy@rocksolidbbs.com (Retro Guy) wrote:

>> This is a simple result from a search. Still doing this in a separate
>> script.

> I have it working fine creating snippets. I can use these for overboard
> also, which should speed that up significantly.

I'm getting the interface going for search in rslight now. The backend is up and running fine.

It's quite fast, and produces a reasonably useful snippet for each result. Results are ranked with hopefully most accurate at top. So, words closer together, or more occurences should rank higher. Attached is a pic of a simple result for 'vichan OR fudforum'

Retro Guy

Retro Guy wrote:

> I'm getting the interface going for search in rslight now. The backend is up and running fine.

There we go. It now searches the entire 'section'. Not the whole site, just the section where you click Search. I can add per-group later but right now it's working well.

I cleaned up the output to better highlight the results, and pad the space between results to make it easier to read. pic attached.

Retro Guy

Retro Guy wrote:

> Retro Guy wrote:

>> I'm getting the interface going for search in rslight now. The backend is up and running fine.

This is now live on rslight.i2p. It's not finished, but it's easier now to test on a live site with a lot of messages.

If you want to try it, the radio button still defaults to 'Subject', so you need to click 'Body' to search the body of a message. Right now it searches all the groups in the 'section' you are in.

Retro Guy
--
Posted on Rocksolid Light
news.novabbs.org

Retro Guy wrote:

> Retro Guy wrote:

>> Retro Guy wrote:

>>> I'm getting the interface going for search in rslight now. The backend is up and running fine.

> This is now live on rslight.i2p. It's not finished, but it's easier now to test on a live site with a lot of messages.

Seems to be working ok. I've also upgraded novabbs.com.

> If you want to try it, the radio button still defaults to 'Subject', so you need to click 'Body' to search the body of a message. Right now it searches all the groups in the 'section' you are in.

This has been fixed. Now it defaults to Body, unless you've clicked on a name link, then it defaults to Poster. It still searches the section. I plan to add a link to search individual groups also.

Search is simple, but does support 'AND, OR, NOT and NEAR', which are case sensitive.

Retro Guy
--
Posted on Rocksolid Light
news.novabbs.org

Retro Guy wrote:

> to Poster. It still searches the section. I plan to add a link to search individual groups also.

I've set it up now so that if you click search in a specific group, body only searches that group. If you click search in the group list, body searches all groups in that section.

Subject, Poster and Message-ID always search the entire site.

Retro Guy
--
Posted on Rocksolid Light
news.novabbs.org

Retro Guy wrote:

> Retro Guy wrote:

>> to Poster. It still searches the section. I plan to add a link to search individual groups also.

> I've set it up now so that if you click search in a specific group, body only searches that group. If you click search in the group list, body searches all groups in that section.

This works fine, but was still sorting by group first, then rank. I've fixed it now so the entire output is sorted by rank regardless of group. Seems to produce a much nicer result.

Retro Guy
--
Posted on Rocksolid Light
news.novabbs.org

Retro Guy wrote:

> Retro Guy wrote:

>> Retro Guy wrote:

>>> to Poster. It still searches the section. I plan to add a link to search individual groups also.

>> I've set it up now so that if you click search in a specific group, body only searches that group. If you click search in the group list, body searches all groups in that section.

> This works fine, but was still sorting by group first, then rank. I've fixed it now so the entire output is sorted by rank regardless of group. Seems to produce a much nicer result.

Added body snippets to all search results, not just body (Subject, Poster and Message-ID), and sort the latter by date (newest first).

Now when you click search in a section, it searches the section. Click search in a group and it searches just the group. Message-ID always searches the entire section and eventually will search entire site.

Retro Guy
--
Posted on Rocksolid Light
www.rocksolidbbs.com