Post #472775

2026-03-01 06:00 UTC

I'm experimenting with making a toy search engine from scratch that'll be just for browsing the indie web and if there's any goobers who are ok with me testing it on their site please lmk (it shouldn't be too bad and only when I'm testing it but there's a chance that while I'm working it could do a lot of requests as it crawls since I haven't finished robots.txt stuff and I'm inpatient so it doesn't have intervals between requests) boosts for reach are appreciated

Replies (13)

@ivy_gay@mastodon.catgirl.cloud 2026-03-01 06:02

very happy to see it kinda work

Open ##2349948
@freya@chaosfem.tw 2026-03-01 06:03

@ivy_gay@mastodon.catgirl.cloud highenergymagic.net is ok with it

Open ##2349950
@freya@chaosfem.tw 2026-03-01 06:03

@ivy_gay@mastodon.catgirl.cloud not a lot there, mostly Solaris things

Open ##2349952
@lina@tech.lgbt 2026-03-01 14:49

@ivy_gay@mastodon.catgirl.cloud my website [natri.fyi] could be an interesting challenge (the html is basically wrappers around svg images but there's a pdf for every page (of which there are two currently, lol))

Open ##2349961
@arichtman@eigenmagic.net 2026-03-01 14:52

@ivy_gay@mastodon.catgirl.cloud idk if it's indie web but knock yourself out on www.richtman.au - a unique but consistent user-agent would help me pull out any metrics and lmk if you get stuck on iocaine

Open ##2349970
@h4kor@chaos.social 2026-03-01 14:55

@ivy_gay@mastodon.catgirl.cloud feel free to test on https://blog.libove.org I've experimented with my own indexing in the past and I wasn't able to crash it myself :D

Open ##2349977
@ganyuss@fedi.ganyuss.com 2026-03-01 14:55

@ivy_gay@mastodon.catgirl.cloud you can definitely put https://ganyuss.com/constellation/ in your tests! I don't know how search engine friendly it is though :neocat_think:

Open ##2349979
@solonovamax@tech.lgbt 2026-03-01 14:59

@ivy_gay@mastodon.catgirl.cloud sure you can use my site

Open ##2349985
@littlefox@gotosocial-dev.svc.0x0a.network 2026-03-01 15:04

@ivy_gay@mastodon.catgirl.cloud lf-net.org and lf-os.org are fine with it x3

Open ##2349993
@technomancy@hey.hagelb.org 2026-03-01 15:36

@ivy_gay@mastodon.catgirl.cloud I did this last year, it's very fun https://search.technomancy.us/why

Open ##2349995
@tom@epsom.social 2026-03-01 16:05

@ivy_gay@mastodon.catgirl.cloud Feel free to scrape https://people.cobaltqu.be :)

Open ##2349999
@subtlepseudonym@mastodon.sdf.org 2026-03-01 19:47

@ivy_gay@mastodon.catgirl.cloud feel free to crawl https://subtlepseudonym.com. It's a static site but there are a number of large-ish images that might be an interesting edge case for your engine

Open ##2350001
@elithebearded@fed.qaz.red 2026-03-01 20:13

@ivy_gay@mastodon.catgirl.cloud You can add my site if you like. I have an uncommon but standards compliant sitemap to catch bad bots. Might be a good test case if you are going to implement sitemap reading. https://qaz.wtf/

Open ##2350003