What’s Bing doing above Wikipedia?
Default search engine on Microsoft Edge browser/Windows search
Wikipedia going down like that makes me sad, especially since due to ai crawlers, their traffic costs went up significantly.
Why would anyone crawl Wikipedia when you can freely download the complete databases in one go, likely served on a CDN…
But sure, crawlers, go ahead and spend a week doing the same thing in a much more expensive, disruptive and error-prone way…
There are valid reasons for not wanting the whole database e.g. storage constraints, compatibility with ETL pipelines, and incorporating article updates.
What bothers me is that they – apparently – crawl instead of just… using the API, like:
I’m guessing they just crawl the whole web and don’t bother to add a special case to turn Wikipedia URLs into their API versions.
valid reasons for not wanting the whole database e.g. storage constraints
If you’re training AI models, surely you have a couple TB to spare. It’s not like Wikipedia takes up petabytes or anything.
Vibe coding
My comment was based on a podcast I listened to (Tech won’t save us, I think?). My guess is they also wanna crawl all the edits, discussion etc. which is usually not included in the complete dumps.
Dumps with complete page edit history can be downloaded too, as far as I can see, so no need to crawl that.
Good pod cast
Yes they should really block crawlers or force them to pay. The only way I can think of that they could do is make you have to register an account to access content but that goes against what they originally intended. But these are new times and it’s probably for the best. Wouldn’t be hard to flag obvious AI scrappers.
It seems there are ways to stop crawlers. Do a web search for “stop ai crawlers” or similar to learn more. I hope it doesn’t escalate into an arms race, but I realise I might be disappointed.
And click through rate is dropping.
Can’t reddit just die or something?
Ironically it’s getting more popular but to me it seems it’s getting more popular in the way facebook used to get more popular. At some point your weird uncle is on it and all the good content creators just leave.
Also google makes it the first result for a lot of searches now.
Reddit stopped being fun last November.
I am not engaging with it as much as I used to, and now I am actively annoyed with it and trying to make Lemmy my default again.
What happened last November? I quit when they turned off the API
In fact, I think at least in Brazil Reddit is becoming more and more popular. Go back like 5 years ago and Brazilian subs in Portuguese were small and low-traffic except for one or two (ex: /r/brasil).
Now there are a bunch of different themes and I see new topics being discussed quite often.
I believe the reason is Facebook has enshittified so much that its communities are dying quickly and Brazilians are finding Reddit works better for simple discussions. Also no one posts anything personal on FB anymore, it’s all Instagram style reposts so it lost any purpose.
By whose measurements? [citation needed]
Top 10 websites?
Where’s porn?
In 3, 4 & 7 IG
Didn’t really support the porn industry since the beginning but I kinda miss the porn magazines and the website made for it.
It’s far more dangerous for multiple reasons to view porn on mainstream socials than intended websites, for young specifically… Sadly… One more thing that tend to worsen…
deleted by creator
