TTD reimagined research and ideas

[xavier242]

I'm stepping down from Primary Admin due to serious health issues.

U2Lynne may close TTD or not, and that possibly depends on someone here stepping up to the plate. I just checked and technically we had 2 Admins, me and U2Lynne. So she's still an Admin and might agree to let you reimagine the site without having to do all the tasks an Admin might do. I had a job Oct 2023 to July 2024 that took most of my time. I'm thankful we have a "Ex-Admin" who essentially kept the site running.

That means if you have the skills and time, you could build a new TTD with your vision. But before I step down, here's what I found and some ideas:

The current TTD dedicated server pre-dates modern remote admin boot features, so it has to go. We need the ability to reboot the box without having to rely on a help ticket.

Storage needs to be on a RAID1 SSD array (2 drives) using database SSDs, preferably high-endurance type. Tell Quickpacket you need the fast speed these offer. Standard nVME drives are for consumer use, not in a server that heavily uses a database.

If they want you to use software RAID rather than hardware, it has to be RAID1. It's the only RAID level that has sane data recovery (hardware or software RAID). If you get lucky and have hardware RAID, make sure it's not Adaptec (unreliable and can wipe out all drives).

RAM, I don't know. WhatCD said the Ocelot process used 3GB with 5 million peers, which is pretty damn efficient. TTD currently has 32GB. So if you go with Ocelot you may be able to reduce this. Most dedicated boxes have at least 8GB to 16GB now without paying extra. You may need to run a virtual machine for the old site which takes a fair amount of RAM. If you are a hypervisor wiz, that might be the way to go.

Yes please do keep the old TTD online, at least in a read-only mode. The historical value is immense. No where do I know of such a detailed archive of information on past gigs. The torrents will have to go to the new site, probably one by one manually (automation for this might require a team of developers). Reposts are not that big a deal. It's how all other sites handle expired torrents.

Ocelot claims to not need much CPU. The quad core 3.2GHz Xeon TTD has is plenty fast, so when deciding, this isn't critical.

Finding a hosting provider for any torrent site is very tough these days. Quickpacket has grandfathered TTD in, so you may have to work with them. They're a decent provider, but in my opinion their connectivity is poor. TTD often reports users being firewalled to the tracker. As I have posted before, this in theory only affects the peers list stats, not your ability to connect to a given peer who likely takes a different route to you than to the tracker (I know some here don't agree with this).

Frontend (aka the web site or forums): the code that greets the member and runs the forums, often in PHP.

Backend (aka the tracker): the code the handles the torrent announce and upload/download requests from your bittorrent program. PHP was fine 20 years ago. Today it's not fast enough. You want a compiled language with efficient code. C++ is often used, but any fast compiled language is good.

Chihaya and Ocelot trackers use fast compiled executables. WhatCD claims Ocelot handled 5 million peers with 30% CPU load from a single core (very efficient code). They said loads were often 12000 hits per second. TTD's PHP tracker dies at around 100 per second... PHP is not complied and far too slow for modern loads (for the tracker, it's fine for forums).

C++
https://github.com/whatcd/ocelot
GitHub - WhatCD/Ocelot
Ocelot is a BitTorrent tracker written in C++ for the Gazelle project. It supports requests over TCP and can only track IPv4 peers. It supports requests over TCP and can only track IPv4 peers.

PHP, ok because it's the frontend
https://github.com/WhatCD/Gazelle
Gazelle is a web framework geared towards private BitTorrent trackers. Although naturally focusing on music, it can be modified for most needs. Gazelle is written in PHP, JavaScript, and MySQL.

C++
https://github.com/OPSnet/Ocelot
GitHub - OPSnet/Ocelot
Ocelot is a BitTorrent tracker written in C++ for the Gazelle project. It supports requests over TCP and can only track IPv4 peers. It supports requests over TCP and can only track IPv4 peers

PHP, ok because it's the frontend
https://github.com/OPSnet/Gazelle
Gazelle is a web framework geared towards private BitTorrent trackers. Although naturally focusing on music, it can be modified for most needs. Gazelle is written in PHP, Twig and JavaScript. It traditionally uses Mysql and Sphinx, but work is under way to replace those with Postgresql.

C++
https://github.com/Mosasauroidea/Ocelot
Mosasauroidea/Ocelot is a BitTorrent tracker written in C++, forked from WhatCD/Ocelot. It supports requests over TCP and can track IPv4/IPv6 peers.
New Features:
Support ipv6 protocol
Discount on torrents, such as 100%, 50%, 30% free, which is supported by traditional tracker
Statistics of users' uploaded/downloaded volume size, last seed time, etc.
Fixed some issues

Golang
https://github.com/chihaya/chihaya
Chihaya is an open source BitTorrent tracker written in Go.

Chihaya is built for developers looking to integrate BitTorrent into a preexisting production environment. Chihaya's pluggable architecture and middleware framework offers a simple and flexible integration point that abstracts the
BitTorrent tracker protocols. The most common use case for Chihaya is enabling peer-to-peer cloud software deployments.

C++
https://xbtt.sourceforge.net/
XBTT about 1/8th as efficient compared to Ocelot (CPU usage).

C++ and PHP:
xbtitFM and Xenforo
In the Reddit thread, they claim it's ready to go (PM the poster). Note it has about 1/8th the max capacity of Ocelot.

All the other trackers I found are PHP. Don't even consider them as they don't have the capacity for modern loads (TTD uses a PHP tracker from 2004).

Any one willing to take on this task is going to have to spend time learning the systems and writing code to make it work. I would suggest creating an entirely new site on a newer server.

Once that is working, bring the old vBulletin site to the new box. Run it in a virtual machine (it should work in CentOS 7). Disable posting in the torrent forums. Or disable all posting and create new discussion forums on the new site. Plan carefully, and one server can handle the old TTD, new TTD + new tracker.

Hosting providers typically have many servers on one box via a hypervisor. In my experience you can run a hypervisor, but shared hosting isn't going to cut it as you may be rebooting a bit while you figure things out.

The new site should have a planned revenue model and incentives to encourage donations. Plan to have ads. See https://www.guitars101.com to see how to properly do ads without being obnoxious. Offer the ability to remove ads with a donation. Determine an algorithm and code that removes ads for long-term seeders.

Note that any revenue model may need adjustment depending on the country the member is in and various legislation that passes.

Study https://sharemania.us for ideas. I like how they've done fundraising and donation goals.

Note our TTD's current vBulletin is poor for having ads or donation modifications. They run code in variables loaded from the database. I can only guess it was to obfusticate the site code to deter piracy but it makes it difficult to add features or change code. The only TTD code that made sense to me was the tracker as that was an addon not created by vBulletin.

Thus before deciding on a frontend, make sure it's written in a manner to allow changes without too much fuss. FYI, every PHP forum I've used before (not vBulletin) could be easily modified.

Have a prototype box you can experiment with, and push working changes to your production server. I tried to do this with TTD, but without any proper admin tools and lack a access to files, that was just a dream. Root login has no access to vast portions of the site code, config files included.

If a brave soul decides to take this challenge on, try hard to get the old server from Quickpacket. As I was unable to make a full backup remotely, you may need the box physically in front of you to do so. It's software RAID 10 using mdadm.

Some relevant links:
https://www.reddit.com/r/trackers/co...t_10_released/
https://torrentfreak.com/what-cd-deb...-peers-101014/
https://www.reddit.com/r/trackers/co...etter_tracker/
https://www.reddit.com/r/trackers/co...rrent_tracker/

[mjcrossuk]

Yes please do keep the old TTD online, at least in a read-only mode. The historical value is immense. No where do I know of such a detailed archive of information on past gigs.

I would like to try to preserve the torrent details from TTD in a database, which would be put online.

I've been trying to scrape the pages in the Music Downloading section of http://www.thetradersden.org/forums/archive/, but I'm getting strange results, both when scraping (using the Python Requests package) but also when browsing the pages in Chrome & Firefox. Sometimes the torrent list contain links to the individual threads, other times there's just a list of torrents without links (which is no use for getting all the torrent details).

Is there a better way for me to get all the data? Could a database extract or dump be provided, in the event that TTD closes?

Why me? I'm a retired software developer with 45+ years of experience in mainframe, workstation, PC, client-server, database and web technologies. I'm also a music collector, & trader.

I created and maintain a database of the Bot List messages from DimeADozen. The database currently contains 1,552,963 messages, each message being a torrent, going back to May 2004 (when it was EZT). The database is online, and available for download by anyone, you don't have to be a Dime member. It uses SQLite so can be used by anyone on their own computer once they've downloaded the database and installed a SQLite browser app.

[mjcrossuk]

Originally Posted by mjcrossuk

The database currently contains 1,552,963 messages, each message being a torrent,

That bit's slightly wrong, it's only 774,809 torrents, the other messages are bans, deletions, etc.