onionoo: historic details.json data
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Hi Karsten, can one download historic onionoo documents (details.json) archived somewhere or would one have to setup onionoo + feed old data into it to achieve that? thanks, Nusenu -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJVIaqIAAoJEFv7XvVCELh0OQgP/3FiBM3uMifPW9A0u8sMuh1E 7iFDmU52MNmkeToBUnuxuUK+7ig96tz/p5Ds6OyaAtC0Ld/IzRlIU5ELjqDwRqMD LPaafWm21QHO8y4fgW5FfGDfFTeLL/b3Lm1E9VmJ8wpOsgpzNnBMUoeDrpDyE3Ow zHXM/IfCSROqmO9lOeU6YGQTwInucc0HSveAWQ9i/x6iks9kMguzGnuWTxgfwA61 RQT4KUKZI+b5w//8V3/IpR5jlQ2N4OJrURpVhaKKNYMy+mqOfl5/a1ZPyQhhDaxj eLfg0qQmsk9IJY/6dZa0XogsmAzRCcEO/Ha0b0ZoiWs+VFo97cczoWdVWHoKpsyS X/qeehkOS13l22mw8va+yuj4zoRBrgK2chzcG2iPgVBeTu1H7B+x57PQrQXaCdaf rq8wwlS7mgaMQ6mgHyOQ1niZDTnvJaY0e3gVsgucR7YOTdALxnnHhQkyNboCWq/E 180SetWZfIHbL3cV38DRxYKzijwSu1gqmkwTJWffylb5SMja1TdW9cCq/RzidqyF v7vLXHQwmrRQ0XOy9UULe5XtU0Fe4YCm/zMNzBHHGIrePd664mYKqqXP2dlempwb BbbVgMLuFC2sy2eqxtwyQrrDUoWrEovUxg/BeoKO8RYXzNhPtha8eQkAhexhNux2 2mosesv/fooaQnt17qAo =q6LM -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Nusenu, On 05/04/15 23:35, Nusenu wrote:
can one download historic onionoo documents (details.json) archived somewhere or would one have to setup onionoo + feed old data into it to achieve that?
There are no archives of Onionoo's documents. But of course there are CollecTor's archives which you'd feed into Onionoo. Though it's very likely easier to parse those directly (possibly using Stem) rather than setting up an Onionoo instance for the exact time you're interested in. Speaking of, what historic data are you looking for? Maybe it's something that we should add to Onionoo itself? All the best, Karsten -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJVI9S5AAoJEJD5dJfVqbCruZMH/36cTEfO1v0+coAV2djdmsph qKnhnJ8nU7sTpDgMXqVsgKeZx/ErmYixCDUZsFJz2nrQmgmEuKkiaci4WG8yFhCX uUOLBtnwj4D4P0ge8+EFO7nh5QTskbKPyBbt0CIEEtrxgN3Xs3EvtTeFxSVg9i26 ngboHKf2x0h8UHLczstUfUlL1+KEnOYA8kfO8vDn/1c0Tda6TA5GDz1Y34o2eb3M IutSRva90xYNUrUqcEaoedcwutQPBnDOlkzipCMg5GkyqQ9Ak3CM49ZIRfyzYmzZ 5nl14zYTQg1QLoQkI1gOcrhhAksks7ebYE+GLp2bVjuQ4MgABVIoJJnAC1ddHi4= =cJw6 -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512
On 05/04/15 23:35, Nusenu wrote:
can one download historic onionoo documents (details.json) archived somewhere or would one have to setup onionoo + feed old data into it to achieve that? There are no archives of Onionoo's documents.
But of course there are CollecTor's archives which you'd feed into Onionoo. Though it's very likely easier to parse those directly (possibly using Stem) rather than setting up an Onionoo instance for the exact time you're interested in.
Speaking of, what historic data are you looking for? Maybe it's something that we should add to Onionoo itself?
I've a few use case for onionoo data, one of them uses onionoo to find groups of relays run by one entity. First_seen combined with last_restarted has proven to be a rather good datapoint for that. Using an array of all restarts (instead of just one) would likely reduce false-positives even more. That is one field but I'll consider (historic) changes to other fields (contactinfo, orport, dirport, ...) as well. Although I could imagine atlas displaying data like "these were previously provided contact details: ..." most of my use cases are to uncommon to add to onionoo. -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJVI/odAAoJEFv7XvVCELh0ztoQAKbcOuA0A0/oPlt51sSMCNqU sWFCPadSOOwpKFW0s4r0r50DRiYioJ4qSTsHeOm9kLFlywoMu4mOhePEGNIDtpCK NH8KYAqWL+1eqHo1fzSNmDxqweORwiyVV4k6FJF6YPi3Lk0pHM8Dy1zllv8CAs2l 0SNaauAPEz1d7PxBBu+9X8VI7i0ChR2zvt/1za3EoEIt5yK0gu+zvyeEOcTRZR1W gwQZ8UnoDrhVYeMtei42ZZtDchOqOHVrq+0BfdWPYnOb46ma1zK535o24lW0L2Ms UwTeFtMQTc1jhM4b1GpzBlcitQCwwwgFCkHC4ZG+jAujCP/cfX/Jp+YnHH9NLe09 N/9YmtxJr2LMtUmGHF9w0TPwlSiT8syJaPPRBpHifpOkNJwP5Mbq3fjCz8No8HDz zNtGetwtsqJTTzRa3Dyd8HBYBXaZc/Ok4I3wQTw6gVF9eKKT72eCvjEmNRGigNt0 VX4hLDrIHulY5U3+9ZOJhrB1ckqfrZOnElp0Ls3xC1LFXp6AvlfulM5MsRT2uad5 3Y1xNMY+7V0o3bQaukof/bg7NcivYteGWASnHR8P6uZQJZ5VBK2jW/uftu3oVlbA JRjHeIxO3BLpNKbCnQSxZU3Y/x5sIXBHYpPokN4y/ofgBt0/U1SSf96aRLgVycfj fNwlLSmuLs09fu2N2/GR =Emnw -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 07/04/15 17:39, Nusenu wrote:
On 05/04/15 23:35, Nusenu wrote:
can one download historic onionoo documents (details.json) archived somewhere or would one have to setup onionoo + feed old data into it to achieve that? There are no archives of Onionoo's documents.
But of course there are CollecTor's archives which you'd feed into Onionoo. Though it's very likely easier to parse those directly (possibly using Stem) rather than setting up an Onionoo instance for the exact time you're interested in.
Speaking of, what historic data are you looking for? Maybe it's something that we should add to Onionoo itself?
I've a few use case for onionoo data, one of them uses onionoo to find groups of relays run by one entity. First_seen combined with last_restarted has proven to be a rather good datapoint for that. Using an array of all restarts (instead of just one) would likely reduce false-positives even more.
So, the problem here is that an array of restarts doesn't scale, so it would have to be limited to the last 10 restarts or so. But even that is not trivial to implement in Onionoo and, as you note below, it's quite specific and not very useful for the average Onionoo client. What you could try is evaluate uptime documents and see if two relays had similar uptime patterns over time: https://onionoo.torproject.org/protocol.html#uptime
That is one field but I'll consider (historic) changes to other fields (contactinfo, orport, dirport, ...) as well.
Although I could imagine atlas displaying data like "these were previously provided contact details: ..." most of my use cases are to uncommon to add to onionoo.
I think I agree with you here. Also, you're much more flexible by using descriptors directly and adapting what you're extracting from them rather than having to wait for me to add a new field to an Onionoo document. By the way, are you aware of Philipp Winter's work on a better Sybil attack detector? He's cc'ed in case you want to brainstorm about good criteria for comparing relays. All the best, Karsten -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJVK3uSAAoJEJD5dJfVqbCr3uoH/2WyEtrm6af/1LTZ2cBW9CgD wWoDpo17zRXse4A17IRYaD2P+8yaKHRS2KDnfewCmNoW52JHUWEvzkODMSEmLjTf l6sEWo3gvYcmp6TGVMpcjTtYcXlj093MzD749p3dDglkB/HkJjamO1bKfzzbRw51 Z00s7aN1bCbdjTYenreJreAHUeUNGWD+c7y8VG4tKU12TbZq8AZD1x5B3jPfg1n7 cQEvSvpr4Y7SHgPkndwtwliBsg51B+oALQk2wEmh1neVDEBG8Rb8i4hLjZB84DsD eMCtteEUVm0TinIoKNO3uGv+hk1Ls5MQF906JeB3ctrwPIbZcPRCwolGLUgmXkA= =xUvB -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512
By the way, are you aware of Philipp Winter's work on a better Sybil attack detector?
If you mean http://notebooks.nymity.ch/detecting_sybils.html then yes, I've seen it. -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJVLrcMAAoJEFv7XvVCELh0Q5UQAIV4qIEADzwVNexSCs5JRun5 swV1EKq+OZtQ4BWf2kxGN388zcZvgVGsjSuj4sBYcTFlzgJ5Rx4A6kqMZ+XGA0x5 7FJ7x61rQ/lrvz9HxrFGnKjU28YNhxAkaumrK2690/+ecpk3Wdo0J+sx3nZLRurc 5CpyGmnWv2ArJhXfGbZ3cF5wvU9mYV1hAx4aoXLCbDVOk274zkyb/XA1gTUrJX2d 5BEzbsfDBbiQRIhCzoZNUIX7krZEp/Wi05MqdOoDXmuHRJ6LoGPvp3NPQnoEizaN zUDp/pNaNv1zV8ygTbAueCBLafbYPZc6xfwreI/bAjfLeJST38ABGTwtydsW3VPd UGEcuct8F8G2xq30fGDu6xCgLUhhN9tef41fpiDRRchnCoiDCZwts0rgPywlZyEp 2sWmlkOy/Dxq0GPi2owBIg9wwT8FYb4OwOe+6QiLsXkx1Xb9Adf75pPateNJ5XCB MEgVYl97mxyXonptbjVm3ToEEgljcROcz+7hhX/dD0mEinacB3t092nYHcfLFeq2 RHKEr1bpyjhX8dqjnXK4LrErhQHZlhEiQc+3PglPqq5Z9kLGOcNQowogB5KMVdj0 duBCyYtEDhUlVMCfzP5COLFkJuNfSThc1v+YO0rsnBfMjMX7RmPyCdvfODzbLCtM +yttIXDy9dB4GXHUFYdP =qb7i -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Hi Karsten,
Though it's very likely easier to parse those directly (possibly using Stem) rather than setting up an Onionoo instance for the exact time you're interested in.
can you say something about what amount of minimal memory and disk space one would probably need for a non-public onionoo instance? thanks, nusenu -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJVNq3SAAoJEFv7XvVCELh02xIP/14QAycUAcAhun36WyzhpsC/ gb8Ta0ryCzEBW3zT4//9pyrz3VO08ryLI6Kg/OgrwOSUfVxQtTkhrnYG9GZlr89Z fPW3PgnilPFAikaH4iJzxbvVS8FrZdG3d93QM6uDsMHOfwuZZJg+PXSTfIdAawYc /eS59HfeXDEz/fCQAl+N9YWKJuKQbPqmG3ah9r27ppJRVCp6Upgm1i40s5z2emsC Wg54HBy3n5To04t8mzaW1eJKMlXFXsps6ywxERuVwyfOBPBBL3WViybJLN0IOW59 1k9qJB9UfvaxHVd52+YJw6eNaLBgVOqllz1Dd4tBctNzgUfJKCBKLsmxovQj9Daw GsA+navUOFHHJwXCs9b2yMZa+LcnbYbP3Y2HT0ZwY5al2hy4SsqA7bAyIk7RozEJ 6p8S/Mr7qh7QS0Gw7Dz4AGJ1fVgsR7FBLj2g0XO27SlqhNYC+Mf75eSkF0MQJ6vj YQCJhSZIn4DAs+AG2Ul42aY3/w0LTQfLmjbAfGrvuO1wrupDQ+6D1nvFc9AYVLHR IWhco4XdQiVOLnmDAs8bMY/llFS+3x1Tf7gVKV0rP17w6NTLsgLXq9B2ufkBYwaR nLWOGIQ3AeoBapONFmFe1Klv7/69mu0NiWsQ8Nx1JIc2Ogt+VzRrWC1jyBcbI8hC yXqlQeW9FNizS9b3SZUo =Hn+K -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 21/04/15 22:06, nusenu wrote:
Hi Karsten,
Though it's very likely easier to parse those directly (possibly using Stem) rather than setting up an Onionoo instance for the exact time you're interested in.
can you say something about what amount of minimal memory and disk space one would probably need for a non-public onionoo instance?
I'd say 8G RAM and 100G disk space could work, though 16G RAM and 250G disk would save you some trouble during the initialization phase when you feed tons of descriptors into it. If you want to give this a try, I'd want to take that opportunity and improve the documentation and maybe also the process for setting up Onionoo, if you're interested in helping with that. All the best, Karsten -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJVNrAnAAoJEJD5dJfVqbCr9ZgIAL3gWt9AJc0YMD4658dtXaPw hKiHEU7ghZl5nzzKBroamDYw2yb1d8J4sT2IIdmn0uY+2Vg/tAypT5Fw9NImbuuW D63DoHoTxpE4EiNQbBlWK8ftpZQc+sOPJBNS46bDTKPWJyt3w1N0tT0ixC3TuyFO XC3jzcgghrHPD/EwxrxUFMWOY3CS09VlyETIFEtsfqapPBFBZKt+Ige9iEnQyrrc gFEFbrNxiPwxd9u3yHhwyP56cjoilbrd7OXpy9G4Nl8kHA3cJqTFWMvKhOWvgIKz hxlrJP1haTO8mPMklio0/n1DLGqgFIvqsm+zRTxZxDt6kKgSZmwaES7LjYa8y54= =6OkR -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Improved documentation and process of setting up Onionoo would be welcomed by more people, including myself. Busy setting up Compass and with Atlas and Globe mirrors active the cherry on the pie would be an own Onionoo instance (if needed also as backup for onionoo.torproject.org). At the moment having onionoo running at thecthulhu.com is a nice backup, I do not know if any more of such mirrors are acceptable or preferable. On 21-4-2015 22:16, Karsten Loesing wrote:
On 21/04/15 22:06, nusenu wrote:
Hi Karsten,
Though it's very likely easier to parse those directly (possibly using Stem) rather than setting up an Onionoo instance for the exact time you're interested in.
can you say something about what amount of minimal memory and disk space one would probably need for a non-public onionoo instance?
I'd say 8G RAM and 100G disk space could work, though 16G RAM and 250G disk would save you some trouble during the initialization phase when you feed tons of descriptors into it.
If you want to give this a try, I'd want to take that opportunity and improve the documentation and maybe also the process for setting up Onionoo, if you're interested in helping with that.
All the best, Karsten _______________________________________________ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
- -- Tim Semeijn Babylon Network pgp 0x5B8A4DDF -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJVNrq6AAoJEIZioqpbik3fR2sP/1njsgQulsnY2OkgXRjTWkkv 5t0gp30bPm8Bgwl4CdxO8NWzXyQt/zeWulf59j1ccvtds0EmclMvQ0iqUsP4RMhC tfawZY0tXWf0nm79sQCgQTZ3SG1FJ6SqoWUSVbQ/GRQlAOIA7XK6C1VJA72LXrp5 wTzo4z9Rv9qWGgZjzNrsnvG7iYZbz4muLIDbbwd5brUWC0UfYXlQXA0UbbyFPiBO J6/5iZPsjInwsIscS0tr38g7LxydNFpR24IcodAzPhRcQmMdnjEfuD1I9H/SZhC0 3d+yYo5tIQx33K+g7NrcIm4WWQjye25dgm7W/DsiiJFCKvYgCAppoT46Y3RU2o/p 8ot06zx4TaMhMcNP3GQ/BX61bOB/vTxBeSptXz/NMBbu8UE1UtxYMQn2/shcvLQD SURtx9J+5m9aeXA88jSqm35GzXPhbmvtf5q8OkJD4bSu0vZTb4obhr4LS3JG8C7P TpHNHsZu+Bx473nKCdgVo59h3Q7i2fM53M6TKW/sDgBaFQTGeSjY7pVWNdfMpyBj ZAEA5fnlRVC6HXPIaemuB/zLXptuAlWiKfx4jpmR6h3t8yzx5nUcTc7mGdH6oNfl 2AIMpdca7vZwxMwaO1kW57kqGOHbcG7LiiHxgoRn52mIEWl6YFIpVSf3Xlk3JaoU DzfQN2rLUI/2lNq+kW1R =ZUk9 -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello Tim, On 21/04/15 23:01, Tim Semeijn wrote:
Improved documentation and process of setting up Onionoo would be welcomed by more people, including myself. Busy setting up Compass and with Atlas and Globe mirrors active the cherry on the pie would be an own Onionoo instance (if needed also as backup for onionoo.torproject.org).
At the moment having onionoo running at thecthulhu.com is a nice backup, I do not know if any more of such mirrors are acceptable or preferable.
Having another mirror or two sure would not hurt. I know that iwakeh was trying to set up a mirror, but I'm not sure whether they succeeded. I have also been thinking about adding a feature to Onionoo where we configure a list of fallback mirrors that are returned in case of a server problem. This would become more relevant if we actually had such fallback mirrors in place. So, yes, if you'd like to set up another Onionoo mirror, that would be really cool! I wrote down some instructions based on Onionoo's INSTALL file but in more detail. It's supposed to become the new INSTALL file. Would you want to refine these instructions (or add more questions) while trying to get an Onionoo mirror running? https://pad.riseup.net/p/YG9eWPopYDJM (If other people on this list would want to help make these instructions better, please feel free.) Let me know if you're running into any problems. All the best, Karsten -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJVO+0AAAoJEJD5dJfVqbCrFMQIAMJ51C6zwLlqpae/RxCg5xU7 TRcnvDaSvRfLjwHWy+DdOrvA9CP56qJMq69k6+uTgTkHovdPHrXN2LIX4Kd6YfAw CFPaF4nFuzJf9JTii0rJrp8RilbCy5grfc+F2vqZkHKjgYvPPtxbRnYzOAr5uvLd IpomHml3Dn9KMsz7HTXxZHCIhXeSrQ8nNAaxZVCGa4D1vYLevkoBzx5FXAdrWlTA s2w/gMHtE0xgzLwXjPeHtMFyZZS9TAsVjgUdynQPFr7pw887hPso+pmqkALdZHtn AWEwFR9n64dhSiyYJgZgr77zGNy83ojvsbAdAFQORei/sUryVFFCWU2CRjdOnGc= =+61A -----END PGP SIGNATURE-----
Hi, Actually I've been meaning to ask a question related to this. I've been wondering if, during the development of Onionoo, you considered any other frameworks? I'm not familiar with the history of Onionoo so I don't know if you made the choice based on some constraint. I read the design doc which made me curious. --leeroy
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 25/04/15 22:33, l.m wrote:
Actually I've been meaning to ask a question related to this. I've been wondering if, during the development of Onionoo, you considered any other frameworks? I'm not familiar with the history of Onionoo so I don't know if you made the choice based on some constraint. I read the design doc which made me curious.
Most design choices were made in favor of making the web front-end part scale. It's a response to building some services like the (discontinued) relay search that started with reasonable performance and degraded a lot over time. It's why, in Onionoo, responses are written to disk by the hourly updater and not put together on-the-fly. And it's why all requests are handled by an in-memory index of all documents rather than by a database. I'm not saying that no other design can achieve the same performance, but I find that much harder, in particular with respect to performance variance. Not sure what frameworks you have in mind. But I'm happy to hear more about frameworks that would make Onionoo easier to extend and not perform worse (or even better) than now. If you have something in mind, please say so. All the best, Karsten -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJVPnRoAAoJEJD5dJfVqbCrPCkH/2u6wtTLcX0HBnVRHjrFlzop x2nOwwBxrfszRJZDT0/eBJFwcQUKQAXHPakwWB0YBWPuKjFLYgTos618Sv2eCCIW MmeTvO93xncAxV7eOZnkF/4XlTG9UD9w/LPijM2vfp6RuMZe/V69ZaoMuhom7pVq bnZkMki2D+XwjyGwcNJTYKSE7WlGbcPm4ZVEwQNMH/OMUZisfRuvKTxhej+l5K37 pQ3DVw/c277hmrRiKAvE0LFFCY5YcGSvze679AuMIZDZDW/YboNvb/jDL7sVzuYv cV99U3F/kyMmn2L2FXD4vCkb4M/lyPHgTxA6kwvJHu4DI5TIgATvwGpLD+TFE9E= =iBpu -----END PGP SIGNATURE-----
Hi Karsten,
Not sure what frameworks you have in mind. But I'm happy to hear more about frameworks that would make Onionoo easier to extend and not perform worse (or even better) than now. If you have something in mind, please say so.
Thanks for the clarification. I'm not against the choice of Java, nor claiming better choices. I have fond memories of Java. In particular I've been working a lot with Django recently. I didn't want to redo works that may have already been performed. I was thinking of some recreational uses of a server. I started looking at the onionoo documentation and my curiosity was piqued. Precisely because the first thing I thought of was reusing a cloned server for, well, a onionoo-clone. The JSON formatted files could be used as fixtures for setup. The two apps could be run separately as you've already mentioned. The other development specifics are: nginx-gunicorn(greenlets/aiohttp) postgresql-pgbouncer Is it an experiment worth pursuing? Your thoughts are appreciated. Thanks in advance. --leeroy
Hi, Django (and by implication, python) are an accepted technology at tor, but as much as I wish it would be different, the tor web infrastructure is still based on python 2.7 (basically, you can only depend on whatever is in wheezy and wheezy-backports if you want something to run on tor's infrastructure). Of course if you don't intend for your project to ever replace tor's own onionoo deployment, that doesn't matter. Best, Luke PS: I'm also going to take this opportunity to plug my onionoo client library that you can use to check that your onionoo clone performs to spec ;-) https://github.com/duk3luk3/onion-py
Hi Luke,
Django (and by implication, python) are an accepted technology at tor, but as much as I wish it would be different, the tor web infrastructure is still based on python 2.7 (basically, you can only depend on whatever is in wheezy and wheezy-backports if you want something to run on tor's infrastructure). Of course if you don't intend for your project to ever replace tor's own onionoo deployment, that doesn't matter.
Thanks for pointing that out. I think that won't be a big problem as this isn't intended to be a replacement. If it ends up being a fruitful experiment then it's a success. It's a success if it demonstrates some improvement over the currently deployed design. If I stay away from python3 then the main difference is the use of postgresql+pgbouncer/pgpool. My instincts are telling me that python3 is needed for aiohttp to demonstrate that asynchronous io, lightweight concurrency, and various database optimizations can yield improvements. If the results end up meriting reproduction then virtual environments can be used for testing without breaking existing infrastructure.
PS: I'm also going to take this opportunity to plug my onionoo client library that you can use to check that your onionoo clone performs to spec ;-) https://github.com/duk3luk3/onion-py
I saw that. I'll definitely keep it in mind for comparison. Thanks again. --leeroy
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 27/04/15 23:39, l.m wrote:
Hi Karsten,
Not sure what frameworks you have in mind. But I'm happy to hear more about frameworks that would make Onionoo easier to extend and not perform worse (or even better) than now. If you have something in mind, please say so.
Thanks for the clarification. I'm not against the choice of Java, nor claiming better choices. I have fond memories of Java. In particular I've been working a lot with Django recently. I didn't want to redo works that may have already been performed. I was thinking of some recreational uses of a server. I started looking at the onionoo documentation and my curiosity was piqued. Precisely because the first thing I thought of was reusing a cloned server for, well, a onionoo-clone.
The JSON formatted files could be used as fixtures for setup. The two apps could be run separately as you've already mentioned.
The other development specifics are: nginx-gunicorn(greenlets/aiohttp) postgresql-pgbouncer
Is it an experiment worth pursuing? Your thoughts are appreciated. Thanks in advance.
Regarding Python, we already tried rewriting Onionoo in Python a few years back and failed. It's a larger project than it seems, and the possible benefits probably won't justify that. (Just think of the many new features we could write while rewriting existing ones.) Using Java for the back-end and Python for the front-end is a bit ugly, but could work if there's a true benefit in that. Though we might be able to re-use the concepts from the Python experiment and incorporate them in the current Java implementation. I very much doubt that performance advantages would be attributed to Python vs. Java, but here I am starting to argue about programming languages, which I shouldn't. I think the best way to improve things is to look into switching to a SQL database. I already started experimenting with that in the past two weeks and just wrote down my findings here: https://trac.torproject.org/projects/tor/ticket/15844 If you're interested in some database performance hacking, you'll love this ticket! Much appreciated! All the best, Karsten -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJVP37JAAoJEJD5dJfVqbCrE9oH/207mKD4Hhc1dcBpWvNP+oYR p4u1Xgx4j6804Hhl26kiPXm4K43FDMG0vZ/YmIxJKGggPvdP8ZesiunH2bGtYISr FeHu2+72H4d8p1WadFN5UBWENrYX49qCcuUBSzquRtGK1pnlbTlbOiLGc2iSdSZT cLoTZcAtyIEms8+lAswNmrZR6E3FMQqvmhtGSWeL7KihoNcKqsH0y5Dnogkj/Vyb Lez5eCDfJy4sV2eJtfPqySzMt67wbBCJ4PdpXsXX3iTx1H0clUwQbzKwiEv9PpFT IQpyWIKA9ErW0hMUiz192sB6dGYRSySfZHG/z9Jrs9M6t6UmhJbA1jHvvvabs/o= =IeUy -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hey Karsten, Good to hear more onionoo mirrors are welcome. The instructions are nicely detailed so I will be able to try set up onionoo most likely next weekend. I will provide you with feedback on the instructions. A feature with a list of fallback mirrors available for the main onionoo instance sounds like something awesome but of course multiple mirrors have to be available. Best regards, On 25-4-2015 21:37, Karsten Loesing wrote:
Hello Tim,
On 21/04/15 23:01, Tim Semeijn wrote:
Improved documentation and process of setting up Onionoo would be welcomed by more people, including myself. Busy setting up Compass and with Atlas and Globe mirrors active the cherry on the pie would be an own Onionoo instance (if needed also as backup for onionoo.torproject.org).
At the moment having onionoo running at thecthulhu.com is a nice backup, I do not know if any more of such mirrors are acceptable or preferable.
Having another mirror or two sure would not hurt. I know that iwakeh was trying to set up a mirror, but I'm not sure whether they succeeded. I have also been thinking about adding a feature to Onionoo where we configure a list of fallback mirrors that are returned in case of a server problem. This would become more relevant if we actually had such fallback mirrors in place.
So, yes, if you'd like to set up another Onionoo mirror, that would be really cool!
I wrote down some instructions based on Onionoo's INSTALL file but in more detail. It's supposed to become the new INSTALL file. Would you want to refine these instructions (or add more questions) while trying to get an Onionoo mirror running?
https://pad.riseup.net/p/YG9eWPopYDJM
(If other people on this list would want to help make these instructions better, please feel free.)
Let me know if you're running into any problems.
All the best, Karsten _______________________________________________ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
- -- Tim Semeijn Babylon Network pgp 0x5B8A4DDF -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJVPqRvAAoJEIZioqpbik3f8qIP/iIDCzVVhCeWffFNb+zdP4Nn WjInwouDJjuE5urgthG6z/VTpHmQJNh0jtp9C20FXSPbKHYPhHZ89w76FdFTJQXx 0vBc1RQyoYflMp6v4oHrhVPmq+PvP2QApEVU2Z9qUNMnB+ahQRTtDb/uLnB8LNwm 4Xrrgwl2h+TNQ1EbYsiaa2UjhKSl1so40KthhREqk/m+bWrooIB6Pe+Zhx1W1lK5 aRP6e6ikV0IkWIV/Ee6fq4uxgOj4bNG5bh/+tWif7EBPQbDzgHeTG4lKgzQhAFza mJ5iZJ4LoUe9ZxhLOAw7NSLq5JgbckhJyY9twfsGyvuifagxSHUVuana09Wo5DjL MyVUa5BmiGqA0mQBGmstJ1JY8IZt0NKkpBle5X40YwFWI1/LwkdudE40vjNSjoZo +0XcCHiCz79sSswRrA8z0C88QNwWBJ/LSmngYL67bpXjMR6oBDRpxifXlM/xuUrP y0z55AjH54fAcDXF8xxWZcAnJiD73rOjU2a2SeMq2s+uwro+yqxN0IUqupw1ZzKi ueH4P1fUvJuuqFZM/knAQrVHtg7YiK0iKUdCp+KXgo9EZ5EdEoy3Cv6SFqnODMxl S2eBylNgFV5ZfqEfR+xboQPhMNLQAewi7X4zvYg2Jag1jydxxNHrc0MuVvfZ0WP2 2R6Cml++XZHrkqqYZqRP =Z15e -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 27/04/15 23:04, Tim Semeijn wrote:
Hey Karsten,
Good to hear more onionoo mirrors are welcome. The instructions are nicely detailed so I will be able to try set up onionoo most likely next weekend. I will provide you with feedback on the instructions.
Great! Let me know how that goes.
A feature with a list of fallback mirrors available for the main onionoo instance sounds like something awesome but of course multiple mirrors have to be available.
Well, we already have two mirrors, and usually one of them works. I just created a ticket for this: https://trac.torproject.org/projects/tor/ticket/15843 All the best, Karsten
Best regards,
On 25-4-2015 21:37, Karsten Loesing wrote:
Hello Tim,
On 21/04/15 23:01, Tim Semeijn wrote:
Improved documentation and process of setting up Onionoo would be welcomed by more people, including myself. Busy setting up Compass and with Atlas and Globe mirrors active the cherry on the pie would be an own Onionoo instance (if needed also as backup for onionoo.torproject.org).
At the moment having onionoo running at thecthulhu.com is a nice backup, I do not know if any more of such mirrors are acceptable or preferable.
Having another mirror or two sure would not hurt. I know that iwakeh was trying to set up a mirror, but I'm not sure whether they succeeded. I have also been thinking about adding a feature to Onionoo where we configure a list of fallback mirrors that are returned in case of a server problem. This would become more relevant if we actually had such fallback mirrors in place.
So, yes, if you'd like to set up another Onionoo mirror, that would be really cool!
I wrote down some instructions based on Onionoo's INSTALL file but in more detail. It's supposed to become the new INSTALL file. Would you want to refine these instructions (or add more questions) while trying to get an Onionoo mirror running?
(If other people on this list would want to help make these instructions better, please feel free.)
Let me know if you're running into any problems.
All the best, Karsten _______________________________________________ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
_______________________________________________ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJVP2RcAAoJEJD5dJfVqbCrNHoIAKVyuby7BawXti9LVhnkUAau uXr1OcchvLoaLdN0zF7dK+kTJyu8mK2+L8IGCingxw3N3bl4h2Gr1SWkKnXASEJo XplFufV/ahdR57L0rvU4hpTD+Xqj7+464WPG3CaNPizWvfRgTgK9eFMI4m0kZb8O OYsZzQP2MLpvSJKZ2JLNOY9oezrzh3o/IefjWAnx4v2jL7SyG7OwfoOLF7tOIpTl +GHvC4NIy6WVnyYWp9hxgM7oGHBVpq4zBT/08qVOg1VOCVOo6Ui9HgwawuY34o7h cYnCC9vYgPrnCrNoDnLp3bqd1sLg59rrf/Zb95j6gtjFEZZ7ctwbzsOcsE6mkHU= =Daju -----END PGP SIGNATURE-----
participants (6)
-
Karsten Loesing -
l.m -
Lukas Erlacher -
Nusenu -
nusenu
-
Tim Semeijn