[tor-bugs] #24175 [Metrics/Website]: Use an embedded Jetty in metrics-web and use metrics-base as build environment.

[Tor Bug Tracker & Wiki]

#24175: Use an embedded Jetty in metrics-web and use metrics-base as build
environment.
-----------------------------+------------------------------
 Reporter:  iwakeh           |          Owner:  iwakeh
     Type:  enhancement      |         Status:  needs_review
 Priority:  Medium           |      Milestone:
Component:  Metrics/Website  |        Version:
 Severity:  Normal           |     Resolution:
 Keywords:                   |  Actual Points:
Parent ID:                   |         Points:
 Reviewer:                   |        Sponsor:
-----------------------------+------------------------------
Changes (by iwakeh):

 * status:  accepted => needs_review


Comment:

 Please find a first [https://gitweb.torproject.org/user/iwakeh/metrics-
 web.git/log/?h=task-24175 patch branch].
 There is only one commit as this consists in moving files around,
 replacing scripts accordingly etc., all in all nothing that can be easily
 done incrementally.
 Please test thoroughly and with some more data than I have available with
 some emphasis on the R related functionality.

 == What changed
 Major changes are the move to an embedded Jetty and finally the move to
 using metrics-base only.
 For the latter, the interim step ([https://gitweb.torproject.org/metrics-
 web.git/commit/shared/build-
 base.xml?id=40d3d39361d513c083aa0f871afc518371e18be5 from a year ago]) of
 first making all modules use a common build environment really helped with
 the current step to finally apply Metrics' java structure and use metrics-
 base.

 I'm also going for a two-step-solution for configuration and operation
 issues here:
 The current patch replaces all operational/deployment bash scripts with
 ant tasks and keeps the overall directory structure, as well as the
 current configuration situation, i.e., this patch is taylored to the
 current deployment server and not yet to easy test configuration, which is
 tackled in ticket #24328.
 I didn't change any package names nor script (R, sql, python) names nor do
 we have a consistent logging approach as these are all new tickets:
 #24328, #24329, #24330, (and the ticket for moving ernie and renaming
 packages, need to look up the number).

 == Comparison to the old situation:
 `ant run-web-prepare` is the former `run-web.sh`.
 The scripts from `shared/bin` are replaced by ant tasks: advbwdist,
 connbidirect, hidserv, clients, legacy, onionperf, webstats, collectdescs.
 Task `make-data-available` copies everything as the earlier script `99
 -copy-stats-files.sh`.
 The R server start.sh is replaced by `ant run-rserver`.
 The various modules run in their own sub-folders (if not changed in
 build.xml these will be generated/modules/<submodule> and logging for the
 logback parts happens to generated/modules/logs/...).

 `ant legacy-create-config` copies the legacy config template to
 `${basedir}/legacy.config`, which is used by `ant legacy` as config file.
 This is just because I found it, but it shouldn't be this way, discussion
 about such topics are referred to ticket #24328.

 The creation of metrics-lib's javadoc has its own sub-task, which is
 called as part of the war file creation.


 == How to test web related things and new modules:
 It is not necessary to run all modules nor to start an R server.
 The war should run fine locally using `java  -DLOGBASE=logs -jar
 generated/dist/metrics-web-1.0.0-dev.war` and then open 127.0.0.1:8080 in
 a browser, of course no graphs or tables will be supplied.


 == Open topic
 On a fresh installation with only current data running detection.py only
 generates an empy (except for the header) userstats-ranges.csv, which
 prevents further processing.
 Also, many other tasks take ages to complete on a fresh system and the
 chain of deployment in build.xml needs some tweaking.
 Please, take a closer look and improve during this first round.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/24175#comment:2>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online