[tor-bugs] #32239 [Internal Services/Tor Sysadmin Team]: setup a cache frontend for the blog

Tor Bug Tracker & Wiki blackhole at torproject.org
Thu Nov 7 21:49:01 UTC 2019


#32239: setup a cache frontend for the blog
-------------------------------------------------+-------------------------
 Reporter:  anarcat                              |          Owner:  anarcat
     Type:  task                                 |         Status:
                                                 |  merge_ready
 Priority:  Medium                               |      Milestone:
Component:  Internal Services/Tor Sysadmin Team  |        Version:
 Severity:  Normal                               |     Resolution:
 Keywords:                                       |  Actual Points:
Parent ID:  #32090                               |         Points:
 Reviewer:                                       |        Sponsor:
-------------------------------------------------+-------------------------

Comment (by anarcat):

 at first glance, our hit ratio is not great. i used the following lnav log
 format parser:

 {{{
 {
     "cache" : {
         "title" : "Tor Nginx cache Log Format",
         "description" : "Log format used by the Tor Project for its Nginx
 cache servers.",
         "url" : "http://example.com/log-format.html",
         "regex" : {
             "basic" : {
                 "pattern" : "^(?<client>[\\w\\d:.-]+) -
 (?<user>[\\w\\d:.-]+)
 \\[(?<timestamp>\\d{2}/\\w{3}/\\d{4}:\\d{2}:\\d{2}:\\d{2}(?:\\.\\d+)?(?:
 [+-]\\d{4}))?\\] \"(?<method>\\w+) (?<path>\\S+) (?<transport>\\S+)\"
 (?<status_code>\\d+) (?<resp_size>\\d+|-) \"(?<referer>[^\"]+)\" \"-\"
 (?<upstream_addr>\\S+) (?<upstream_response_time>[\\d.-]+)
 (?<upstream_cache_status>\\S+) (?<request_time>[\\d.-]+)$"
             }
         },
         "level-field" : "upstream_cache_status",
         "level" : {
             "error" : "MISS",
             "warning" : "BYPASS|EXPIRED|STALE|UPDATING|REVALIDATED",
             "info" : "HIT"
         },
         "value" : {
             "client" : { "kind" : "string", "identifier" : true,
 "collate": "ipaddress" },
             "user" : { "kind" : "string", "identifier" : true },
             "method" : { "kind" : "string", "identifier" : true },
             "path" : { "kind" : "string", "identifier" : true },
             "transport" : { "kind" : "string", "identifier" : true },
             "status_code" : { "kind" : "integer", "identifier" : true },
             "resp_size" : { "kind" : "integer", "identifier" : true },
             "referer" : { "kind" : "string", "identifier" : true },
             "upstream_addr" : { "kind" : "string", "identifier" : true,
 "collate": "ipaddress" },
             "upstream_response_time" : { "kind" : "float", "identifier" :
 true },
             "upstream_cache_status" : { "kind" : "string", "identifier" :
 true },
             "request_time" : { "kind" : "float", "identifier" : true }
         },
         "sample" : [
             {
                 "line" : "0.0.0.0 - - [07/Nov/2019:21:28:09 +0000] \"GET
 /blog/feed HTTP/2.0\" 200 483311 \"-\" \"-\" - - HIT 0.373",
                 "level" : "info"
             },
             {
                 "line" : "0.0.0.0 - - [07/Nov/2019:21:28:10 +0000] \"GET
 /new-release-tor-browser-90 HTTP/2.0\" 200 26764
 \"https://www.google.com/\" \"-\" [2620:12a:8000::2]:443 0.416 MISS
 0.420",
                 "level" : "error"
             },
             {
                 "line" : "0.0.0.0 - - [07/Nov/2019:21:28:10 +0000] \"GET
 /robots.txt HTTP/1.1\" 200 1594 \"-\" \"-\" [2620:12a:8001::2]:443 0.020
 MISS 0.020",
                 "level" : "error"
             }
         ]
     }
 }

 }}}

 and then ran lnav on the access logfile, then ran the following query
 (after hitting the `;` key):

 {{{
 select count(*), upstream_cache_status from logline where status_code <
 300 group by upstream_cache_status;
 }}}

 which gave the following results:

 {{{
 count(*) upstream_cache_status
 │
        6│-
 │
        7│EXPIRED
 │
     2925│HIT
 │
      680│MISS
 │
        5│REVALIDATED
 }}}

 That's an 80% hit ratio, and that's when ignoring any status code above
 2xx. If those are included, our hit ratio is around 72%.

 So there is still some work to be done to improve the hit ratio.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/32239#comment:16>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online


More information about the tor-bugs mailing list