[tor-bugs] #32845 [Core Tor/Tor]: Add UTF-8 validation unit tests

[Tor Bug Tracker & Wiki]

#32845: Add UTF-8 validation unit tests
------------------------------+----------------------------------------
     Reporter:  teor          |      Owner:  teor
         Type:  task          |     Status:  assigned
     Priority:  Medium        |  Milestone:  Tor: 0.4.3.x-final
    Component:  Core Tor/Tor  |    Version:
     Severity:  Normal        |   Keywords:  043-should, utf-8, Unicode
Actual Points:                |  Parent ID:
       Points:  0.2           |   Reviewer:
      Sponsor:                |
------------------------------+----------------------------------------
 We should add unit tests for the following UTF-8 sequences. Their validity
 varies between different programming languages. We should go with the
 common case (if it matches the standard).

 Invalid:
 surrogate nullsurrog threehigh
 EDA081 3000EDA081 EDBFBF

 fourhigh fivebyte sixbyte sixhigh
 F490BFBF FB80808080 FD80808080 FDBFBFBFBF

 Valid:
 fourbyte fourbyte2
 F0908D88 F0BFBFBF

 Valid in the Unicode standard, invalid in torrcs and directory documents:
 nullbyte
 3031320033

 See proposal 285 for details, and for the null byte exception:
 https://gitweb.torproject.org/torspec.git/tree/proposals/285-utf-8.txt

 Test Case Source:
 `POC||GTFO 19`, page 43
 https://www.alchemistowl.org/pocorgtfo/

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/32845>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online