py3: remove safe_unicode in places where it no longer is needed because all strings (except bytes) already *are* unicode strings

(The remaining safe_unicode calls are still needed and can't just be removed,
generally because we in these cases still have to convert from bytes to unicode
strings.)

py3: migrate from urllib2 to urllib

Based on 2to3 urllib, but with unnecessary imports removed.

vcs: drop the superfluous and leaky hgcompat "layer"

Explicit is better. And gives less pyflakes noise.

lib: refactor _get_access_path as get_path_info

We will need it later when it gets more tricky to get the path from environ ...

middleware: unify Mercurial and Git _handle_request in the VCS base class

Finally, it is more clear in what the VCSs are different ... and what generic
setup code is needed.

middleware: move VCS specific functionality to the VCS apps

For Git, inline _handle_githooks into a wrapper app so it is simple and
explicit and executed exactly when the request is processed.

For Mercurial, similarly, only set REPO_NAME when the request actually is
processed.

middleware: also parse action in BaseVCSController parse_request

Some of the parsing in parse_request can be reused.

middleware: introduce BaseVCSController parse_request retrieving repo_name and use that for VCS dispatch

Return a parsed_request namespace. For now, it just contains repo_name, thus
replacing is_hg / is_git and __get_repository.

The fallback to the wrapped application now happens in the base class and the
VCS implementations can be simplified.

middleware: move handling of permanent repo URLs to separate middleware

This is about the handling of repo URLs like '_123' for the repo with repo_id
123. The de-mangling of such URLs was spread out across multiple layers. It
fits much more nicely as a middleware layer. The code in routing and simplehg /
simplegit can thus be removed.

The base _get_by_id function was confusing - fix it by removing it. To do that,
refactor utils introducing fix_repo_id_name to replace get_repo_by_id.

We now assume in the application that we never have any extra leading '/' in
URL paths.

And while trailing extra '/' might be fine in actual URLs, they must be handled
at the routing level, not propagated through all layers. This changeset is not
really changing that.

middleware: drop fix_PATH

Back out ccbdff90e5a0. That seemed like an odd hack. In order to work properly,
it should not only be applied for protocol access middleware, but also for web
UI and for commands. So evidently, it is not really necessary.

The problem it describes is fixed much better in 5e501b6ee639 by setting the
right python executable in the hook scripts, further improved in 1bafb2d07709
and 6df08d78f8e7 to *actually* use the right python executable.

middleware: drop special handling of hg RepoError

The 'not found' case should never happen - we check that repo exists before
calling out to hg.

And the Mercurial RepoError handling was muting if *not* 'not found'. That
seems very harmful - better to get a clear error.

We are thus better without the special handling of RepoError.

middleware: simplify pygrack wrapping - there is no need for any extras

For symmetry, do the same in simplehg where it is completely unused.

utils: refactor make_ui to always read from db, optionally also augmenting with hgrc content

simplehg was the only user of reading hgrc content ... and it would rather just
have everything at once.

The implementation in utils is moved around and re-indented, but without
significant changes.

locking: drop the pull-to-lock / push-to-unlock functionality

The feature is not worth the maintenance cost. The locking is too coarse and
unflexible with insufficient UI and UX. The implementation is also quite
invasive in tricky areas of the code, and thus high maintenance.
Dropping this will enable other cleanup ... or at least make it easier.

hg: make protocol access control more explicit

Enumerate all currently known commands, and default to require 'push' access
for all unknown commands.

This change mitigates some privilege escalation problems like CVE-2018-1000132
which was fixed in Mercurial 4.5.1 and currently is described on
https://www.mercurial-scm.org/wiki/WhatsNew#Mercurial_4.5.1_.2F_4.5.2_.282018-03-06.29 .

hg: explicit handling of the 'batch' protocol command - consider it a "push" command if any of the batch commands are

This change mitigates some privilege escalation problems like CVE-2018-1000132
which was fixed in Mercurial 4.5.1 and currently is described on
https://www.mercurial-scm.org/wiki/WhatsNew#Mercurial_4.5.1_.2F_4.5.2_.282018-03-06.29 .

cache: move cache invalidation from web handler to post push hook

We need the post push hook anyway ... and having the cache invalidation here
will also work for ssh pushes in the future.

The name log_push_action is thus no longer spot-on. That might change later,
but requires some care as it also is used directly as hook name.

Note that having cache invalidation in the hook will do that debug logging no
longer will appear in the server log.

Based on a patch by Dominik Ruf.

middleware: inline the _set_extras invocation from the __inject_extras method, leaving behind only functionality for augmenting ui from hgrc

I don't know about the logic for merging hg configs ... but for now it is left untouched.

hg: support introduction of wsgiresponse in Mercurial 4.6

Lock tests would fail without this.

codingstyle: trivial whitespace fixes

Reported by flake8.

lock: fix for Mercurial 3.6+ - wrap hgweb to catch Locked exceptions from hooks

With Mercurial 3.6, the handling of WSGI responses changed. The hook exceptions
are no longer raised directly when app(environ, start_response) is called so
the 'except HTTPLockedRC as e' block in _handle_request (a few lines above )
does not work anymore because the exception happens later.

Therefore I created a wrapper class that can catch the exceptions.

This makes locking work again and fixes lock related tests like
TestVCSOperations.test_clone_after_repo_was_locked_hg which expect certain
output of the hg client in case of an HTTPLockedRC exception.

Depending on how https://bz.mercurial-scm.org/show_bug.cgi?id=5232 gets
handled, this fix might become obsolete in the future.

(Modified by Mads Kiilerich)

middleware: convert check_locking_state to be a separate function

Prepare for use with SSH.

lib: move special Mercurial HTTP listkey exception out of _check_locking_state

Prepare for use with SSH.

cleanup: refer less to User.DEFAULT_USER

Down the road we might want to identify the default user in another
way than by username.

vcs: dedup auth code between Hg and Git middleware

There's a lot of verbatim duplicated code in the Hg and Git VCS
middleware. This attempts to deduplicate a bit of it.

The _authorize function is a bit awkward, but for now the goal is simply
to remove duplicated code, not improving program structure and design.

As such, the code in _authorize is almost a verbatim copy of the code
removed in the two controllers.

vcs: remove confusing and unnecessary local variable

This removes the "username" local variable, substituting the equivalent
"user.username". The function also has another, unrelated use of the
same local variable name; this OTHER "username" variable is untouched.

The relevant code is duplicated between Hg and Git... and thus also
this change.

vcs: remove non-sensical conditional block

__get_action can only return "pull" or "push"; any other value is an
error on our end. And indeed, if not, the old code would crash further
down in the function (under "CHECK LOCKING") because the "user" local
variable was not assigned.

This also corrects the comment that incorrectly suggests the perm check
is only relevant for anonymous access.

The relevant code is duplicated between Hg and Git... and thus also
this change.

(Diff becomes clearer if whitespace changes are ignored.)

hg: return 400 Bad Request for hg commands that not are commands

Avoid throwing bare Exceptions which requires framework specific testing.
Instead, return a reasonable http error code and make the test more framework
independent.

The "helpful" message will just be a description of the http exception and not
sent to the client.

hg: drop pointless push_ssl configuration setting - if there is a risk push can be compromised, credentials can also easily be stolen for pull

Everybody should have a ssl-only setup now. Alternatively, there is a use case
for 'only anonymous traffic on ssl - all authentication and authenticated
traffic must be on ssl'. That can be done with proper web server configuration.

protocols: fix assertion error when accessing repositories with "permanent" urls (Issue #202)

I am not aware of any good way to test this, so it is tested with a Mercurial only hack.

lock: fix for Mercurial 3.6+ - wrap hgweb to catch Locked exceptions from hooks

With Mercurial 3.6, the handling of WSGI responses changed. The hook exceptions
are no longer raised directly when app(environ, start_response) is called so
the 'except HTTPLockedRC as e' block in _handle_request (a few lines above )
does not work anymore because the exception happens later.

Therefore I created a wrapper class that can catch the exceptions.

This makes locking work again and fixes lock related tests like
TestVCSOperations.test_clone_after_repo_was_locked_hg which expect certain
output of the hg client in case of an HTTPLockedRC exception.

Depending on how https://bz.mercurial-scm.org/show_bug.cgi?id=5232 gets
handled, this fix might become obsolete in the future.

(Modified by Mads Kiilerich)

middleware: allow git and hg users to use email address instead of the username

This commit also replaces __get_user('default') with a call to a more widely
used User.get_default_user() function, and removes no longer really used
__get_user() methods from both SimpleHg and SimpleGit.

middleware: decode the repo_name received from http header to unicode

The middlewares seemed to make the incorrect assumption that the headers
contained unicode. Or to put it differently: They relied on the Python default
encoding to be able to convert to unicode instead of using safe_unicode. It
would thus fail if running with LANG=C.

Instead, utilize that the header actually contains str_repo_name and explicitly
decode that to unicode.

vcs: invalidate repo caches _after_ a push finishes

Caches were invalidated right after creating the result iterator, before
actually applying the change. Caches would thus be refreshed too early and soon
be outdated.

This bug was especially seen causing errors with missing revisions when
creating PRs right after pushing.

cleanup: consistently use 'except ... as ...:'

Use the Python 2.6+ syntax instead of the old confusing 'except ..., ...'
syntax.

simplehg: don't quiet Hg output for push

I want to be able to see informational messages from e.g. the Hg Bugzilla
extension.

Here's output without the patch. This is identical to the output produced if
the Bugzilla extension is disabled:

remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 1 changes to 1 files
remote: Repository size .hg:158.2 kB repo:0 B total:158.2 kB
remote: Last revision is now r324:6c03abbabb46

and with the patch:

remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 1 changes to 1 files
remote: Repository size .hg:158.2 kB repo:0 B total:158.2 kB
remote: Last revision is now r324:6c03abbabb46
remote: bug 3270 already knows about changeset 6c03abbabb46

The last line indicates that the Bugzilla bug text already contains an entry
for that changeset. It's produced by a call to self.ui.status() in the
extension.

I think the point here is that the deleted code is ensuring that the remote hg
(i.e. the Kallithea hg) always runs with the --quiet flag when receiving a
push. This seems an arbitrary decision to me, and one that removes potentially
useful information from the output. I believe this behaviour is different to
that seen by the user if pushing to hgweb (I've not tried it, but inspected the
hgweb source for setting quiet) and is certainly different to pushing over ssh.

https: introduce https_fixup config setting to enable the special https hacks

Without https_fixup, correctly configured WSGI systems work correctly.

The https_fixup middleware will only be loaded when enabled in the configuration.

Correct licensing information in individual files.

The top-level license file is now LICENSE.md.

Also, in various places where there should have been joint copyright holders
listed, a single copyright holder was listed. It does not appear easy to add
a link to a large list of copyright holders in these places, so it simply
refers to the fact that various authors hold copyright.

In future, if an easy method is discovered to link to a list from those
places, we should do so.

Finally, text is added to LICENSE.md to point to where the full list of
copyright holders is, and that Kallithea as a whole is GPLv3'd.

Second step in two-part process to rename directories.
This is the actual directory rename.