db: move URL_SEP constant from db to top level kallithea module

URL_SEP is used in several places - having it in the db module is too high
level.

__init__ might not be the best place for this, but it does no harm to the
dependencies graph to place constants there.

vcs: fix cloning remote repository with HTTP authentication (Issue #379)

Using a remote clone URI of
http://user:pass@host/...
triggered an exception:

...
E File ".../kallithea/lib/utils.py", line 256, in is_valid_repo_uri
E GitRepository._check_url(url)
E File ".../kallithea/lib/vcs/backends/git/repository.py", line 183, in _check_url
E passmgr.add_password(*authinfo)
E File "/usr/lib/python3.7/urllib/request.py", line 848, in add_password
E self.reduce_uri(u, default_port) for u in uri)
E File "/usr/lib/python3.7/urllib/request.py", line 848, in <genexpr>
E self.reduce_uri(u, default_port) for u in uri)
E File "/usr/lib/python3.7/urllib/request.py", line 875, in reduce_uri
E host, port = splitport(authority)
E File "/usr/lib/python3.7/urllib/parse.py", line 1022, in splitport
E match = _portprog.fullmatch(host)
E TypeError: cannot use a string pattern on a bytes-like object


The authinfo tuple is obtained via mercurial.util.url, which unfortunately
returns a tuple of bytes whereas urllib expects strings.
It seems that mercurial internally has some more hacking around urllib as
urllibcompat.py, which we don't use.

Therefore, transform the bytes into strings before passing authinfo to
urllib. As the realm can be None, we need to check it specifically otherwise
safe_str would return a string 'None'.

A basic test that catches the mentioned problem is added, even though it
does not actually test that cloning with auth info will actually work (it
only tests that it fails cleanly if the URI is not reachable).

Additionally, one use of 'test_uri' in hg/repository.py still needed to be
transformed from bytes to string. For git this was already ok.

tests: consistent use of more helpful raise_exception "fixture"

"Total Crash" was too misleading when debugging problems.

py3: remove safe_unicode in places where it no longer is needed because all strings (except bytes) already *are* unicode strings

(The remaining safe_unicode calls are still needed and can't just be removed,
generally because we in these cases still have to convert from bytes to unicode
strings.)

py3: migrate from urllib2 to urllib

Based on 2to3 urllib, but with unnecessary imports removed.

cleanup: get rid of most "import *"

Apply script generated with the following hack:
(
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.*'(.*)' may be undefined, or defined from star imports.*/sed -ri 's,\\\\<\2\\\\>([^=]|$),XXXX.\2\\\\1,g' \1/gp" | sort -u
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.* undefined name '(.*)'$/sed -ri 's,\\\\<\2\\\\>([^=]|$),XXXX.\2\\\\1,g' \1/gp" | sort -u
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.*'(from .*)\.([^.]*) import \*' used.*/sed -ri 's,\\\\<XXXX\\\\.,\3.,g' \1/gp" | sort -u
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.*'(from .*)\.([^.]*) import \*' used.*/sed -ri 's,\2\\\\.\3 .*,\2 import \3,g' \1/gp" | sort -u
) | grep -v kallithea/bin/kallithea_cli_ishell.py > fix2.sh

codingstyle: replace comparison for equality against None with "is" expression

Both style and correctness.

Reported by flake8.

codingstyle: trivial whitespace fixes

Reported by flake8.

repositories: make sure repositories not only differ in casing

Repositories only differing in case cause problems:
* it can't be stored on case insensitive filesystems (Windows and MacOS)
* some databases can't easily handle case sensitive queries
* users will most certainly be confused by names that only differ in case

We will keep trying to be case sensitive on systems that can ... but on some
systems wrong casings might work. We don't care.

The validators are changed to prevent mixed case repo and repo group names.

Repository sensitivity tests are removed, and insensitivity tests are added
instead.

tests: fix test_permanent_url_protocol_access instability

Since the test repositories are created in the database by scanning the
file system, their IDs depends on the (non-deterministic) order in which
files are listed by the file system.

The solution, of course, is to ask the database for the ID instead of
assuming it will be ID 1.

hg: return 400 Bad Request for hg commands that not are commands

Avoid throwing bare Exceptions which requires framework specific testing.
Instead, return a reasonable http error code and make the test more framework
independent.

The "helpful" message will just be a description of the http exception and not
sent to the client.

db: PullRequest/Repository/RepoGroup/UserGroup: change 'user' to 'owner'

Rename the 'user' and 'user_id' fields on the four classes to something
more informative. The database column names remain unchanged for now;
a later Alembic script can fix the name of these and other columns to
match their Python name.

This might break rcextensions, though, and external scripts that use
the HTML form interface.

db: remove superfluous Session.add calls

Don't re-add objects to the SQLAlchemy Session just because they were
modified. Session.add is only for freshly constructed objects that
SQLAlchemy doesn't know about yet.

The rules are quite simple:

When creating a database object by calling the constructor directly, it
must explicitly be added to the session.

When creating an object using a factory function (like "create_repo"),
the returned object has already (by convention) been added to the
session, and should not be added again.

When getting an object from the session (via Session.query or any of the
utility functions that look up objects in the database), it's already
added, and should not be added again. SQLAlchemy notices attribute
modifications automatically for all objects it knows about.

tests: add as little code as possible in __init__.py

kallithea/tests/__init__.py contained quite a lot of code, including the test
base class TestController. This in itself may be considered bad practice.

Specifically, this poses a problem when using pytest 3.0+, in which asserts
in some files are not automatically rewritten to give improved assert
output. That problem can be fixed by explicitly registering such files for
assertion rewriting, but that register call should be executed _before_ said
files are imported. I.e. if the register call is in
kallithea/tests/__init__.py, assert calls in __init__.py itself can not be
rewritten.

Since the TestController base class does effectively contain asserts, and we
do not want to execute the register call from somewhere outside the
kallithea/tests directory, we need to move the TestController class to
another file (kallithea/tests/base.py) so we can have a register call in
__init__.py before loading base.py.

While not strictly necessary to fix the mentioned pytest problem, we take
the opportunity to fully clean __init__.py and move everything to
the new kallithea/tests/base.py. While doing so, unnecessary imports are
removed, and imports are ordered alphabetically. Explicit imports of symbols
from modules that were already imported as a whole, are removed in favor of
fully qualifying the references (e.g. tempfile._RandomNameSequence).

protocols: fix assertion error when accessing repositories with "permanent" urls (Issue #202)

I am not aware of any good way to test this, so it is tested with a Mercurial only hack.

pytest migration: rename TestControllerPytest back to TestController

The name TestControllerPytest was introduced to allow a temporary situation
where nose/unittest and pytest-based tests could coexist.
This situation is now over, so the base test class can be renamed again.

pytest migration: functional: switch to standard assert statements

Use unittest2pytest to replace unittest-style assert statements (e.g.
assertEqual) with standard Python assert statements to benefit from pytest's
improved reporting on assert failures.

The conversion by unittest2pytest was correct, except for line wrapping
problems.

pytest migration: simplify hg+git test class hierarchies

No need to use multiple inheritance when single inheritance can do it just
as well. Let the git/hg test cases just derive from _BaseTestCase which in
turn derives from TestControllerPytest.

Also remove empty setup_class/teardown_class methods.

pytest migration: convert functional tests with setup/teardown methods

pytest also supports setup/teardown methods like unittest (even though
pytest fixtures are more powerful and should be the end goal).
Only difference is the naming and signature of setUp (setup_method) and
tearDown (teardown_method).

tests: don't use the TESTS_TMP_PATH string

it differs each time you run the test
and therefore doesn't work if you use a separat kallithea test instance
(KALLITHEA_NO_TMP_PATH=1)

instead get the 'real' path from the DB

FIXME: breaks kallithea/tests/functional/test_admin_repos.py

tests: more consistently use unicode where unicode is expected

Nothing but extra u annotation to turn str constants into unicode.

This has been verified by hacking sqlalchemy to fail if wrong string types are passed.

tests: cleanup of unicode in test_create_non_ascii repository test

Repository names are generally conceptually unicode. Prepare for future changes
by explicitly encoding repo_name as utf8 before appending a utf8 string.

Drop test_create_non_ascii - test_delete_non_ascii contains exactly the same
test.

tests: always convert unicode paths to str before passing to vcs layer

Repo names are generally unicode. Don't rely on implicit conversion.

security: apply CSRF check to all non-GET requests

The automatic CSRF protection was broken for POST requests with no
request payload parameters (but possibly containing request URI
parameters); a security hole was narrowly avoided because the code
base quite consistently checks the request method in the same way,
and because of browser protection against PUT/DELETE CSRF attacks.

Since explicit is better than implicit, the better way of checking
the HTTP request method is to simply check request.method, instead
of checking if request.POST is non-empty, which is subtly different
(it doesn't catch POST requests if all parameters are in the query
string) and non-obvious (because it also applies to PUT requests).

The commit also fixes some tests which relied on the CSRF protection
being broken. It does not fix all the controllers that still does
the misleading request.POST check, but since the CSRF check has now
been tightened, those are no longer a potential security issue.

repos: fix abuse of dead 'repo' routing

The repo "show" controller didn't do anything and was unused. There was a
routing GET entry for it but it was only used for generating URLs for DELETE
and PUT operations that have separate controllers that happen to have the same
URL.

Use the right routing entries when generating URLs and drop the dead code.

repos: refactor repo group handling - extract generic functionality that can be used consistently

By using the right repo group list in the form, it is no longer possible for
users to pick wrong repo creation locations ... and doing so anyway will be
caught earlier on with a more generic error message at the specific place.

tests: avoid executing tests in base classes

Fixes lots of failures with pytest by making sure base test classes don't
inherit from unittest.TestCase or by prefacing name with an underscore so tests
don't get executed by pytest.

tests: provide _authentication_token when POSTing

So far not used, just preparing for the the time when the actual checking is
introduced ...

This change is very verbose. self.app.post should perhaps just at this value
automagically ...

spelling: consistent capitalization of URL

Change Url / url into URL.
Additionally, convert the sole use of 'Uri' to URL.

cleanup: avoid some 'except Exception' catching - catch specific exceptions or log it and show what happened

This has a risk of introducing regressions ... but we want to get rid of all
exception muting and make the whole system less fragile and easier to debug.

Second step in two-part process to rename directories.
This is the actual directory rename.