hg: restore make_ui reading of .hg/hgrc

This regressed with f83326e2e66c, released in 0.6.0 .

Fix problem reported by Felipe Cardoso Resende.

hooks: use internal Mercurial module for update after push

Don't rely on having a working 'hg' command in $PATH.

repo-scan: log error and continue after failure creating a scanned repo

Continue after for example:
Exception: invalid repo name Foo

repo-scan: overwrite_git_hooks also implies install_git_hooks

(Otherwise, we should probably report an error when ignoring it.)

hooks: move the vcs hook entry points and setup code out of lib

Mercurial hooks are running in a process that already has been initialized, so
they invoke the hooks lib directly. Git hooks are binaries and need a lot of
initialization before they can do the same. Move this extra setup code
elsewhere.

Having this high level code in bin is perhaps also not ideal, but it also
doesn't seem that bad: that is where other command line entry points invoke
make_app.

(It seems like it could be adventageous to somehow use "real" bin commands for
hooks ... but for now we use the home-made templates.)

Note: As a side effect of this change, all git hooks *must* be re-installed
when upgrading.

db: consistently use the name settings for the return value from get_app_settings

Get rid of yet another old rc reference.

(The "settings" in auth is slightly different, but pretty much the same.)

config: move config.conf to kallithea.lib

kallithea.config is mainly the WSGI entry point with TG stuff, and thus a high
level controller thing - not a place to store configuration.

Note: The content of conf.py is kind of similar to what we have in
kallithea/__init__.py . These two should perhaps be merged somehow.

lib: move extract_mentioned_users out of utils2

Avoid model.db reference to keep utils2.py independent of Kallithea classes.

db: move URL_SEP constant from db to top level kallithea module

URL_SEP is used in several places - having it in the db module is too high
level.

__init__ might not be the best place for this, but it does no harm to the
dependencies graph to place constants there.

extensions: rename 'rcextensions' into 'extensions' but provide compatibility

The 'rc' prefix is legacy.
Rename 'rcextensions' into 'extensions', updating all references.
Compatibility with the old name will be retained for a while, and removed in
a later release. Migrating is as simple as renaming a file.

extensions: preparatory refactoring

Small refactoring:

- use 'try..except' rather than 'if-then', which removes the need for a
separate 'path' variable and is more 'pythonic'.
- promote log from 'debug' to 'info' and execute it immediately after
loading before anything else.
- clarify comments related to INDEX_EXTENSIONS

extensions: rename internal names away from 'rcextensions'

Additionally, get rid of the unnecessary 'EXT' variable in load_extensions.

scm: refactor install_git_hooks

Rename, simplify, and negate some logic to make the flow more readable to me
and give better logging.

For example, "force_create" were more about "force overwrite". Calling it
"force" is more precise.

config: set base_path config in set_app_settings using Ui.get_repos_location() instead of in app_cfg using make_ui()

Only hit the database once (when starting the application) to get this.

It would perhaps be more elegant to set it directly, for example in
kallithea.base_path ... but it seems like a setting that really belongs in the
.ini file ...

hg: handle Mercurial RepoError correctly in is_valid_repo_uri

RepoError would be leaked by is_valid_repo_uri.

Now, when for example validating an ssh URL without having the ssh client
binary, it will be shown/logged as:
remote: /bin/sh: ssh: command not found
2020-03-17 17:28:53.907 WARNI [kallithea.model.validators] validation of clone URL 'ssh://no-ssh.com/' failed: Mercurial RepoError: no suitable response from remote hg
and shown in the UI as 'Invalid repository URL'.

hg: read everything from hgrc, without config section whitelisting (Issue #246)

The whitelisting seems pointless, is hard to maintain, and can't be customized.

Also, mercurial.localrepo.instance will read the full config file anyway.

hg: always show and run Mercurial hooks in alphabetical order (Issue #246)

Mercurial will generally run hooks in the order they are found in the
configuration. For entries found in the database, there is no such order.
Instead, always use alphabetical order for these.

Since we now want to order things explicitly in the db query, we want an index
with a composite key. We do that even though we don't really need it for the
few entries in this table, and even though it might/could use the same index as
the existing unique constraint. This composite UniqueConstraint was added in
b9f4b444a172 where it replaced a wrong UniqueConstraint that could/should have
been removed in c25191aadf92. Fix that while touching this area and running a
migration script.

hg: set HGPLAIN globally to minimize potential impact from reading hgrc configuration when creating ui

See `hg help scripting` for HGPLAIN description ... but Kallithea will invoke
Mercurial internals directly, so HGPLAIN shouldn't really impact anything.

validators: don't catch all Exceptions as invalid clone URIs, be specific

When adding a new repository with a remote clone URI, the URI will be
validated in some way. Several exceptions could occur during that
validation.

Previously, the code would catch based on 'Exception', which means that
_any_ exception would cause the URI to be found invalid. This means that
errors in the code (e.g. related to Python 3 conversion) were also
categorized as 'invalid clone URI'. And thus, the tests that test an
actually invalid URI would pass, even though there was a bug.

Now, things have been refactored so it only is relevant to catch
InvalidCloneUriException. Any other exception will now yield a 500 Internal
Server Error, as expected.

utils: let is_valid_repo_uri report URLError from ._check_url as InvalidCloneUriException

Better encapsulation of internals.

utils: don't raise NotImplementedError in is_valid_repo_uri when the user provides an "invalid" URL

We prefer to reserve NotImplementedError for programming errors that allow
access to unsupported corner cases in code.

validators: introduce InvalidCloneUriException instead of throwing bare Exceptions for invalid clone URIs

When adding a new repository with a remote clone URI, the URI will be validated
in some way. In case of errors, it would raise 'Exception' to report invalid
URLs. Code calling the validation would thus have to catch 'Exception', which
means that _any_ exception would cause the URI to be found invalid.

Instead, create a special exception type intended to be used for all exceptions
we know can occur during normal URL validation.

py3: remove safe_unicode in places where it no longer is needed because all strings (except bytes) already *are* unicode strings

(The remaining safe_unicode calls are still needed and can't just be removed,
generally because we in these cases still have to convert from bytes to unicode
strings.)

cache: drop setup_cache_regions - tg will already have done that and coerced the types correctly

The configuration and type fixing will be invoked from make_base_app, and we
will thus not have to do it:

File "kallithea/config/middleware.py", line 31, in make_app_without_logging
return make_base_app(global_conf, full_stack=full_stack, **app_conf)
File ".../python3.7/site-packages/tg/configuration/app_config.py", line 176, in make_base_app
wrap_app)
File ".../python3.7/site-packages/tg/configurator/application.py", line 112, in _make_app
app = TGApp(conf)
File ".../python3.7/site-packages/tg/wsgiapp.py", line 49, in __init__
app_wrapper = wrapper(self.wrapped_dispatch, self.config)
File ".../python3.7/site-packages/tg/appwrappers/caching.py", line 36, in __init__
self.options = parse_cache_config_options(config)
File ".../python3.7/site-packages/beaker/util.py", line 430, in parse_cache_config_options

This will fix a py3 problem where setup_cache_regions was run *after* beaker
had coerced types, thus introducing string types in the config where beaker
expected the integers it had put there.

vcs: drop the superfluous and leaky hgcompat "layer"

Explicit is better. And gives less pyflakes noise.

py3: add b'' annotations in some places where they will be needed later

Mostly entirely trivial adding of b prefix that is a ignored for py2 ... and
also a bit of related trivial reformatting/refactorings.

lib: let get_git_version invoke git as all other commands do, without special options

There is no need for _bare or _safe. It is fine to have '-c
core.quotepath=false' before '--version', and it is perfectly fine to get a
RepositoryError if things go terribly wrong.

lib: some cleanup of utils.py imports

Utils really shouldn't rely on model ...

utils: drop make_ui parameter clear_session - this utility function should never mess with sessions

clear_session seems to have been based on the idea that utility functions that
happened to create a session also should clean up. But instead, we now
consistently take care of session removal at a high level.

git: exit early if Git is found but too old (Issue #342)

The warning in the logs were too easy to miss.

It is still OK if git isn't found at all... for example if git_path
configuration doesn't point at something that looks like git.

The git_path configuration already had no default, so just make the code path
for that case more clean. (An easy next/alternative step could be to remove git
from BACKENDS if it isn't configured ...)

The system exit is similar to what is done in 0e33880b2897 ... even though
exiting from from a library might be a bit worse ...

git: refactor check_git_version

Emphasize the code path where a wrong git version is found.

middleware: move handling of permanent repo URLs to separate middleware

This is about the handling of repo URLs like '_123' for the repo with repo_id
123. The de-mangling of such URLs was spread out across multiple layers. It
fits much more nicely as a middleware layer. The code in routing and simplehg /
simplegit can thus be removed.

The base _get_by_id function was confusing - fix it by removing it. To do that,
refactor utils introducing fix_repo_id_name to replace get_repo_by_id.

We now assume in the application that we never have any extra leading '/' in
URL paths.

And while trailing extra '/' might be fine in actual URLs, they must be handled
at the routing level, not propagated through all layers. This changeset is not
really changing that.

utils: refactor make_ui to always read from db, optionally also augmenting with hgrc content

simplehg was the only user of reading hgrc content ... and it would rather just
have everything at once.

The implementation in utils is moved around and re-indented, but without
significant changes.

hooks: make sure push and pull hooks always are enabled

Don't put things in the database when we pretty much assume they always have
exact content, without any reasonable use case for customization.

locking: drop the pull-to-lock / push-to-unlock functionality

The feature is not worth the maintenance cost. The locking is too coarse and
unflexible with insufficient UI and UX. The implementation is also quite
invasive in tricky areas of the code, and thus high maintenance.
Dropping this will enable other cleanup ... or at least make it easier.

utils: let make_ui return an empty ui.ui() object if the requested hgrc file isn't present

89c30b145bb8 lost an "if repoui" in simplehg, and Kallithea would thus fail on
serving repos without a .hg/hgrc :

File ".../kallithea/lib/middleware/simplehg.py", line 285, in _augment_hgrc
for k, v in repoui.configitems(section):
AttributeError: 'bool' object has no attribute 'configitems'

It is *only* simplehg that invoke make_ui with read_from 'file'. In this case
it could return False if the requested file wasn't present.

It was odd and inconsistent that make_ui with read_from 'file' either returned
an ui object or the value False.

To fix the problem, just let make_ui return the empty ui.ui() if no .hg/hgrc is
present. An ui object that just doesn't have any config sections is much better
than any alternative.

utils: move repo_name_slug to utils2 to prevent import cycle on setup_db

After commit 57a733313e4f, 'gearbox setup-db -c my.ini' fails with an import
cycle as follows:

Traceback (most recent call last):
File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/bin/gearbox", line 11, in <module>
sys.exit(main())
File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/gearbox/main.py", line 199, in main
return gearbox.run(args)
File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/gearbox/main.py", line 145, in run
return self._run_subcommand(remainder)
File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/gearbox/main.py", line 149, in _run_subcommand
subcommand = self.command_manager.find_command(argv)
File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/gearbox/commandmanager.py", line 78, in find_command
cmd_factory = cmd_ep.resolve()
File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2324, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/home/tdescham/repo/contrib/kallithea/kallithea-release/kallithea/lib/paster_commands/setup_db.py", line 27, in <module>
from kallithea.lib.db_manage import DbManage
File "/home/tdescham/repo/contrib/kallithea/kallithea-release/kallithea/lib/db_manage.py", line 47, in <module>
from kallithea.model.repo_group import RepoGroupModel
File "/home/tdescham/repo/contrib/kallithea/kallithea-release/kallithea/model/repo_group.py", line 35, in <module>
import kallithea.lib.utils
File "/home/tdescham/repo/contrib/kallithea/kallithea-release/kallithea/lib/utils.py", line 48, in <module>
from kallithea.model.repo_group import RepoGroupModel
ImportError: cannot import name RepoGroupModel


i.e. kallithea.model.repo_group wants to import kallithea.lib.utils which
in turn wants to import kallithea.model.repo_group.

In fact there exists kallithea.lib.utils and kallithea.lib.utils2.
The current split is that 'utils2' contains 'simple' utilities, none of which
depend on kallithea models, controllers, ... In contrast, 'utils' does rely
on such kallithea classes.

As kallithea.model.repo_group was only include kallithea.lib.utils for its
repo_name_slug method, which has no dependency on other kallithea classes,
move that method (and its dependent recursive_replace) to
kallithea.lib.utils2 instead. This fixes the import cycle.

utils: move clone URI validator function to more generic utils.is_valid_repo_uri

No changes to the functionality, even though the API and implementation could
use some clean-up ...

utils: move clone URI validator function to more generic utils.is_valid_repo_uri

No changes to the functionality, even though the API and implementation could
use some clean-up ...

codingstyle: trivial whitespace fixes

Reported by flake8.

utils: remove Session after we are done using it in set_app_settings

When the Kallithea WSGI application or celeryd is started, TurboGears
app_config.make_base_app calls Kallithea app_cfg.setup_configuration which runs
utils.set_app_settings. That function will read settings from the database and
store them in the global config. It uses a database session which is created on
demand, but this session was not dismissed but left around for the next thing
that asked for a database session. MySQL will by default close connections
after 1 hour, so when celery tried to run a task after 1 hour of inactivity, it
could fail because of the closed connection.

utils.set_app_settings must thus remove the Session after use, just like
auth.set_available_permissions do.

This will thus fix for example some MySQL connection problems seen with Celery.

git: clarify that non-bare git repos not are supported (Issue #254)

Non-bare repos must have a branch checked out ... and then you can't push to
the branch.

repo-scan: rewrite get_filesystem_repos to use os.walk instead of stupid recursion

I think this is more readable. It is also faster. Perhaps because the more
readable implementation makes it easier to optimize.

backend: replace Pylons with TurboGears2

Replace the no-longer-supported Pylons application framework by TurboGears2
which is largely compatible/similar to Pylons.
Some interesting history is described at:
https://en.wikipedia.org/wiki/TurboGears

Changes by Dominik Ruf:
- fix sql config in test.ini

Changes by Thomas De Schampheleire:
- set-up of test suite
- tests: 'fix' repo archival test failure
Between Pylons and TurboGears2, there seems to be a small difference in the
headers sent for repository archive files, related to character encoding.
It is assumed that this difference is not important, and that the test
should just align with reality.
- remove need to import helpers/app_globals in lib
TurboGears2 by default expects helpers and app_globals to be available
in lib. For this reason kallithea/lib/__init__.py was originally changed
to include those files. However, this triggered several types of
circular import problems. If module A imported something from lib (e.g.
lib.annotate), and lib.helpers imported (possibly indirectly) module A,
then there was a circular import. Fix this by overruling the relevant
method of tg AppConfig, which is also hinted in the TurboGears2 code.
Hereby, the include of something from lib does not automatically import
helpers, greatly reducing the chances of circular import problems.
- make sure HTTP error '400' uses the custom error pages
TurboGears2 does not by default handle HTTP status code
'400 (Bad Request)' via the custom error page handling, causing a
standard non-styled error page.
- disable transaction manager
Kallithea currently handles its own transactions and does not need the
TurboGears2 transaction manager. However, TurboGears2 tries to enable it
by default and fails, throwing an error during application initialization.
The error itself seemed to be harmless for normal application functioning,
but was nevertheless confusing.
- add backlash as required dependency: backlash is meant as the WebError
replacement in TurboGears2 (originally WebError is part of Pylons). When
debug==true, it provides an interactive debugger in the browser. When
debug==false, backlash is necessary to show backtraces on the console.
- misc fixes

model: remove BaseModel class

The only remaining purpose of this class was to provide the "sa" field,
allowing a custom SQLAlchemy session to be used for model operations.
However, this never actually worked, nor was it used anywhere.

There's always a global SQLAlchemy session associated with the current
thread; using another session for a single function call does not make
any sense (as sessions cannot be mixed), unless the code works carefully
to ensure the two sessions (and all objects loaded from them) are kept
completely separate. Suffice to say that Kallithea does no such thing,
thus there's no need to pretend to support multiple concurrent sessions.

cleanup: do Session.add directly inside _create_default_perms

Let the various _create_default_perms functions add the created object
to the SQLAlchemy database session, instead of having the caller do it.

This is in accordance with the Kallithea contribution guidelines:

When creating an object using a factory function (like create_repo),
the returned object has already (by convention) been added to the
session, and should not be added again.

cleanup: remove SQLAlchemy session argument to action_logger

There's always a global SQLAlchemy session associated with the current
thread; using another session for a single function call does not make
any sense (as sessions cannot be mixed), unless the code works carefully
to ensure the two sessions (and all objects loaded from them) are kept
completely separate. Suffice to say that Kallithea does no such thing,
thus there's no need to pretend to support multiple concurrent sessions.

config: refactor beaker configuration

While refactoring:

* drop support for 'cache.' prefix - it is only 'beaker.config.' that ever has
been documented or used in examples
* make prefix stripping more safe (if the prefix should appear twice in the key)
* trust exactly what has been configured - drop stripping of strings
* drop int conversion of 'expire' - beaker documentation suggests strings are fine

tests: move test environment setup code out of utils.py

Move create_test_index and create_test_env, which are two functions
that create test fixtures, into kallithea.tests.fixture.

lib: move jsonify from utils to base

Suggested by Mads Kiilerich.

The jsonify method is the only thing in utils that directly uses pylons.
Move it to base where it fits better and we can use existing global imports.

git: clarify that non-bare git repos not are supported (Issue #254)

Non-bare repos must have a branch checked out ... and then you can't push to
the branch.

utils: fix broken URL in jsonify warning

Also add a note summarizing the linked concerns.

lib: move get_custom_lexer from utils to pygmentsutils

get_custom_lexer is the only dependency from helpers to utils.
In attempting to get a clearer dependency tree, we can move out
get_custom_lexer to a different place so that helpers does not depend on
utils.

It so happens that there already is a pygmentsutils.py file in lib, which is
a very good fit, since the lexers used in Kallithea are effectively provided
by pygments.

db: PullRequest/Repository/RepoGroup/UserGroup: change 'user' to 'owner'

Rename the 'user' and 'user_id' fields on the four classes to something
more informative. The database column names remain unchanged for now;
a later Alembic script can fix the name of these and other columns to
match their Python name.

This might break rcextensions, though, and external scripts that use
the HTML form interface.

tests: add as little code as possible in __init__.py

kallithea/tests/__init__.py contained quite a lot of code, including the test
base class TestController. This in itself may be considered bad practice.

Specifically, this poses a problem when using pytest 3.0+, in which asserts
in some files are not automatically rewritten to give improved assert
output. That problem can be fixed by explicitly registering such files for
assertion rewriting, but that register call should be executed _before_ said
files are imported. I.e. if the register call is in
kallithea/tests/__init__.py, assert calls in __init__.py itself can not be
rewritten.

Since the TestController base class does effectively contain asserts, and we
do not want to execute the register call from somewhere outside the
kallithea/tests directory, we need to move the TestController class to
another file (kallithea/tests/base.py) so we can have a register call in
__init__.py before loading base.py.

While not strictly necessary to fix the mentioned pytest problem, we take
the opportunity to fully clean __init__.py and move everything to
the new kallithea/tests/base.py. While doing so, unnecessary imports are
removed, and imports are ordered alphabetically. Explicit imports of symbols
from modules that were already imported as a whole, are removed in favor of
fully qualifying the references (e.g. tempfile._RandomNameSequence).

Turbogears2 migration: remove some references to Pylons in comments

In order to minimize the diff of the actual Turbogears2 migration, this
commit already removes certain unnecessary references to Pylons from the
Kallithea source base.
Places where the reference to Pylons is important are still kept for now, as
well as references in kallithea/config where many changes are made for
Turbogears2 anyway.

Turbogears2 migration: use sqlalchemy.url iso sqlalchemy.db1.url

In Turbogears2, much of the application initialization is handled by the
framework, whereas in Pylons the application was responsible for it.
Initializing SQLAlchemy is one such part of initialization which is handled
by Turbogears2.

Turbogears2 expects the configuration file to refer to the database using
'sqlalchemy.url' rather than the current 'sqlalchemy.db1.url'. While the
exact name is not really important, not following this approach means we'll
need to override the sqlalchemy initialization method.

Therefore, as a preparation to the Turbogears2 migration, already change the
database reference string under Pylons.
When upgrading to a version of Kallithea containing this commit, the .ini
file will manually need to be adapted to remove the .db1 strings.

utils: when building a Mercurial ui object with configuration, don't use 'None' for NULL values

If Ui had an entry with
ui_section='extensions', ui_key='largefiles', ui_value=NULL
it would be passed to Mercurial as if the .ini file had
[extensions]
largefiles = None
and it would fail to load the largefiles extension because it couldn't find
'./None/'.

Note: get_current_revision might currently mask this problem. It will not get
the Ui from the database and will thus read a normal .ini file from the system
and (if configured) read the largefiles extension from the default location.
That will make the largefiles extension available for later largefiles imports
even if they specify the bogus path.

As f8a714c2c5a1 noted in a FIXME: ui_value should perhaps not be nullable.

For now, just handle NULL in extension configuration.

celeryd: move the celeryd paster command to the place where paster commands live

Also, partial implementations of other Celery functionality is dropped.

utils: rename add_cache to setup_cache_regions

Rename method 'add_cache' to something that better describes what it
actually does.

utils: group cache-related methods

Note: method itself is unchanged

hg: drop pointless push_ssl configuration setting - if there is a risk push can be compromised, credentials can also easily be stolen for pull

Everybody should have a ssl-only setup now. Alternatively, there is a use case
for 'only anonymous traffic on ssl - all authentication and authenticated
traffic must be on ssl'. That can be done with proper web server configuration.

paster: split paster specifics out of kallithea.lib.utils

BasePasterCommand and ask_ok are only useful in a Paster/command-line
context, and can thus be removed from the already overly cluttered main
utils module.

(The new common.py has been added to Mercurial as a copy of utils.py,
preserving its file history, but creating a somewhat bewildering diff.)

repo-scan: rewrite get_filesystem_repos to use os.walk instead of stupid recursion

I think this is more readable. It is also faster. Perhaps because the more
readable implementation makes it easier to optimize.