api: fix get_changeset() when incomplete raw_id is passed with with_reviews

Previously, ChangesetStatusModel was queried with the raw_id passed as an
argument to the API function. When the raw_id was incomplete (i.e. shortened),
no reviews were found. Using the full raw_id from the changeset instance fixes
that.

Someone might argue that the caller is supposed to pass a full raw_id to the
API function. However, in any case, the return value should not be incomplete
without notice.

api docs: add placeholder docstrings where docstrings are missing

They will hopefully soon be replaced by actual API documentation.

api docstrings: drop "ERROR OUTPUT" from docstring

It is not so valuable, and not included in the user doc.

repo group: fix API for updating parent

With other bugs out of the way, the API implementation just has to pass the new
group id correctly.

Note that - as in many other places - moving things to the top level doesn't
work.

Original patch has been modified by Mads Kiilerich.

repo: introduce enable_downloads and enable_statistics when creating repos

These booleans were not shown in the normal repo creation form, so the form
validation applied the "default" values of False. These values were however not
used by the model when creating repos - it just unconditionally used the real
global defaults.

The API already exposed some of this, but it wasn't implemented.

The web form for creating repos lacked these fields, but it was present in the
repo edit form. Just make these fields mandatory. There will thus not be any
defaults to apply in the model for creating repos.

api docs: more consistent use of space around ":" in examples

The examples are informal pseudo code - not json and not python. But let's try
to be consistent anyway.

api docs: update docstrings

Changes contributed by toras <toras9000@gmail.com>.

celery: drop tracking of task_id - we use ignore_result=True and will never get anything back

There is thus no need for configuration of celery.result_backend .

The alternative would be to fix it. That could give better error reporting from
failing repo creations, but would require quite a bit of additional changes
before it actually works reliably.

api: fix update_repo check for permission to create top level repos

The hg.create.repository permission only apply to creation (or renaming) of top
level repos - it is not relevant for other kinds of renaming.

Moving or renaming repos in other locations is now covered by other checks.

api: fix repo group permission check for repo creation

hg.create.repository is only controlling whether all users have write access at
top level. For other repo locations, check for write permission to the repo
group.

Note: This also covers "repo creation" by forking or by moving another repo.

api: fix repo creation inside repo groups when using Celery

Repo creation would fail when Celery tried serializing a whole RepoGroup
object. It only worked when not using Celery.

api: don't pass owner to create_repo in form_data

create_repo gets owner as 'cur_user', but 'owner' in form_data was unused.

api: fix repo creation from API when using celery

Repo creation would fail when celery tried serializing a whole User object. It
only worked when not using celery.

cleanup: get rid of redundant assignments 'foo = foo'

Redundant assignments were searched with:

grep '^ *\(.*\) = \1$' `hg files`

Some of these cases have resulted from earlier cleanup, like:

foo = safe_unicode(foo)

and later batch removing safe_unicode usage. See e.g. commit
e35373106528b29461b7f5144780cb9ec79f765d .


Note: two false positives occur:

kallithea/tests/models/test_settings.py:39:
setting.app_settings_value = setting.app_settings_value
kallithea/tests/models/test_settings.py:42:
setting.app_settings_value = setting.app_settings_value

which is not a plain assignment but where 'app_settings_value' is a property
method with side effects.

api: stop using 'Optional', 'OAttr'/'OptionalAttr' classes

There does not seem to be a good reason to use the 'Optional' and
'OptionalAttr' classes.
It makes the code harder to understand. And worse, the 'default value'
specified is not always used, which can thus give false information to
users.

The way Optional was used in the API calls is twofold:

1.either by effectively extracting a value, via Optional.extract(param).
If 'param' was indeed specified by the user, then this would yield that
user-specified value. Otherwise, it would yield the value declared in the
parameter declaration, e.g. param=Optional(defaultvalue).

2.or by checking if a variable is an instance of the Optional class. In case
a user effectively passed a value, this value will not be of the
Optional class. So if a parameter is an object of class Optional, we know
the user did not pass a value, and we can apply some default.

In the declaration of the parameter, the specified default value will only
be used if the 'extract' method is used, i.e. method 1 above.

A simpler way to address this problem of default values is just with Python
default values, using 'None' as magic value if the default will be
calculated inside the method.

The docstrings still specify something like:
type: Optional(bool)
which is humanly readable and does not necessarily refer to a class called
'Optional', so such strings are kept.

api: new method: edit_reviewers

Allow adding and removing reviewers of a pull request with the API call
'edit_reviewers', taking a pull request ID and a list of usernames or
userids to add and remove. For single-user operation, a string can be used
instead of a list.

Note that the 'kallithea-api' tool only accepts strings, so can only perform
single-user operations. Use python 'requests', 'curl' or similar instead if
you need to operate on multiple users at a time.

scripts: drop isort --wrap-length 160 - it is broken with py3 and not really necessary

Under Python 3, isort 4.3.21 failed with
https://github.com/timothycrosley/isort/issues/741 on
kallithea/controllers/api/api.py :

Traceback (most recent call last):
File "data/env/bin/isort", line 10, in <module>
sys.exit(main())
File ".../env/lib64/python3.7/site-packages/isort/main.py", line 379, in main
for sort_attempt in attempt_iterator:
File ".../env/lib64/python3.7/site-packages/isort/main.py", line 377, in <genexpr>
attempt_iterator = (sort_imports(file_name, **arguments) for file_name in file_names)
File ".../env/lib64/python3.7/site-packages/isort/main.py", line 88, in sort_imports
result = SortImports(file_name, **arguments)
File ".../env/lib64/python3.7/site-packages/isort/isort.py", line 207, in __init__
self._add_formatted_imports()
File ".../env/lib64/python3.7/site-packages/isort/isort.py", line 606, in _add_formatted_imports
self._add_from_imports(from_modules, section, section_output, sort_ignore_case)
File ".../env/lib64/python3.7/site-packages/isort/isort.py", line 526, in _add_from_imports
import_statement = self._multi_line_reformat(import_start, from_import_section, comments)
File ".../env/lib64/python3.7/site-packages/isort/isort.py", line 552, in _multi_line_reformat
dynamic_indent, indent, line_length, comments)
File ".../env/lib64/python3.7/site-packages/isort/isort.py", line 705, in _output_grid
if len(next_statement.split(self.line_separator)[-1]) + 1 > line_length:
TypeError: '>' not supported between instances of 'int' and 'str'

py3: use exception .args instead of .message

Args seems slightly more fragile and *could* introduce problems for trivial use
if args is empty. But .message is gone.

tests: fix ordering of options of "ret_type must be one of ..." returned from api

Py3 would fail becauase it use another ordering.

flake8: fix some E712 comparison to True should be 'if cond is True:' or 'if cond:'

add_user_to_group is really odd ...

Note that the SqlAlchemy query API cause a lot of this kind of warnings.

scm: don't try to get IP address from web request in model

Remove a layering violation and make functions more reusable when they no
longer depend on global state.

At this level, the IP address (and information about the current user) is only
used for hooks logging push / pull operations. Arguably, IP address logging
only belongs in an HTTP access log, not in the log of push/pull operations. But
as long as we have IP addresses in the logs, we have to provide it. The (good?)
alternative would be to drop IP address from the push / pull logs ...

locking: drop the pull-to-lock / push-to-unlock functionality

The feature is not worth the maintenance cost. The locking is too coarse and
unflexible with insufficient UI and UX. The implementation is also quite
invasive in tricky areas of the code, and thus high maintenance.
Dropping this will enable other cleanup ... or at least make it easier.

repos: only allow api repo creation in existing groups

Fix problem with '../something' paths being allowed; '..' will always exist and
can't be created.

This also introduce a small API change: Repository groups must now exist before
repositories can be created. This makes the API more explicit and simpler.

This issue was found and reported by
Kacper Szurek
https://security.szurek.pl/

repos: only allow api repo creation in existing groups

Fix problem with '../something' paths being allowed; '..' will always exist and
can't be created.

This also introduce a small API change: Repository groups must now exist before
repositories can be created. This makes the API more explicit and simpler.

This issue was found and reported by
Kacper Szurek
https://security.szurek.pl/

api: add max_revisions option to get_changesets

The returning JSON can become pretty big and hard to parse.
Therefore add an option that allows a client to request the changesets in
smaller chuncks.

api: allow pulling from a custom remote

The 'pull' API call would currently pull from the configured repository
remote or the fork origin in case of a fork.

This commit allows to specify an optional 'clone_uri' parameter to the API
call that will be used during the pull.

sqlalchemy: fix warnings from running the test suite

Mainly warnings about strings being passed where unicode was expected.

api: add get_pullrequest and comment_pullrequest methods

Modified by Mads Kiilerich, mainly to let the test helper function
create_pullrequest use model directly.

api: add optional branches, tags and bookmarks information to get_repo API data

Modified by Mads Kiilerich to make these revision names optional - there can be
a lot of them.

Add JSON-RPC API for reading changeset status

Allows reading a changeset status as json. This is useful for e.g. reporting,
extracting review status for all changesets between two releases.

tg: minimize future diff by some mocking and replacing some pylons imports with tg

No actual tg dependency yet, just a temporary hack faking tg as an alias for
pylons.

Based on work by Alessandro Molina.

cleanup: refer less to User.DEFAULT_USER

Down the road we might want to identify the default user in another
way than by username.

cleanup: use list comprehensions

It's often the same number of lines, but avoids introducing a needless
"result" variable, and makes the item expression stand out more clearly.

It's also a tiny bit faster, but the readability gains is what matters.

auth: simplify user group permission checks

In practice, Kallithea has the 'usergroup.admin' permission imply the
'usergroup.write' permission, which again implies 'usergroup.read'.

This codifies this practice by replacing the HasUserGroupPermissionAny
"perm function" with the new HasUserGroupLevel function, reducing the
risk of errors and saving quite a lot of typing.

auth: simplify repository group permission checks

In practice, Kallithea has the 'group.admin' permission imply the
'group.write' permission, which again implies 'group.read'.

This codifies this practice by replacing HasRepoGroupPermissionAny
"perm function" with the new HasRepoGroupLevel function, reducing the
risk of errors and saving quite a lot of typing.

auth: simplify repository permission checks

In practice, Kallithea has the 'repository.admin' permission imply the
'repository.write' permission, which again implies 'repository.read'.

This codifies/enforces this practice by replacing HasRepoPermissionAny
"perm function" with the new HasRepositoryLevel function, reducing the
risk of errors and saving quite a lot of typing.

controllers: avoid setting request state in controller instances - set it in the thread global request variable

In TurboGears, controllers are singletons and we should avoid using instance
variables for any volatile data. Instead, use the "global thread local" request
context.

With everything in request, some use of c is dropped.

Note: kallithea/controllers/api/__init__.py still use instance variables that
will cause problems with TurboGears.

db: rename Gist.gist_owner to Gist.owner_id

For consistency with the existing "owner" relationship.

db: PullRequest/Repository/RepoGroup/UserGroup: change 'user' to 'owner'

Rename the 'user' and 'user_id' fields on the four classes to something
more informative. The database column names remain unchanged for now;
a later Alembic script can fix the name of these and other columns to
match their Python name.

This might break rcextensions, though, and external scripts that use
the HTML form interface.

db: inline calls to get_all

This method saves basically no typing, compared to "query().all()".
Additionally, "all()" returns a list, forcing all records to be loaded
into a memory at the same time, but some callers just need to iterate
over the objects one at a time, in which case "query()" alone is more
efficient. In one case, the caller can even use "count()" and avoid
loading any objects from the database at all.

auth: cleanup of EXTERN_TYPE_INTERNAL

Don't set it in top level namespace - it is a weak link between the database
and the actual implementation. Don't make it more than that.

Don't hardcode in that many places that 'internal' is the default - just call
it DEFAULT_AUTH_TYPE.

Don't use it for extern_name - it is only intended for use as extern_type.

Remove unused uses.

api: drop odd creation of password when extern_name is set

The check of extern_name and the interpretation of it seems odd.

Also, there is no point in setting a random password. It can't be retrieved and
used, and not setting a password is more secure.

celerylib: improve handling of sync results and get rid of BaseAsyncResult handling

A better wrapper of sync results simplifies the code.

Note: Results are currently not really used.

model: fix get_all call missed in 974d6470cbec

This would have been caught by test coverage ...

model: inline BaseModel.get_all

This is just needless indirection (and doesn't actually add any
abstraction on top of the database object), so inline all calls to it.

Don't touch PullRequestModel.get_all, though, since it is an entirely
unrelated method that just shadows the inherited get_all.

db: remove deprecated getAll method

Not sure why this wasn't done immediately when get_all was added. Maybe
search and replace was out of order at the time.

auth: remove HasPermissionAll and variants

First, find all calls to HasPermissionAll with only a single permission
given, and convert to equivalent calls to HasPermissionAny.

Next, observe that it's hard to envision situations requiring multiple
permissions (of the same scope: global/repo/repo group) to be satisfied.
Sufficiently hard that there are actually no such examples in the code.

Finally, considering that (should it ever be needed) HasPermissionAll
can be trivially built as a conjunction of HasPermissionAny calls (the
decorators, too) with only a small performance impact, simply remove
HasPermissionAll and related classes and functions.

api: fix forking of repos in repo groups (Issue #210)

Patch by Alexandre Beaulieu.

api: avoid duplicating group name when updating repo (Issue #37)

The api incorrectly passed repo.repo_name as repo_name, and the model update
function incorrectly always required repo_name.

db: always match user emails case insensitively

This commit removes case-sensitive email matching.
It also adds a couple of tests which fail, to demonstrate a defect in the
current implementation (using ILIKE matching instead of case-insensitive
equality comparison).

cleanup: make module self-naming consistent

The self naming is quite pointless ... but at least make it right.

cleanup: consistently use 'except ... as ...:'

Use the Python 2.6+ syntax instead of the old confusing 'except ..., ...'
syntax.

UserModel: remove methods that are redundant with db.User

UserModel().get_by_api_key is exactly equivalent to User.get_by_api_key.

UserModel's get_by_username and get_by_email are not exactly identical
to their db.User counterparts, due to a difference in the order of
optional arguments. Fortunately, these optional arguments are never
used.

cleanup: check for None object identity in cases where that is what the 'contract' says

Avoid applying bool() on complex objects - it might return something unexpected
such as the key (which might be 0 and thus be false). Checking for None is more
safe and faster.

api: make update_repo check permissions to check owner like create_repo does

Close loophole for reassigning repository owners.

Test by Thomas De Schampheleire.

api: check repo create permissions for update_repo and fork_repo as for create-repo

Close loophole for creating repos everywhere.

Tests by Thomas De Schampheleire.

spelling: fix various typos

This commit fixes various typos or basic English grammar mistakes found by
reviewing the kallithea.pot file.

Full correction of sentences that are not very well formulated, like missing
articles, is out of scope for this commit. Likewise for inconsistent
capitalization of strings like 'Repository group'/'Repository Group'.

Correct licensing information in individual files.

The top-level license file is now LICENSE.md.

Also, in various places where there should have been joint copyright holders
listed, a single copyright holder was listed. It does not appear easy to add
a link to a large list of copyright holders in these places, so it simply
refers to the fact that various authors hold copyright.

In future, if an easy method is discovered to link to a list from those
places, we should do so.

Finally, text is added to LICENSE.md to point to where the full list of
copyright holders is, and that Kallithea as a whole is GPLv3'd.

Second step in two-part process to rename directories.
This is the actual directory rename.