Add JSON-RPC API for reading changeset status

Allows reading a changeset status as json. This is useful for e.g. reporting,
extracting review status for all changesets between two releases.

repositories: make sure repositories not only differ in casing

Repositories only differing in case cause problems:
* it can't be stored on case insensitive filesystems (Windows and MacOS)
* some databases can't easily handle case sensitive queries
* users will most certainly be confused by names that only differ in case

We will keep trying to be case sensitive on systems that can ... but on some
systems wrong casings might work. We don't care.

The validators are changed to prevent mixed case repo and repo group names.

Repository sensitivity tests are removed, and insensitivity tests are added
instead.

auth: refactor auth plugin importing

The list of authentication plugins, configured in the database, may contain
plugins which are no longer available. Therefore directly import the plugin in
get_auth_plugins and report the ImportError to the log instead of
breaking kallithea completely.

Patch modified by Mads Kiilerich.

tg: minimize future diff by some mocking and replacing some pylons imports with tg

No actual tg dependency yet, just a temporary hack faking tg as an alias for
pylons.

Based on work by Alessandro Molina.

cleanup: refer less to User.DEFAULT_USER

Down the road we might want to identify the default user in another
way than by username.

cleanup: use list comprehensions

It's often the same number of lines, but avoids introducing a needless
"result" variable, and makes the item expression stand out more clearly.

It's also a tiny bit faster, but the readability gains is what matters.

model: refactor listing of commenters

The ChangesetComment.get_users method was only called in two places,
with each call triggering independent codepaths inside the method.

model: simplify guess_instance invocations

The second argument to the guess_instance classmethod is in practice
fixed for Gist, Permission, RepoGroup, Repository, User, UserGroup;
so might as well move this logic into per-class specializations of
guess_instance.

repo_group: more flexible handling of trailing '/' in repository group names

Don't fail on /_admin/repo_groups/REPO_GROUP_NAME/ .

db: it should be "PullRequestReviewer" (singular)

sed -i 's/\bPullRequestReviewers\b/PullRequestReviewer/g' $(hg files)

Just as it's "User", "PullRequest", etc. It's not a collection.

db: rename Gist.gist_owner to Gist.owner_id

For consistency with the existing "owner" relationship.

db: rename ChangesetStatus.changeset_comment_id to comment_id

This ensures consistency with the existing "comment" relationship, as
well as the foreign key (ChangesetComment.comment_id).

db: rename UserFollowing.follows_repo_id to follows_repository_id

Since the relationship is 'follows_repository', rename the column to be
'follows_repository_id', not 'follows_repo_id'. This also makes the
Python column name match the actual database column name.

(The inconsistency dates back to early RhodeCode days.)

db: rename RepoGroup.group_parent_id to parent_group_id

Also for consistency with the existing parent_group relationship.

model: rename db.BaseModel to avoid confusion with base.BaseModel

There exist two BaseModel classes in kallithea.model:
- kallithea.model.base.BaseModel (previously kallithea.model.BaseModel)
- kallithea.model.db.BaseModel

Needless to say, this is very confusing, so rename one of them.

model: change ChangesetComment 'user' to 'author'

Rename the 'user_id' field to 'author_id' and replace other references
to the comment 'user' throughout the model. The database column name
'user_id' remain unchanged for now; a later Alembic script can fix the
name of these and other columns to match their Python name.

db: PullRequest/Repository/RepoGroup/UserGroup: change 'user' to 'owner'

Rename the 'user' and 'user_id' fields on the four classes to something
more informative. The database column names remain unchanged for now;
a later Alembic script can fix the name of these and other columns to
match their Python name.

This might break rcextensions, though, and external scripts that use
the HTML form interface.

db: remove superfluous Session.add calls

Don't re-add objects to the SQLAlchemy Session just because they were
modified. Session.add is only for freshly constructed objects that
SQLAlchemy doesn't know about yet.

The rules are quite simple:

When creating a database object by calling the constructor directly, it
must explicitly be added to the session.

When creating an object using a factory function (like "create_repo"),
the returned object has already (by convention) been added to the
session, and should not be added again.

When getting an object from the session (via Session.query or any of the
utility functions that look up objects in the database), it's already
added, and should not be added again. SQLAlchemy notices attribute
modifications automatically for all objects it knows about.

db: always add to session in Setting.create_or_update

There's no use case for not adding the newly created Setting to the
SQLAlchemy session (and thus, once we commit, the database).

With this change, all the various "create" model methods add their
result to the session.

db: do case-insensitive explicit sorting of RepoGroup names

This does not change the implicit sorting enabled via __mapper_args__,
which is a bad idea anyway (and now deprecated), and will have to be
dealt with in a later changeset.

The use of __mapper_args__ implicitly adds sorting to every query of
RepoGroup objects throughout the code (including implicit queries via
relationships). For the relationships, __mapper_args can be replaced
with "order_by" on each individual relationship, and it's reasonably
straight-forward to identify every RepoGroup query throughout the code,
and add explicit sorting. But we don't really need that sorting most
of the time, so a better way forward may be to identify all the places
that actually needs the sorting, make it explicit there, and then kill
the __mapper_args__. (Anyway, future work.)

db: always do case-insensitive sorting of repository names

We retain the implicit order_by on the follows_repository relationship.
This is probably a bad idea, since it causes sorting even when it's not
needed; but for now, at least, we consistently do case-insensitive sort.

db: add some PullRequest.query() shortcuts

This makes database query code more explicit and increases readability.

E.g. the function name get_pullrequest_cnt_for_user was bad, because the
concept of "pullrequest for user" is incredibly vague, and could refer
to any kind of association between PRs and users. (Quiz time! Does it
mean that the user is the PR owner, that the user is reviewing, or that
the user has commented on the PR and thus is receiving notifications?)

A descriptive name could be "get_open_pull_request_count_for_reviewer",
because the function is indeed only concerned with reviewers and only
with open pull requests. But at this point, we might as well say
PullRequest.query(reviewer_id=user, include_closed=False).count()
which is only slightly longer, and doesn't require us to write dozens
of little wrapper functions (including, any moment now, a separate
function for listing the PRs instead of counting them).

Note that we're not actually going down an abstraction level by doing
this. We're still operating on the concepts of "pull request", "open"
and "reviewer", and are not leaking database implementation details.

The query() shortcuts are designed so they default to not altering
the query. Any processing requires explicit opt-in by the caller.

db: inline calls to get_all

This method saves basically no typing, compared to "query().all()".
Additionally, "all()" returns a list, forcing all records to be loaded
into a memory at the same time, but some callers just need to iterate
over the objects one at a time, in which case "query()" alone is more
efficient. In one case, the caller can even use "count()" and avoid
loading any objects from the database at all.

auth: cleanup of EXTERN_TYPE_INTERNAL

Don't set it in top level namespace - it is a weak link between the database
and the actual implementation. Don't make it more than that.

Don't hardcode in that many places that 'internal' is the default - just call
it DEFAULT_AUTH_TYPE.

Don't use it for extern_name - it is only intended for use as extern_type.

Remove unused uses.

model: refactor and simplify _get_instance

_get_instance is a BaseModel method, but never uses "self". Instead
it takes a class argument, which indicates that it's better suited as
a classmethod on said classes.

Also rename to something more descriptive, remove leading underscore
since it's not a private API, and refactor for readability.

db: remove deprecated getAll method

Not sure why this wasn't done immediately when get_all was added. Maybe
search and replace was out of order at the time.

db: use consistent key/index/constraint names when creating database

This simplifies future Alembic upgrades/downgrades.
http://alembic.zzzcomputing.com/en/latest/naming.html

A future changeset will update old databases to match this naming
convention.

Bump SQLAlchemy minimum version to 1.0, required for naming_convention
support.

db: remove SQLAlchemy Migrate database table

Use Alembic to migrate database away from SQLAlchemy Migrate. This
eliminates the last vestiges of SQLAlchemy Migrate. Since we drop
the Migrate table (and its contents), it is not possible to revert
this database change; however, downgrading to this Alembic schema
revision (9358dc3d6828) will in practice be enough to ensure
compatibility with all previous Kallithea versions.

(As always, the Alembic migration script is committed in the same
revision as the database schema changes.)

db: drop RhodeCode compatibility (database rebranding etc.)

To migrate from RhodeCode, users will have to first migrate to
Kallithea 0.3.2.

db: remove redundant unique constraint for repository groups

There's already a unique constraint on 'group_name' alone, no need for
one on the combination of 'group_name' and 'group_parent_id'. (The extra
constraint likely stems from confusion over what exactly goes into
group_name; add comment to clarify that it is the full group path.)

db: remove redundant unique constraints from primary keys

Primary keys are always unique, both in the SQL standard and in SQLite,
MySQL and PostgreSQL.

Setting unique=True cases SQLAlchemy to explicitly add additional,
redundant UNIQUE indexes to the columns, which is at best needless
metadata overhead, and at worst causes the database engine to waste
time maintaining an extra index that serves no purpose.

db: drop length limits on TEXT columns

As of the upgrade to version 1.0 in 3c4b6ddf6735, SQLAlchemy began to
pass TEXT length limits to the database during table creation. Such
limits are however not supported by SQLite, MySQL nor PostgreSQL, and
while SQLite simply ignores it, it is a syntax error in PostgreSQL,
breaking the creation of new PostgreSQL databases. With the lengths
being unused and quite arbitrary, just drop them.

db: ensure git hooks work when the repositories base path is a symlink

When a Git hook starts, it thinks its repo_path is its cwd.
However, if the repositories base path is a symlink to a different
location, base path won't match the location of the repository
where the symlink will be resolved.

tests: introduce tests and reference dump for notification mails

The mails are dumped to a tracked html file:
* changes shows up as diffs and are easy to spot and review
* all mails can easily can be investigated in a browser and checked for content
and consistency

The tests are mocking canonical_url because it has deep dependencies to
pylons.url which requires (thread local?) environment setup that the tests
doesn't have.

pullrequests: reword the grumpy "Rejected" to the more neutral and factual "Not approved"

"Rejecting" could be considered rude and some reviewers were thus reluctant to
use it. Rewording it might give reviewers a voice to better express their
feelings.

status: refactor internal dict for mapping status string to description

Slightly better performance ... but more importantly: do it "the right way".

db: cache SCM instance short-term (tied to SQLAlchemy session lifetime)

Repeatedly checking whether SCM instances are invalidated is slow, and
we don't actually _want_ SCM instances to invalidate half-way through
a request either.

Therefore cache them in on the db.Repository object, the lifetime
of which is directly tied to the lifetime of the SQLAlchemy session,
the lifetime of which is tied directly to the individual HTTP request.
This way, we only check for invalidation the first time the SCM instance
is accessed in a request.

This will improve performance in cases where we have (by definition) badly
written code that retrieves repo objects several times.

db: get rid of vcs_full_cache - it should always be used

It might make tests less deterministic, but it makes sure that we test what we actually use.

db: name the scm_instance_cached cache entries - reduce the risk of collisions

- and same with other cache regions

db: inline repo __get_instance to scm_instance_no_cache

Having two names for the same thing just made it look even more complicated
than it is.

There _should_ not be any reason for calling scm_instance_no_cache directly
anywhere ... but for now that is how it is.

pullrequests: optimize iteration over reviewers - avoid fetching users one by one

.reviewers was mainly used for iteration and then dereferencing .user one by
one. That gave lots of queries and round trips to the database and was slow.

Instead, do something else. Either query directly or use a new method for
getting the list of reviewer users.

Reviewers will explicitly be shown in the order they have been added (assuming
database id's are monotonic).

vcs: the git backend requires string encoded repository names too

Shown by test.functional.test_admin_repos.test_delete_non_ascii on Windows.

summary, changelog: add reviewer username to status change tooltips

Change the status change (circle) icon tooltips from "Changeset status:
NEW_STATE" to "Changeset status: NEWSTATE by REVIEWER_USERNAME".

This makes it much faster to check who is reviewing what and who approved or
rejected what.

db: match case-insensitively using func.lower, not ilike

ilike() uses SQL ILIKE operator internally, which means it interprets
'%' and '_' in the match pattern as wildcards. Instead of ilike(), it's
better to turn both operands to the lower case and compare them.

This also unbreaks the test case introduced in 13d0fe6f751a.

db: always match user emails case insensitively

This commit removes case-sensitive email matching.
It also adds a couple of tests which fail, to demonstrate a defect in the
current implementation (using ILIKE matching instead of case-insensitive
equality comparison).

db: fix Ui.get_by_key to also filter on section

Key alone is not enough to identify a Ui row.

comments: limit comment deletion to a 5 minutes grace period

It is really a bad idea to delete a comment after the content and references to
it has been sent in mails.

Ultimately, comments should be editable and versioned - "delete" should then
just hide the comment.

Note: this is just done at the UI level - not verified on the server

db: make sure all (non-primary) columns have nullable set explicitly

The default of nullable=True is rarely good for us so nullable should always be
specified unless there is a reason to allow nullable ... and if the default is
fine, xplicit is better than implicit.

The declared nulliness of some fields are changed where it seems like code
already enforced it.

Some fields are marked as FIXME when they need (trivial?) data conversion to
convert NULLs to default values.

db: stop messing with convert_unicode in .ini and db scheme

convert_unicode=False is the default for String fields ... except when the .ini file contains sets it to True.

String fields are however no longer used for unicode and we can drop convert_unicode everywhere.

See
http://docs.sqlalchemy.org/en/latest/core/type_basics.html#sqlalchemy.types.String.params.convert_unicode
for the advice of never setting convert_unicode.

db: change unicode fields to have Unicode type instead of String

Drop convert_unicode for Unicode fields where it doesn't apply; these fields
always use unicode in the database and returns unicode.

See
http://docs.sqlalchemy.org/en/latest/core/type_basics.html#sqlalchemy.types.String.params.convert_unicode
for the advice of using Unicode for unicode fields.

db: avoid specifying uniqueness constraints twice

Having both UniqueConstraint for a column on a table and unique=True on the
same column is both wrong, redundant, deprecated and cause warnings.

db: introduce _table_args_default_dict with dict values for __table_args__

DRY.

db: fix unknown exception type in commit error handling

efce61aac33d was a blind fix. It failed because `from sqlalchemy import *`
doesn't import exc and the new except clause would thus fail. It also failed
because the session has to be rolled back after a commit failure.

Now, rework it to fix these issues.

Note that we are able to detect whether the commit failed for valid reasons ...
but we can't use that information to much ...

notifications: mark notifications to self "pre-read"

When a user e.g. comments on its own pull request, that user receives a
notification about its own comment. This is slightly dubious behavior,
but at least brings a level of continuity to the notification history.

However, at the very least, the notification should not show as unread.

cache: when invalidating a cache, always just delete all 'live cache' records instead of marking them inactive

Keep it simple. Adding the record again might be slightly more expensive than
just updating the active flag but instead we get get a simpler model and
automatic cleanup without using the cache-keys paster command.

pull requests: call it 'owner' instead of 'author'

Prepare for making it possible to transfer ownership of PRs.

spelling: fix inappropriate use of title case

These are not titles or labels, and should not be title cased.

cleanup: remove unused imports

Found with pyflakes.

db: better handling of decoding of changeset_cache values from the database

Do some validation of the data to prevent that incorrect data cause weird
failures when they are used later on.

auth: construct AuthUser from either user_id or db.User object

If the caller already has the database User object, there's no reason
for AuthUser to look it up again.

The `api_key` lookup functionality is dropped, because 1) it's only
used in one place, and 2) it's simple enough for the caller to do the
lookup itself.

The `user_id` lookup functionality is kept, because 1) it's frequently
used, and 2) far from a simple `User.get(id)` lookup, it has a complex
interaction with UserModel. (That cleanup will have to wait for another
day.)

All calls of the form `AuthUser(user_id=x.user_id)` can be replaced with
`AuthUser(dbuser=x)`, assuming `x` is a db.User. However, verifying that
assumption requires a manual audit of every call site, since `x` might
also be another `AuthUser` object, for instance. Therefore, only the
most obvious call sites have been fixed here.

auth: remove redundant AuthUser constructor arguments

AuthUser looks up the user based on the first of `user_id`, `api_key`
and `username` that is not None *and* does not refer to the default
(anonymous) user. In practice, the arguments always refer to the same
user, so there's no point in specifying more than one of them.

cleanup: check for None object identity in cases where that is what the 'contract' says

Avoid applying bool() on complex objects - it might return something unexpected
such as the key (which might be 0 and thus be false). Checking for None is more
safe and faster.

db: cleanup of repo group choices - better handling of root and better sorting

groups_choices always get a groups list so there is no reason have code for
creating a list.

All the responsibility of creating the tuples is moved to _generate_choice -
also for the magic top level entries.

Also sort the choices by all path elements - not just the top level element.