celery: move send_email task to a better home in notification model

Avoid bundling everything from many different layers in one big task library.

This is more feasible now when we don't need kallithea.CELERY_APP set at import
time.

lib: move some template filter functions from utils2 to webutils

While quite Kallithea specific, we prefer to have these functions in webutils
where they soon can be exposed when templates don't need the whole helpers
module.

lib: move urlification to webutils

Less use of helpers in model.

notifications: only render body with mentions if there is a body

Notifications for registering a user would pass body=None. It is unfortunate to
pass None all the way to render_w_mentions - it seems fair enough that this
function expects a str.

model: remove unused 'subject' parameter of NotificationModel.create()

The subject of emails is determined with EmailNotificationModel._subj_map,
based on the notification type. The 'subject' parameter passed to
NotificationModel.create is completely unused.

Remove this parameter and update its callers, removing code that is now no
longer used.

templates: rename email_templates to email

The '_templates' suffix is not really needed as we are already in a
'templates' dir.

This is more in line with the other templates, and the new 'ini' template
directory.

comments: add changeset author in mail notification subject

In analogy to the handling of comments on pull requests, as changed in
commit d4061c6cc0e2.

notifications: explicitly add author of pull request in invitation mail

When an author of a pull request does not get feedback from any of the
reviewers, he is left wondering whether they got the invitation mail at all.
By explicitly sending the invitation also to the author, he can at least know
if e-mail works at the server side and how reviewers see it.
The mail can also be forwarded as a 'reminder'.

celery: fix send_email to work with JSON encoding (Issue #363)

Long time ago, c935bcaf7086 introduced an optional User object parameter to the
send_email task and used the computed full_name_or_username property. Due to the
magic of pickle, that also worked when using Celery to run the task async.

Now, Celery 4 changed the default encoding from Pickle to JSON, which we
anticipated in e539db6cc0da. That broke send_email in some cases, for example
when a user comments on another user's changeset.

Fixed by passing the "From" name as string instead of passing the whole User
object.

Thanks to vyom for reporting.

py3: drop .keys when we don't need them

In python 3 they will be iterators and mostly useless ... but they are already
mostly redundant in py2.

model: move notification types from Notification to NotificationModel

This commit is part of the removal of the UI notification feature from
Kallithea, which is not deemed useful in its current form. Only email
notifications are preserved.

As there is no database storage of notifications anymore, the Notification
class will be removed. However, the notification type definitions are still
used for email notifications, and need to live somewhere. As creating
notifications is always passing via NotificationModel, it makes sense to
move the types there.

model: remove UI notification feature

This commit is part of the removal of the UI notification feature from
Kallithea, which is not deemed useful in its current form. Only email
notifications are preserved.

model: notification: don't round-trip via list if you want a set

'create' in NotificationModel starts from an empty list, appends entries to
it, then converts it to a set.
You could equally start with a set and add to it, avoiding the final
conversion.

notification: don't repeat common actions for each email recipient

The emails sent upon PR creation or comments are identical for all
recipients, except for the 'To:' header added by the send_email method.
Calculation of subject, body, etc. can thus be moved outside of the
loop.

Move the manipulation of the recipient list down to where it is used, for
clarity.

HTML email templates: restrict amount of visible hyperlinks

Users report that some HTML email templates are confusing due to the large
number of hyperlinks in them. They typically click the first thing they see
that looks clickable without actually reading what they are clicking on,
expecting that it all points to the same thing.

In pullrequest invitations, the pullrequest and changeset URLs should be
clickable, but the source and target repository should not: this is not what
you typically want to see when taking action on this mail.

Similarly in comment emails, the corresponding repository should not be
clickable.

Unfortunately, even if we remove the <a> hyperlink tags on these repository
URLs, most mail clients will still recognize the text as being URLs
and make them clickable anyway. Worse, they will do so in a different
styling than the other links.

Therefore, do add <a> tags in the HTML, but make use of styling like that of
data_style, i.e. regular text color, grey background with dark grey border.
The links _will_ thus be clickable but it will not be as inviting as before,
thus removing confusion.

HTML email templates: don't use link color for non-links

The link color should only be used for actual hyperlinks.
Using it for anything else is very confusing for users.

For text like repository names, branch names, etc. introduce a new style
'data_style' that is normal text with a grey background and border.

HTML email templates: introduce link_style variable

Avoid repetition inside the templates and simplify future refactoring by
using 'style' variables provided by the model.

codingstyle: trivial whitespace fixes

Reported by flake8.

email: don't crash on sending mails with unicode comments without appropriate environment configuration (Issue #275)

For example, on Linux, running `gearbox serve` with LANG=C, would crash in:

File ".../kallithea/lib/celerylib/tasks.py", line 307, in send_email
% (' '.join(recipients), headers, subject, body, html_body))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 224: ordinal not in range(128)

Replacing render with render_unicode works in this case ... but there might be
other problems elsewhere.

email: add X-Kallithea-Notification-Type header

To simplify email filtering, add a header indicating the type of email being
sent. The 'type_' value corresponds to one of the types defined in class
Notification in kallithea/model/db.py, e.g. 'cs_comment', 'pull_request',
'pull_request_comment', ...

email: don't crash on sending mails with unicode comments without appropriate environment configuration (Issue #275)

For example, on Linux, running `gearbox serve` with LANG=C, would crash in:

File ".../kallithea/lib/celerylib/tasks.py", line 307, in send_email
% (' '.join(recipients), headers, subject, body, html_body))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 224: ordinal not in range(128)

Replacing render with render_unicode works in this case ... but there might be
other problems elsewhere.

backend: replace Pylons with TurboGears2

Replace the no-longer-supported Pylons application framework by TurboGears2
which is largely compatible/similar to Pylons.
Some interesting history is described at:
https://en.wikipedia.org/wiki/TurboGears

Changes by Dominik Ruf:
- fix sql config in test.ini

Changes by Thomas De Schampheleire:
- set-up of test suite
- tests: 'fix' repo archival test failure
Between Pylons and TurboGears2, there seems to be a small difference in the
headers sent for repository archive files, related to character encoding.
It is assumed that this difference is not important, and that the test
should just align with reality.
- remove need to import helpers/app_globals in lib
TurboGears2 by default expects helpers and app_globals to be available
in lib. For this reason kallithea/lib/__init__.py was originally changed
to include those files. However, this triggered several types of
circular import problems. If module A imported something from lib (e.g.
lib.annotate), and lib.helpers imported (possibly indirectly) module A,
then there was a circular import. Fix this by overruling the relevant
method of tg AppConfig, which is also hinted in the TurboGears2 code.
Hereby, the include of something from lib does not automatically import
helpers, greatly reducing the chances of circular import problems.
- make sure HTTP error '400' uses the custom error pages
TurboGears2 does not by default handle HTTP status code
'400 (Bad Request)' via the custom error page handling, causing a
standard non-styled error page.
- disable transaction manager
Kallithea currently handles its own transactions and does not need the
TurboGears2 transaction manager. However, TurboGears2 tries to enable it
by default and fails, throwing an error during application initialization.
The error itself seemed to be harmless for normal application functioning,
but was nevertheless confusing.
- add backlash as required dependency: backlash is meant as the WebError
replacement in TurboGears2 (originally WebError is part of Pylons). When
debug==true, it provides an interactive debugger in the browser. When
debug==false, backlash is necessary to show backtraces on the console.
- misc fixes

tg: minimize future diff by some mocking and replacing some pylons imports with tg

No actual tg dependency yet, just a temporary hack faking tg as an alias for
pylons.

Based on work by Alessandro Molina.

model: remove BaseModel class

The only remaining purpose of this class was to provide the "sa" field,
allowing a custom SQLAlchemy session to be used for model operations.
However, this never actually worked, nor was it used anywhere.

There's always a global SQLAlchemy session associated with the current
thread; using another session for a single function call does not make
any sense (as sessions cannot be mixed), unless the code works carefully
to ensure the two sessions (and all objects loaded from them) are kept
completely separate. Suffice to say that Kallithea does no such thing,
thus there's no need to pretend to support multiple concurrent sessions.

mails: make title in header boxes slightly bolder

In my opinion, the top headers in the new mail styling seemed a bit "thin"
compared to the other headers.

email: add X-Kallithea-Notification-Type header

To simplify email filtering, add a header indicating the type of email being
sent. The 'type_' value corresponds to one of the types defined in class
Notification in kallithea/model/db.py, e.g. 'cs_comment', 'pull_request',
'pull_request_comment', ...

model: move code from __init__.py to base.py

Having too much code, in particular too much imports, inside a package's
__init__.py is a recipe for circular imports, and considered bad practice in
Python [1]

Move out everything from kallithea/model/__init__.py to a new file
kallithea/model/base.py and adapt the existing imports.

[1] http://docs.python-guide.org/en/latest/writing/structure/#packages

db: remove superfluous Session.add calls

Don't re-add objects to the SQLAlchemy Session just because they were
modified. Session.add is only for freshly constructed objects that
SQLAlchemy doesn't know about yet.

The rules are quite simple:

When creating a database object by calling the constructor directly, it
must explicitly be added to the session.

When creating an object using a factory function (like "create_repo"),
the returned object has already (by convention) been added to the
session, and should not be added again.

When getting an object from the session (via Session.query or any of the
utility functions that look up objects in the database), it's already
added, and should not be added again. SQLAlchemy notices attribute
modifications automatically for all objects it knows about.

celeryd: annotate tasks so they can be run directly without run_task

This also makes the system less forgiving about celery configuration problems
and thus easier to debug. I like that.

notifications: new email templates with html styling

- with heavy interfering and hacking by Mads Kiilerich.

model: refactor and simplify _get_instance

_get_instance is a BaseModel method, but never uses "self". Instead
it takes a class argument, which indicates that it's better suited as
a classmethod on said classes.

Also rename to something more descriptive, remove leading underscore
since it's not a private API, and refactor for readability.

model: drop BaseModel.cls

It no longer serves a purpose.

model: inline superfluous "helper" methods

__get_changeset_status wasn't even used and has simply been removed.

Notification.__get_notification reimplemented BaseModel._get_instance,
warts and all, and has been replaced by calls to that method.

The rest were just needless indirection, and have been inlined.

notifications: tweak PR mail subject lines

The subject line is used for mail threading in gmail and can thus not be
changed without impacting users ... but now we do it.

* The tag '[Review]' is more spot-on than '[Added]'.
* The subject should be short so it fits on one line, so abbreviate "pull
request" to PR.
* Add the PR owner - convenient for filtering comments on own PRs from comments
on other PRs.

email: add relevant title to subject of emails

For comment and pullrequest mails, add changeset/pullrequest title (shortened
to first 50 characters of first line)

notifications: make more template strings available for mails

To be used soon ...

notifications: improve response time when number of notifications is large

NotificationsController always retrieved materialized list of all notifications
in database, even to display only 10 of them. This is improved by feeding
SQLAlchemy Query object directly to webhelpers.paginate.Page, avoiding eager
load of all notifications.

comments: linkify revision hashes and issue tracker references

This was hard with RST markup - much easier to do reasonably safely in plain text.

comments: change comment formatting to plain text instead of rst

There might be value in enabling rich markup (especially if it is markdown
instead of rst) ... or it might be a waste of time. We might revisit that
later.

But either way: Changing to plain text makes it more feasible to do markup of
hashes and issues - for now that is more important than rtf.

email: remove user from subject of emails

For comment and pullrequest mails:
- remove user that triggered the email, as this is now in the From header.

Based on code by Cedric De Herdt.

email: send comment and pullrequest mails with the author's name in 'From'

When emails are sent for comments and pullrequest invitations, set the From
header to:
Author's Name (no-reply) <generic email address>

Using the name of the person that causes the email, makes the emails more
useful and interpretable for the recipient of the emails.
To avoid replies directly to the author, triggering an 'offline' email
discussion that is not visible in the Kallithea interface, a generic
'no-reply' email address is used instead of the author's email
address.
This approach is assumed to be accepted by spam filters, as several other
web services are using the same approach.

The sender used for other email types, e.g. password reset mails, is
untouched and remains the value configured in app_email_from.

The sender used for the SMTP envelope is untouched as well.

Based on code by Cedric De Herdt.

cleanup: consistently use 'except ... as ...:'

Use the Python 2.6+ syntax instead of the old confusing 'except ..., ...'
syntax.

cleanup: check for None object identity in cases where that is what the 'contract' says

Avoid applying bool() on complex objects - it might return something unexpected
such as the key (which might be 0 and thus be false). Checking for None is more
safe and faster.

notifications: use different strings for descriptions with age and datetime

Improves both English wording and makes it easier to translate correctly.

notifications: use stupid mail static-except-[] subjects to please gmail and its broken threading

This gives reasonable threading, both with gmail and proper mail clients, at
the cost of making the email subjects slightly obscure.

Gmail is clueless - also about mail threading, ignoring our References headers
and the wisdom of jwz. Gmail will start a new thread for each subject. The only
way to make all mails related to the same PR belong to the same thread is thus
to let all mails for a PR have the same subject. Gmail will however ignore
content in square brackets at the beginning of the subject so we can put the
interesting parts there.

pullrequests: add PullRequest.nice_id method

Currently, a pull request id is referenced as #5, and the '#' symbol is
fixed and repeated in several places. This commit adds a class method
make_nice_id, that returns a string reference to a pull request,
currently in the form '#5'.

This function could be overridden by an organization if they need references
in another form, for example PR-5.

The method nice_id uses this for the pullrequest's own id.

notification: handle unicode translations (Issue #95)

Would fail with localization after 8e1491a7e3af.

email templates: send text/plain part as well

This change adds text parts to the email templates, as
HTML and text templates may be way to different to be
handled automatically. Also, use proper dash-dash-space
signature separator, so the email clients recognise it
for sure.

notifications: insert 'References' mail headers to help MUA threading

gmail is however fubar.

Correct licensing information in individual files.

The top-level license file is now LICENSE.md.

Also, in various places where there should have been joint copyright holders
listed, a single copyright holder was listed. It does not appear easy to add
a link to a large list of copyright holders in these places, so it simply
refers to the fact that various authors hold copyright.

In future, if an easy method is discovered to link to a list from those
places, we should do so.

Finally, text is added to LICENSE.md to point to where the full list of
copyright holders is, and that Kallithea as a whole is GPLv3'd.

Second step in two-part process to rename directories.
This is the actual directory rename.