================

Kallithea README

================

About

-----

**Kallithea** is a fast and powerful management tool for Mercurial_ and Git_

with a built-in push/pull server, full text search and code-review. It works on

http/https and has a built in permission/authentication system with the ability

to authenticate via LDAP or ActiveDirectory. Kallithea also provides simple API

so it's easy to integrate with existing external systems.

Kallithea is similar in some respects to GitHub_ or Bitbucket_, however

Kallithea can be run as standalone hosted application on your own server. It is

open-source donationware and focuses more on providing a customised,

self-administered interface for Mercurial_ and Git_ repositories. Kallithea

works on Unix-like systems and Windows, and is powered by the vcs_ library

created by Łukasz Balcerzak and Marcin Kuźmiński to uniformly handle multiple

version control systems.

Kallithea was forked from RhodeCode in July 2014 and has been heavily modified.

Installation

------------

Kallithea requires Python_ 2.x and it is recommended to install it in a

virtualenv_. Official releases of Kallithea can be installed with::

    pip install kallithea

The development repository is kept very stable and used in production by the

developers -- you can do the same.

Please visit https://docs.kallithea-scm.org/en/latest/installation.html for

more details.

There is also an experimental `Puppet module`_ for installing and setting up

Kallithea. Currently, only basic functionality is provided, but it is still

enough to get up and running quickly, especially for people without Python

background. See

https://docs.kallithea-scm.org/en/latest/installation_puppet.html for further

information.

Source code

-----------

The latest sources can be obtained from

https://kallithea-scm.org/repos/kallithea.

The issue tracker and a repository mirror can be found at Bitbucket_ on

https://bitbucket.org/conservancy/kallithea.

Kallithea features

------------------

- Has its own middleware to handle Mercurial_ and Git_ protocol requests. Each

  request is authenticated and logged together with IP address.

- Built for speed and performance. You can make multiple pulls/pushes

  simultaneously. Proven to work with thousands of repositories and users.

- Supports http/https, LDAP, AD, proxy-pass authentication.

- Full permissions (private/read/write/admin) together with IP restrictions for

  each repository, additional explicit forking, repositories group and

  repository creation permissions.

- User groups for easier permission management.

- Repository groups let you group repos and manage them easier. They come with

  permission delegation features, so you can delegate groups management.

- Users can fork other users repos, and compare them at any time.

- Built-in versioned paste functionality (Gist) for sharing code snippets.

- Integrates easily with other systems, with custom created mappers you can

  connect it to almost any issue tracker, and with a JSON-RPC API you can make

  much more.

- Built-in commit API lets you add, edit and commit files right from Kallithea

  web interface using simple editor or upload binary files using simple form.

- Powerful pull request driven review system with inline commenting, changeset

  statuses, and notification system.

- Importing and syncing repositories from remote locations for Git_, Mercurial_

  and Subversion.

- Mako templates let you customize the look and feel of the application.

- Beautiful diffs, annotations and source code browsing all colored by

  pygments. Raw diffs are made in Git-diff format for both VCS systems,

  including Git_ binary-patches.

- Mercurial_ and Git_ DAG graphs and Flot-powered graphs with zooming and

  statistics to track activity for repositories.

- Admin interface with user/permission management. Admin activity journal, logs

  pulls, pushes, forks, registrations and other actions made by all users.

- Server side forks. It is possible to fork a project and modify it freely

  without breaking the main repository.

- reST and Markdown README support for repositories.

- Full text search powered by Whoosh on the source files, commit messages, and

  file names. Built-in indexing daemons, with optional incremental index build

  (no external search servers required all in one application).

- Setup project descriptions/tags and info inside built in DB for easy,

  non-filesystem operations.

- Intelligent cache with invalidation after push or project change, provides

  high performance and always up to date data.

- RSS/Atom feeds, Gravatar support, downloadable sources as zip/tar/gz.

- Optional async tasks for speed and performance using Celery_.

- Backup scripts can do backup of whole app and send it over scp to desired

  location.

- Based on Pylons, SQLAlchemy, SQLite, Whoosh, vcs.

License

-------

**Kallithea** is released under the GPLv3 license. Kallithea is a `Software

Freedom Conservancy`_ project and thus controlled by a non-profit organization.

No commercial entity can take ownership of the project and change the

direction.

Kallithea started out as an effort to make sure the existing GPLv3 codebase

would stay available under a legal license. Kallithea thus has to stay GPLv3

compatible ... but we are also happy it is GPLv3 and happy to keep it that way.

A different license (such as AGPL) could perhaps help attract a different

community with a different mix of Free Software people and companies but we are

happy with the current focus.

Community

---------

**Kallithea** is maintained by its users who contribute the fixes they would

like to see.

Get in touch with the rest of the community:

- Join the mailing list users and developers -- see

  http://lists.sfconservancy.org/mailman/listinfo/kallithea-general.

- Use IRC and join #kallithea on FreeNode (irc.freenode.net) or use

  http://webchat.freenode.net/?channels=kallithea.

- Follow Kallithea on Twitter, **@KallitheaSCM**.

- Issues can be reported at `issue tracker

  <https://bitbucket.org/conservancy/kallithea/issues>`_.

   .. note::

       Please try to read the documentation before posting any issues,

       especially the **troubleshooting section**

Online documentation

--------------------

Online documentation for the current version of Kallithea is available at

https://pythonhosted.org/Kallithea/. Documentation for the current development

version can be found on https://docs.kallithea-scm.org/.

You can also build the documentation locally: go to ``docs/`` and run::

   make html

.. note:: You need to have Sphinx_ installed to build the

          documentation. If you don't have Sphinx_ installed you can

          install it via the command: ``pip install sphinx`` .

.. _virtualenv: http://pypi.python.org/pypi/virtualenv

.. _Python: http://www.python.org/

.. _Sphinx: http://sphinx.pocoo.org/

.. _Mercurial: http://mercurial.selenic.com/

.. _Bitbucket: http://bitbucket.org/

.. _GitHub: http://github.com/

.. _Subversion: http://subversion.tigris.org/

.. _Git: http://git-scm.com/

.. _Celery: http://celeryproject.org/

.. _vcs: http://pypi.python.org/pypi/vcs

.. _Software Freedom Conservancy: http://sfconservancy.org/

.. _Puppet module: https://forge.puppetlabs.com/rauch/kallithea

.. _upgrade:

===================

Upgrading Kallithea

===================

This describes the process for upgrading Kallithea, independently of the

Kallithea installation method.

1. Stop the Kallithea web application

-------------------------------------

This step depends entirely on the web server software used to serve

Kallithea, but in any case, Kallithea should not be running during

the upgrade.

.. note::

    If you're using Celery, make sure you stop all instances during the

    upgrade.

2. Create a backup of both database and configuration

-----------------------------------------------------

You are of course strongly recommended to make backups regularly, but it

is *especially* important to make a full database and configuration

backup before performing a Kallithea upgrade.

Back up your configuration

^^^^^^^^^^^^^^^^^^^^^^^^^^

Make a copy of your Kallithea configuration (``.ini``) file.

If you are using :ref:`rcextensions <customization>`, you should also

make a copy of the entire ``rcextensions`` directory.

Back up your database

^^^^^^^^^^^^^^^^^^^^^

If using SQLite, simply make a copy of the Kallithea database (``.db``)

file.

If using PostgreSQL, please consult the documentation for the ``pg_dump``

utility.

If using MySQL, please consult the documentation for the ``mysqldump``

utility.

Look for ``sqlalchemy.db1.url`` in your configuration file to determine

database type, settings, location, etc.

3. Activate the Kallithea virtual environment (if any)

------------------------------------------------------

Verify that you are using the Python environment that you originally

installed Kallithea in by running::

    pip freeze

This will list all packages installed in the current environment. If

Kallithea isn't listed, activate the correct virtual environment.

See the appropriate installation page for details.

4. Install new version of Kallithea

-----------------------------------

Please refer to the instructions for the installation method you

originally used to install Kallithea.

If you originally installed using pip, it is as simple as::

    pip install --upgrade kallithea

If you originally installed from version control, it is as simple as::

    cd my-kallithea-clone

    hg pull -u

    pip install -e .

5. Upgrade your configuration

-----------------------------

Run the following command to upgrade your configuration (``.ini``) file::

    paster make-config Kallithea my.ini

This will display any changes made by the new version of Kallithea to your

current configuration, and attempt an automatic merge. It is recommended

that you check the contents after the merge.

.. note::

    Please always make sure your ``.ini`` files are up to date. Errors

    can often be caused by missing parameters added in new versions.

6. Upgrade your database

------------------------

To do this simply run::

    paster upgrade-db my.ini

This will upgrade the schema and update some of the defaults in the database,

and will always recheck the settings of the application, if there are no new

options that need to be set.

.. note::

    The DB schema upgrade library has some limitations and can sometimes fail if you try to

    upgrade from older major releases. In such a case simply run upgrades sequentially, e.g.,

    upgrading from 0.1.X to 0.3.X should be done like this: 0.1.X. > 0.2.X > 0.3.X.

    You can always specify what version of Kallithea you want to install e.g. using pip:

    ``pip install Kallithea==0.2``

7. Rebuild the Whoosh full-text index

-------------------------------------

It is recommended that you rebuild the Whoosh index after upgrading since

new Whoosh versions can introduce incompatible index changes.

8. Start the Kallithea web application

--------------------------------------

This step once again depends entirely on the web server software used to

serve Kallithea.

Before starting the new version of Kallithea, you may find it helpful to

clear out your log file so that new errors are readily apparent.

.. note::

    If you're using Celery, make sure you restart all instances of it after

    upgrade.

.. _virtualenv: http://pypi.python.org/pypi/virtualenv

# -*- coding: utf-8 -*-

# This program is free software: you can redistribute it and/or modify

# it under the terms of the GNU General Public License as published by

# the Free Software Foundation, either version 3 of the License, or

# (at your option) any later version.

#

# This program is distributed in the hope that it will be useful,

# but WITHOUT ANY WARRANTY; without even the implied warranty of

# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

# GNU General Public License for more details.

#

# You should have received a copy of the GNU General Public License

# along with this program.  If not, see <http://www.gnu.org/licenses/>.

"""

kallithea

~~~~~~~~~

Kallithea, a web based repository management based on pylons

versioning implementation: http://www.python.org/dev/peps/pep-0386/

This file was forked by the Kallithea project in July 2014.

Original author and date, and relevant copyright and licensing information is below:

:created_on: Apr 9, 2010

:author: marcink

:copyright: (c) 2013 RhodeCode GmbH, (C) 2014 Bradley M. Kuhn, and others.

:license: GPLv3, see LICENSE.md for more details.

"""

import sys

import platform

VERSION = (0, 3, 99)

BACKENDS = {

    'hg': 'Mercurial repository',

    'git': 'Git repository',

}

CELERY_ON = False

CELERY_EAGER = False

# link to config for pylons

CONFIG = {}

# Linked module for extensions

EXTENSIONS = {}

try:

except ImportError:

    pass

# Users.extern_type and .extern_name value for local users

# db_migrate_version.repository_id value, same as kallithea/lib/dbmigrate/migrate.cfg

try:

    from kallithea.lib import get_current_revision

    _rev = get_current_revision(quiet=True)

    if _rev and len(VERSION) > 3:

        VERSION += (_rev[0],)

except ImportError:

    pass

__version__ = ('.'.join((str(each) for each in VERSION[:3])))

__dbversion__ = 31  # defines current db version for migrations

__platform__ = platform.system()

__license__ = 'GPLv3'

__py_version__ = sys.version_info

__author__ = "Various Authors"

__url__ = 'https://kallithea-scm.org/'

is_windows = __platform__ in ['Windows']

is_unix = not is_windows

if len(VERSION) > 3:

    __version__ += '.'+VERSION[3]

    if len(VERSION) > 4:

        __version__ += VERSION[4]

    else:

        __version__ += '0'

# Hack for making the celery dependency kombu==1.5.1 compatible with Python

# 2.7.11 which has https://hg.python.org/releases/2.7.11/rev/24bdc4940e81

import uuid

if not hasattr(uuid, '_uuid_generate_random'):

    uuid._uuid_generate_random = None

# -*- coding: utf-8 -*-

# This program is free software: you can redistribute it and/or modify

# it under the terms of the GNU General Public License as published by

# the Free Software Foundation, either version 3 of the License, or

# (at your option) any later version.

#

# This program is distributed in the hope that it will be useful,

# but WITHOUT ANY WARRANTY; without even the implied warranty of

# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

# GNU General Public License for more details.

#

# You should have received a copy of the GNU General Public License

# along with this program.  If not, see <http://www.gnu.org/licenses/>.

"""

kallithea.lib.utils

~~~~~~~~~~~~~~~~~~~

Utilities library for Kallithea

This file was forked by the Kallithea project in July 2014.

Original author and date, and relevant copyright and licensing information is below:

:created_on: Apr 18, 2010

:author: marcink

:copyright: (c) 2013 RhodeCode GmbH, and others.

:license: GPLv3, see LICENSE.md for more details.

"""

import os

import re

import logging

import datetime

import traceback

import paste

import beaker

import tarfile

import shutil

import decorator

import warnings

from os.path import abspath

from os.path import dirname

from paste.script.command import Command, BadCommand

from webhelpers.text import collapse, remove_formatting, strip_tags

from beaker.cache import _cache_decorate

from kallithea.lib.vcs.utils.hgcompat import ui, config

from kallithea.lib.vcs.utils.helpers import get_scm

from kallithea.lib.vcs.exceptions import VCSError

from kallithea.model import meta

from kallithea.model.db import Repository, User, Ui, \

    UserLog, RepoGroup, Setting, UserGroup

from kallithea.model.meta import Session

from kallithea.model.repo_group import RepoGroupModel

from kallithea.lib.utils2 import safe_str, safe_unicode, get_current_authuser

from kallithea.lib.vcs.utils.fakemod import create_module

log = logging.getLogger(__name__)

REMOVED_REPO_PAT = re.compile(r'rm__\d{8}_\d{6}_\d{6}_.*')

def recursive_replace(str_, replace=' '):

    """

    Recursive replace of given sign to just one instance

    :param str_: given string

    :param replace: char to find and replace multiple instances

    Examples::

    >>> recursive_replace("Mighty---Mighty-Bo--sstones",'-')

    'Mighty-Mighty-Bo-sstones'

    """

    if str_.find(replace * 2) == -1:

        return str_

    else:

        str_ = str_.replace(replace * 2, replace)

        return recursive_replace(str_, replace)

def repo_name_slug(value):

    """

    Return slug of name of repository

    This function is called on each creation/modification

    of repository to prevent bad names in repo

    """

    slug = remove_formatting(value)

    slug = strip_tags(slug)

    for c in """`?=[]\;'"<>,/~!@#$%^&*()+{}|: """:

        slug = slug.replace(c, '-')

    slug = recursive_replace(slug, '-')

    slug = collapse(slug, '-')

    return slug

#==============================================================================

# PERM DECORATOR HELPERS FOR EXTRACTING NAMES FOR PERM CHECKS

#==============================================================================

def get_repo_slug(request):

    _repo = request.environ['pylons.routes_dict'].get('repo_name')

    if _repo:

        _repo = _repo.rstrip('/')

    return _repo

def get_repo_group_slug(request):

    _group = request.environ['pylons.routes_dict'].get('group_name')

    if _group:

        _group = _group.rstrip('/')

    return _group

def get_user_group_slug(request):

    _group = request.environ['pylons.routes_dict'].get('id')

    _group = UserGroup.get(_group)

    if _group:

        return _group.users_group_name

    return None

def _extract_id_from_repo_name(repo_name):

    if repo_name.startswith('/'):

        repo_name = repo_name.lstrip('/')

    by_id_match = re.match(r'^_(\d{1,})', repo_name)

    if by_id_match:

        return by_id_match.groups()[0]

def get_repo_by_id(repo_name):

    """

    Extracts repo_name by id from special urls. Example url is _11/repo_name

    :param repo_name:

    :return: repo_name if matched else None

    """

    _repo_id = _extract_id_from_repo_name(repo_name)

    if _repo_id:

        from kallithea.model.db import Repository

        repo = Repository.get(_repo_id)

        if repo:

            # TODO: return repo instead of reponame? or would that be a layering violation?

            return repo.repo_name

    return None

def action_logger(user, action, repo, ipaddr='', sa=None, commit=False):

    """

    Action logger for various actions made by users

    :param user: user that made this action, can be a unique username string or

        object containing user_id attribute

    :param action: action to log, should be on of predefined unique actions for

        easy translations

    :param repo: string name of repository or object containing repo_id,

        that action was made on

    :param ipaddr: optional IP address from what the action was made

    :param sa: optional sqlalchemy session

    """

    if not sa:

        sa = meta.Session()

    # if we don't get explicit IP address try to get one from registered user

    # in tmpl context var

    if not ipaddr:

        ipaddr = getattr(get_current_authuser(), 'ip_addr', '')

    if getattr(user, 'user_id', None):

        user_obj = User.get(user.user_id)

    elif isinstance(user, basestring):

        user_obj = User.get_by_username(user)

    else:

        raise Exception('You have to provide a user object or a username')

    if getattr(repo, 'repo_id', None):

        repo_obj = Repository.get(repo.repo_id)

        repo_name = repo_obj.repo_name

    elif isinstance(repo, basestring):

        repo_name = repo.lstrip('/')

        repo_obj = Repository.get_by_repo_name(repo_name)

    else:

        repo_obj = None

        repo_name = u''

    user_log = UserLog()

    user_log.user_id = user_obj.user_id

    user_log.username = user_obj.username

    user_log.action = safe_unicode(action)

    user_log.repository = repo_obj

    user_log.repository_name = repo_name

    user_log.action_date = datetime.datetime.now()

    user_log.user_ip = ipaddr

    sa.add(user_log)

    log.info('Logging action:%s on %s by user:%s ip:%s',

             action, safe_unicode(repo), user_obj, ipaddr)

    if commit:

        sa.commit()

def get_filesystem_repos(path, recursive=False, skip_removed_repos=True):

    """

    Scans given path for repos and return (name,(type,path)) tuple

    :param path: path to scan for repositories

    :param recursive: recursive search and return names with subdirs in front

    """

    # remove ending slash for better results

    path = safe_str(path.rstrip(os.sep))

    log.debug('now scanning in %s location recursive:%s...', path, recursive)

    def _get_repos(p):

        if not os.access(p, os.R_OK) or not os.access(p, os.X_OK):

            log.warning('ignoring repo path without access: %s', p)

            return

        if not os.access(p, os.W_OK):

            log.warning('repo path without write access: %s', p)

        for dirpath in os.listdir(p):

            if os.path.isfile(os.path.join(p, dirpath)):

                continue

            cur_path = os.path.join(p, dirpath)

            # skip removed repos

            if skip_removed_repos and REMOVED_REPO_PAT.match(dirpath):

                continue

            #skip .<something> dirs TODO: rly? then we should prevent creating them ...

            if dirpath.startswith('.'):

                continue

            try:

                scm_info = get_scm(cur_path)

                yield scm_info[1].split(path, 1)[-1].lstrip(os.sep), scm_info

            except VCSError:

                if not recursive:

                    continue

                #check if this dir contains other repos for recursive scan

                rec_path = os.path.join(p, dirpath)

                if not os.path.islink(rec_path) and os.path.isdir(rec_path):

                    for inner_scm in _get_repos(rec_path):

                        yield inner_scm

    return _get_repos(path)

def is_valid_repo(repo_name, base_path, scm=None):

    """

    Returns True if given path is a valid repository False otherwise.

    If scm param is given also compare if given scm is the same as expected

    from scm parameter

    :param repo_name:

    :param base_path:

    :param scm:

    :return True: if given path is a valid repository

    """

    full_path = os.path.join(safe_str(base_path), safe_str(repo_name))

    try:

        scm_ = get_scm(full_path)

        if scm:

            return scm_[0] == scm

        return True

    except VCSError:

        return False

def is_valid_repo_group(repo_group_name, base_path, skip_path_check=False):

    """

    Returns True if given path is a repository group False otherwise

    :param repo_name:

    :param base_path:

    """

    full_path = os.path.join(safe_str(base_path), safe_str(repo_group_name))

    # check if it's not a repo

    if is_valid_repo(repo_group_name, base_path):

        return False

    try:

        # we need to check bare git repos at higher level

        # since we might match branches/hooks/info/objects or possible

        # other things inside bare git repo

        get_scm(os.path.dirname(full_path))

        return False

    except VCSError:

        pass

    # check if it's a valid path

    if skip_path_check or os.path.isdir(full_path):

        return True

    return False

def ask_ok(prompt, retries=4, complaint='Yes or no please!'):

    while True:

        ok = raw_input(prompt)

        if ok in ('y', 'ye', 'yes'):

            return True

        if ok in ('n', 'no', 'nop', 'nope'):

            return False

        retries = retries - 1

        if retries < 0:

            raise IOError

        print complaint

#propagated from mercurial documentation

ui_sections = ['alias', 'auth',

                'decode/encode', 'defaults',

                'diff', 'email',

                'extensions', 'format',

                'merge-patterns', 'merge-tools',

                'hooks', 'http_proxy',

                'smtp', 'patch',

                'paths', 'profiling',

                'server', 'trusted',

                'ui', 'web', ]

def make_ui(read_from='file', path=None, checkpaths=True, clear_session=True):

    """

    A function that will read python rc files or database

    and make an mercurial ui object from read options

    :param path: path to mercurial config file

    :param checkpaths: check the path

    :param read_from: read from 'file' or 'db'

    """

    baseui = ui.ui()

    # clean the baseui object

    baseui._ocfg = config.config()

    baseui._ucfg = config.config()

    baseui._tcfg = config.config()

    if read_from == 'file':

        if not os.path.isfile(path):

            log.debug('hgrc file is not present at %s, skipping...', path)