=================================================

Welcome to RhodeCode (RhodiumCode) documentation!

=================================================

``RhodeCode`` (formerly hg-app) is a Pylons framework based Mercurial repository 

browser/management tool with a built in push/pull server and full text search.

It works on http/https and has a built in permission/authentication system with 

the ability to authenticate via LDAP.

RhodeCode is similar in some respects to github or bitbucket_, 

however RhodeCode can be run as standalone hosted application on your own server.  

It is open source and donation ware and focuses more on providing a customized, 

self administered interface for Mercurial(and soon GIT) repositories. 

RhodeCode is powered by a vcs_ library that Lukasz Balcerzak and I created to 

handle multiple different version control systems.

RhodeCode uses `Semantic Versioning <http://semver.org/>`_

RhodeCode demo

--------------

http://demo.rhodecode.org

The default access is anonymous but you can login to an administrative account

using the following credentials:

- username: demo

- password: demo

Source code

-----------

https://hg.rhodecode.org 

http://bitbucket.org/marcinkuzminski/rhodecode

Installation

------------

Please visit http://packages.python.org/RhodeCode/installation.html

RhodeCode Features

------------------

- Has it's own middleware to handle mercurial_ protocol requests. 

  Each request can be logged and authenticated.

  Supports http/https and LDAP

- Full permissions (private/read/write/admin) and authentication per project. 

  One account for web interface and mercurial_ push/pull/clone operations.

- Mako templates let's you customize the look and feel of the application.

- Beautiful diffs, annotations and source code browsing all colored by pygments.

- Mercurial_ branch graph and yui-flot powered graphs with zooming and statistics

- Admin interface with user/permission management. Admin activity journal, logs

  pulls, pushes, forks, registrations and other actions made by all users.

- Server side forks. It is possible to fork a project and modify it freely without

  breaking the main repository.

- Full text search powered by Whoosh on the source files, and file names.

  Build in indexing daemons, with optional incremental index build

  (no external search servers required all in one application)

- Setup project descriptions and info inside built in db for easy, non 

  file-system operations

- Intelligent cache with invalidation after push or project change, provides high 

  performance and always up to date data.

- Rss / atom feeds, gravatar support, download sources as zip/tar/gz

- Async tasks for speed and performance using celery_ (works without them too)  

- Backup scripts can do backup of whole app and send it over scp to desired 

  location 

- Based on pylons / sqlalchemy / sqlite / whoosh / vcs

.. include:: ./docs/screenshots.rst

Incoming / Plans

----------------

- Project grouping

- User groups/teams

- SSH based authentication with server side key management

- Code review (probably based on hg-review)

- Full git_ support, with push/pull server (currently in beta tests)

- Redmine integration

- Public accessible activity feeds

- Commit based built in wiki system

- Clone points and cloning from remote repositories into RhodeCode

- More statistics and graph (global annotation + some more statistics)

- Other advancements as development continues (or you can of course make additions and or requests)

License

-------

Mailing group Q&A

-----------------

Join the `Google group <http://groups.google.com/group/rhodecode>`_

Open an issue at `issue tracker <http://bitbucket.org/marcinkuzminski/rhodecode/issues>`_

Join #rhodecode on FreeNode (irc.freenode.net)

or use http://webchat.freenode.net/?channels=rhodecode for web access to irc.

Online documentation

--------------------

Online documentation for the current version of RhodeCode is available at

http://packages.python.org/RhodeCode/.

You may also build the documentation for yourself - go into ``docs/`` and run::

   make html

(You need to have sphinx_ installed to build the documentation. If you don't

.. _virtualenv: http://pypi.python.org/pypi/virtualenv

.. _python: http://www.python.org/

.. _sphinx: http://sphinx.pocoo.org/

.. _mercurial: http://mercurial.selenic.com/

.. _bitbucket: http://bitbucket.org/

.. _subversion: http://subversion.tigris.org/

.. _git: http://git-scm.com/

.. _celery: http://celeryproject.org/

.. _Sphinx: http://sphinx.pocoo.org/

.. _vcs: http://pypi.python.org/pypi/vcs

.. _changelog:

Changelog

=========

======================

news

----

fixes

-----

- fixed #140 freeze of python dateutil library, since new version is python2.x

  incompatible

1.1.7 (**2011-03-23**)

======================

news

----

fixes

-----

- fixed (again) #136 installation support for FreeBSD

1.1.6 (**2011-03-21**)

======================

news

----

fixes

-----

- fixed #136 installation support for FreeBSD

- RhodeCode will check for python version during installation

1.1.5 (**2011-03-17**)

======================

news

----

- basic windows support, by exchanging pybcrypt into sha256 for windows only

  highly inspired by idea of mantis406

fixes

-----

- fixed sorting by author in main page

- fixed crashes with diffs on binary files

- fixed #131 problem with boolean values for LDAP

- fixed #122 mysql problems thanks to striker69 

- fixed problem with errors on calling raw/raw_files/annotate functions 

  with unknown revisions

- fixed returned rawfiles attachment names with international character

- cleaned out docs, big thanks to Jason Harris

1.1.4 (**2011-02-19**)

======================

# -*- coding: utf-8 -*-

"""

    rhodecode.lib.db_manage

    ~~~~~~~~~~~~~~~~~~~~~~~

    Database creation, and setup module for RhodeCode. Used for creation

    of database as well as for migration operations

    :created_on: Apr 10, 2010

    :author: marcink

    :license: GPLv3, see COPYING for more details.

"""

# This program is distributed in the hope that it will be useful,

# but WITHOUT ANY WARRANTY; without even the implied warranty of

# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

# GNU General Public License for more details.

# You should have received a copy of the GNU General Public License

import os

import sys

import uuid

import logging

from os.path import dirname as dn, join as jn

from rhodecode import __dbversion__

from rhodecode.model import meta

from rhodecode.lib.auth import get_crypt_password

from rhodecode.lib.utils import ask_ok

from rhodecode.model import init_model

from sqlalchemy.engine import create_engine

log = logging.getLogger(__name__)

class DbManage(object):

    def __init__(self, log_sql, dbconf, root, tests=False):

        self.dbname = dbconf.split('/')[-1]

        self.tests = tests

        self.root = root

        self.dburi = dbconf

        self.log_sql = log_sql

        self.db_exists = False

        self.init_db()

    def init_db(self):

        engine = create_engine(self.dburi, echo=self.log_sql)

        init_model(engine)

        self.sa = meta.Session()

    def create_tables(self, override=False):

        """Create a auth database

        """

        log.info("Any existing database is going to be destroyed")

        if self.tests:

            destroy = True

        else:

            destroy = ask_ok('Are you sure to destroy old database ? [y/n]')

        if not destroy:

            sys.exit()

        if destroy:

            meta.Base.metadata.drop_all()

        checkfirst = not override

        meta.Base.metadata.create_all(checkfirst=checkfirst)

        log.info('Created tables for %s', self.dbname)

    def set_db_version(self):

        try:

            ver = DbMigrateVersion()

            ver.version = __dbversion__

            ver.repository_id = 'rhodecode_db_migrations'

            ver.repository_path = 'versions'

            self.sa.add(ver)

            self.sa.commit()

        except:

            self.sa.rollback()

            raise

        log.info('db version set to: %s', __dbversion__)

        """

        from rhodecode.lib.dbmigrate.migrate.versioning import api

        from rhodecode.lib.dbmigrate.migrate.exceptions import \

            DatabaseNotControlledError

        upgrade = ask_ok('You are about to perform database upgrade, make '

                         'sure You backed up your database before. '

                         'Continue ? [y/n]')

        if not upgrade:

            sys.exit('Nothing done')

        repository_path = jn(dn(dn(dn(os.path.realpath(__file__)))),

                             'rhodecode/lib/dbmigrate')

        db_uri = self.dburi

        try:

            curr_version = api.db_version(db_uri, repository_path)

            msg = ('Found current database under version'

                 ' control with version %s' % curr_version)

        except (RuntimeError, DatabaseNotControlledError), e:

            curr_version = 1

            msg = ('Current database is not under version control. Setting'

                   ' as version %s' % curr_version)

            api.version_control(db_uri, repository_path, curr_version)

        print (msg)

        if curr_version == __dbversion__:

            sys.exit('This database is already at the newest version')

        #======================================================================

        # UPGRADE STEPS

        #======================================================================

        class UpgradeSteps(object):

            for example schema with seq 002 == step_2 and so on.

            """

            def __init__(self, klass):

                self.klass = klass

            def step_0(self):

                #step 0 is the schema upgrade, and than follow proper upgrades

                print ('attempting to do database upgrade to version %s' \

                                % __dbversion__)

                api.upgrade(db_uri, repository_path, __dbversion__)

                print ('Schema upgrade completed')

            def step_1(self):

                pass

            def step_2(self):

                print ('Patching repo paths for newer version of RhodeCode')

                self.klass.fix_repo_paths()

                print ('Patching default user of RhodeCode')

                self.klass.fix_default_user()

                log.info('Changing ui settings')

                self.klass.create_ui_settings()

        upgrade_steps = [0] + range(curr_version + 1, __dbversion__ + 1)

        #CALL THE PROPER ORDER OF STEPS TO PERFORM FULL UPGRADE

        for step in upgrade_steps:

            print ('performing upgrade step %s' % step)

            callable = getattr(UpgradeSteps(self), 'step_%s' % step)()

    def fix_repo_paths(self):

        """Fixes a old rhodecode version path into new one without a '*'

        """

        paths = self.sa.query(RhodeCodeUi)\

                .filter(RhodeCodeUi.ui_key == '/')\

                .scalar()

        paths.ui_value = paths.ui_value.replace('*', '')

        try:

            self.sa.add(paths)

            self.sa.commit()

        except:

            self.sa.rollback()

            raise

    def fix_default_user(self):

        """Fixes a old default user with some 'nicer' default values,

        used mostly for anonymous access

        """

        def_user = self.sa.query(User)\

                .filter(User.username == 'default')\

                .one()

        def_user.name = 'Anonymous'

        def_user.lastname = 'User'

        def_user.email = 'anonymous@rhodecode.org'

        try:

            self.sa.add(def_user)

            self.sa.commit()

        except:

            self.sa.rollback()

            raise

    def admin_prompt(self, second=False):

        if not self.tests:

            import getpass

            def get_password():

                confirm = getpass.getpass('Confirm password:')

                if password != confirm:

                    log.error('passwords mismatch')

                    return False

                if len(password) < 6:

                    log.error('password is to short use at least 6 characters')

                    return False

                return password

            username = raw_input('Specify admin username:')

            password = get_password()

            if not password:

                #second try

                password = get_password()

                if not password:

                    sys.exit()

            email = raw_input('Specify admin email:')

            self.create_user(username, password, email, True)

        else:

            log.info('creating admin and regular test users')

    def create_ui_settings(self):

        """Creates ui settings, fills out hooks

        and disables dotencode

        """

        #HOOKS

        hooks1_key = 'changegroup.update'

        hooks1_ = self.sa.query(RhodeCodeUi)\

            .filter(RhodeCodeUi.ui_key == hooks1_key).scalar()

        hooks1 = RhodeCodeUi() if hooks1_ is None else hooks1_

        hooks1.ui_section = 'hooks'

        hooks1.ui_key = hooks1_key

        hooks1.ui_value = 'hg update >&2'

        hooks1.ui_active = False

        hooks2_key = 'changegroup.repo_size'

        hooks2_ = self.sa.query(RhodeCodeUi)\

            .filter(RhodeCodeUi.ui_key == hooks2_key).scalar()

        hooks2 = RhodeCodeUi() if hooks2_ is None else hooks2_

        hooks2.ui_section = 'hooks'

        hooks2.ui_key = hooks2_key

        hooks2.ui_value = 'python:rhodecode.lib.hooks.repo_size'

        hooks3 = RhodeCodeUi()

        hooks3.ui_section = 'hooks'

        hooks3.ui_key = 'pretxnchangegroup.push_logger'

        hooks3.ui_value = 'python:rhodecode.lib.hooks.log_push_action'

        hooks4 = RhodeCodeUi()

        hooks4.ui_section = 'hooks'

        hooks4.ui_key = 'preoutgoing.pull_logger'

        hooks4.ui_value = 'python:rhodecode.lib.hooks.log_pull_action'

        #For mercurial 1.7 set backward comapatibility with format

        dotencode_disable = RhodeCodeUi()

        dotencode_disable.ui_section = 'format'

        dotencode_disable.ui_key = 'dotencode'

        dotencode_disable.ui_value = 'false'

        try:

            self.sa.add(hooks1)

            self.sa.add(hooks2)

            self.sa.add(hooks3)

            self.sa.add(hooks4)

            self.sa.add(dotencode_disable)

            self.sa.commit()

        except:

            self.sa.rollback()

            raise

    def create_ldap_options(self):

        """Creates ldap settings"""

        try:

            for k, v in [('ldap_active', 'false'),

                        ('ldap_host', ''),

                        ('ldap_port', '389'),

                        ('ldap_ldaps', 'false'),

                        ('ldap_dn_user', ''), ('ldap_dn_pass', ''),

                        ('ldap_base_dn', '')]:

                setting = RhodeCodeSettings(k, v)

                self.sa.add(setting)

            self.sa.commit()

        except:

            self.sa.rollback()

            raise

        if not self.tests and not test_repo_path:

            path = raw_input('Specify valid full path to your repositories'

                        ' you can change this later in application settings:')

        else:

            path = test_repo_path

        if not os.path.isdir(path):

            sys.exit()

        self.create_ui_settings()

        #HG UI OPTIONS

        web1 = RhodeCodeUi()

        web1.ui_section = 'web'

        web1.ui_key = 'push_ssl'

        web1.ui_value = 'false'

        web2 = RhodeCodeUi()

        web2.ui_section = 'web'

        web2.ui_key = 'allow_archive'

        web2.ui_value = 'gz zip bz2'

        web3 = RhodeCodeUi()

        web3.ui_section = 'web'

        web3.ui_key = 'allow_push'

        web3.ui_value = '*'

        web4 = RhodeCodeUi()

        web4.ui_section = 'web'

        web4.ui_key = 'baseurl'

        web4.ui_value = '/'

        paths = RhodeCodeUi()

        paths.ui_section = 'paths'

        paths.ui_key = '/'

        paths.ui_value = path

        hgsettings1 = RhodeCodeSettings('realm', 'RhodeCode authentication')

        hgsettings2 = RhodeCodeSettings('title', 'RhodeCode')

        try:

            self.sa.add(web1)

            self.sa.add(web2)

            self.sa.add(web3)

            self.sa.add(web4)

            self.sa.add(paths)

            self.sa.add(hgsettings1)

            self.sa.add(hgsettings2)

            self.sa.commit()

        except:

            self.sa.rollback()

            raise

        self.create_ldap_options()

        log.info('created ui config')

    def create_user(self, username, password, email='', admin=False):

        log.info('creating administrator user %s', username)

        new_user = User()

        new_user.username = username

        new_user.password = get_crypt_password(password)

        new_user.name = 'RhodeCode'

        new_user.lastname = 'Admin'

        new_user.email = email

        new_user.admin = admin

        new_user.active = True

        try:

            self.sa.add(new_user)

            self.sa.commit()

        except:

            self.sa.rollback()

            raise

    def create_default_user(self):

        log.info('creating default user')

        #create default user for handling default permissions.

        def_user = User()

        def_user.username = 'default'

        def_user.password = get_crypt_password(str(uuid.uuid1())[:8])

        def_user.name = 'Anonymous'

        def_user.lastname = 'User'

        def_user.email = 'anonymous@rhodecode.org'

        def_user.admin = False

        def_user.active = False

        try:

            self.sa.add(def_user)

            self.sa.commit()

        except:

            self.sa.rollback()

            raise

    def create_permissions(self):

        #module.(access|create|change|delete)_[name]

        #module.(read|write|owner)

        perms = [('repository.none', 'Repository no access'),

                 ('repository.read', 'Repository read access'),

                 ('repository.write', 'Repository write access'),

                 ('repository.admin', 'Repository admin access'),

                 ('hg.admin', 'Hg Administrator'),

                 ('hg.create.repository', 'Repository create'),

                 ('hg.create.none', 'Repository creation disabled'),

                 ('hg.register.none', 'Register disabled'),

                ]

        for p in perms:

            new_perm = Permission()

            new_perm.permission_name = p[0]

            new_perm.permission_longname = p[1]

            try:

                self.sa.add(new_perm)

                self.sa.commit()

            except:

                self.sa.rollback()

                raise

    def populate_default_permissions(self):

        log.info('creating default user permissions')

        default_user = self.sa.query(User)\

        .filter(User.username == 'default').scalar()

        reg_perm = UserToPerm()

        reg_perm.user = default_user

        reg_perm.permission = self.sa.query(Permission)\

        .filter(Permission.permission_name == 'hg.register.manual_activate')\

        .scalar()

        create_repo_perm = UserToPerm()

        create_repo_perm.user = default_user

        create_repo_perm.permission = self.sa.query(Permission)\

        .filter(Permission.permission_name == 'hg.create.repository')\

        .scalar()

        default_repo_perm = UserToPerm()

        default_repo_perm.user = default_user

        default_repo_perm.permission = self.sa.query(Permission)\

        .filter(Permission.permission_name == 'repository.read')\

        .scalar()

        try:

            self.sa.add(reg_perm)

            self.sa.add(create_repo_perm)

            self.sa.add(default_repo_perm)

            self.sa.commit()

        except:

            self.sa.rollback()

            raise

# -*- coding: utf-8 -*-

"""

    rhodecode.model.scm

    ~~~~~~~~~~~~~~~~~~~

    Scm model for RhodeCode

    :created_on: Apr 9, 2010

    :author: marcink

    :license: GPLv3, see COPYING for more details.

"""

# This program is distributed in the hope that it will be useful,

# but WITHOUT ANY WARRANTY; without even the implied warranty of

# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

# GNU General Public License for more details.

# You should have received a copy of the GNU General Public License

import os

import time

import traceback

import logging

from vcs import get_backend

from vcs.utils.helpers import get_scm

from vcs.exceptions import RepositoryError, VCSError

from vcs.utils.lazy import LazyProperty

from mercurial import ui

from beaker.cache import cache_region, region_invalidate

from rhodecode import BACKENDS

from rhodecode.lib import helpers as h

from rhodecode.lib.auth import HasRepoPermissionAny

from rhodecode.lib.utils import get_repos, make_ui, action_logger

from rhodecode.model import BaseModel

from rhodecode.model.user import UserModel

from rhodecode.model.db import Repository, RhodeCodeUi, CacheInvalidation, \

    UserFollowing, UserLog

from rhodecode.model.caching_query import FromCache

from sqlalchemy.orm import joinedload

from sqlalchemy.orm.session import make_transient

from sqlalchemy.exc import DatabaseError

log = logging.getLogger(__name__)

class UserTemp(object):

    def __init__(self, user_id):

        self.user_id = user_id

class RepoTemp(object):

    def __init__(self, repo_id):

        self.repo_id = repo_id

class ScmModel(BaseModel):

    """Generic Scm Model

    """

    @LazyProperty

    def repos_path(self):

        """Get's the repositories root path from database

        """

        q = self.sa.query(RhodeCodeUi).filter(RhodeCodeUi.ui_key == '/').one()

        return q.ui_value

    def repo_scan(self, repos_path, baseui):

        repository itself. Return a dictionary of repository objects

        :param repos_path: path to directory containing repositories

        :param baseui: baseui instance to instantiate MercurialRepostitory with

        """

        log.info('scanning for repositories in %s', repos_path)

        if not isinstance(baseui, ui.ui):

            baseui = make_ui('db')

        repos_list = {}