RhodeCode (RhodiumCode)

=======================

``RhodeCode`` (formerly hg-app) is Pylons based repository management and 

RhodeCode uses `Semantic Versioning <http://semver.org/>`_

RhodeCode demo

--------------

http://hg.python-works.com

The default access is

- username: demo

- password: demo

Source code

-----------

Source code is along with issue tracker is available at

http://bitbucket.org/marcinkuzminski/rhodecode

Also a source codes can be obtained from demo rhodecode instance

http://hg.python-works.com/rhodecode/summary

Instalation

-----------

 Please visit http://packages.python.org/RhodeCode/installation.html

Features

--------

- Has it's own middleware to handle mercurial_ protocol request. Each request 

  can be logged and authenticated. Runs on threads unlikely to hgweb You can 

  make multiple pulls/pushes simultaneous. Supports http/https

- Full permissions and authentication per project private/read/write/admin. 

  One account for web interface and mercurial_ push/pull/clone.

- Mako templates let's you customize look and feel of application.

- Beautiful diffs, annotations and source codes all colored by pygments.

- Mercurial_ branch graph and yui-flot powered graphs with zooming and statistics

- Admin interface with user/permission management. User activity journal logs

  pulls, pushes, forks,registrations. Possible to disable built in hooks

- Server side forks, it's possible to fork a project and hack it free without

  breaking the main.   

- Full text search on source codes, search on file names. All powered by whoosh

  and build in indexing daemons

  (no external search servers required all in one application)

- Rss / atom feeds, gravatar support, download sources as zip/tarballs  

- Async tasks for speed and performance using celery_ (works without them too)  

- Backup scripts can do backup of whole app and send it over scp to desired 

  location

- Setup project descriptions and info inside built in db for easy, non 

  file-system operations

- Added cache with invalidation on push/repo management for high performance and

  always up to date data. 

- Based on pylons 1.0 / sqlalchemy 0.6 / sqlite

Incoming

--------

- code review (probably based on hg-review)

- full git_ support, with push/pull server

- commit based build in wiki system

- clone points and cloning from remote repositories into rhodecode 

  (git_ and mercurial_)

- some cache optimizations

- other cools stuff that i can figure out (or You can help me figure out)

License

-------

``rhodecode`` is released under GPL_ license.

Documentation

-------------

 Online documentation for current version is available at

 http://packages.python.org/RhodeCode/.

 You may also build documentation for yourself - go into ``docs/`` and run::

   make html

.. _virtualenv: http://pypi.python.org/pypi/virtualenv

.. _python: http://www.python.org/

.. _django: http://www.djangoproject.com/

.. _mercurial: http://mercurial.selenic.com/

.. _subversion: http://subversion.tigris.org/

.. _git: http://git-scm.com/

.. _celery: http://celeryproject.org/

.. _Sphinx: http://sphinx.pocoo.org/

.. _GPL: http://www.gnu.org/licenses/gpl.html

.. _vcs: http://pypi.python.org/pypi/vcs

#!/usr/bin/env python

# encoding: utf-8

# summary controller for pylons

# Copyright (C) 2009-2010 Marcin Kuzminski <marcin@python-works.com>

# 

# This program is free software; you can redistribute it and/or

# modify it under the terms of the GNU General Public License

# as published by the Free Software Foundation; version 2

# of the License or (at your opinion) any later version of the license.

# 

# This program is distributed in the hope that it will be useful,

# but WITHOUT ANY WARRANTY; without even the implied warranty of

# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

# GNU General Public License for more details.

# 

# You should have received a copy of the GNU General Public License

# along with this program; if not, write to the Free Software

# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,

# MA  02110-1301, USA.

"""

Created on April 18, 2010

summary controller for pylons

@author: marcink

"""

from pylons import tmpl_context as c, request, url

from rhodecode.lib.auth import LoginRequired, HasRepoPermissionAnyDecorator

from rhodecode.lib.base import BaseController, render

from rhodecode.model.hg import HgModel

from rhodecode.model.db import Statistics

from webhelpers.paginate import Page

from rhodecode.lib.celerylib import run_task

from rhodecode.lib.celerylib.tasks import get_commits_stats

from datetime import datetime, timedelta

from time import mktime

import calendar

import logging

try:

    import json

except ImportError:

    #python 2.5 compatibility

    import simplejson as json

log = logging.getLogger(__name__)

class SummaryController(BaseController):

    @LoginRequired()

    @HasRepoPermissionAnyDecorator('repository.read', 'repository.write',

                                   'repository.admin')

    def __before__(self):

        super(SummaryController, self).__before__()

    def index(self):

        hg_model = HgModel()

        c.repo_info = hg_model.get_repo(c.repo_name)

        def url_generator(**kw):

            return url('shortlog_home', repo_name=c.repo_name, **kw)

        c.repo_changesets = Page(c.repo_info, page=1, items_per_page=10,

                                 url=url_generator)

        e = request.environ

        uri = u'%(protocol)s://%(user)s@%(host)s%(prefix)s/%(repo_name)s' % {

                                        'protocol': e.get('wsgi.url_scheme'),

                                        'user':str(c.rhodecode_user.username),

                                        'host':e.get('HTTP_HOST'),

                                        'prefix':e.get('SCRIPT_NAME'),

                                        'repo_name':c.repo_name, }

        c.clone_repo_url = uri

        c.repo_tags = OrderedDict()

        for name, hash in c.repo_info.tags.items()[:10]:

        c.repo_branches = OrderedDict()

        for name, hash in c.repo_info.branches.items()[:10]:

        td = datetime.today() + timedelta(days=1)

        y, m, d = td.year, td.month, td.day

        ts_min_y = mktime((y - 1, (td - timedelta(days=calendar.mdays[m])).month,

                            d, 0, 0, 0, 0, 0, 0,))

        ts_min_m = mktime((y, (td - timedelta(days=calendar.mdays[m])).month,

                            d, 0, 0, 0, 0, 0, 0,))

        ts_max_y = mktime((y, m, d, 0, 0, 0, 0, 0, 0,))

        run_task(get_commits_stats, c.repo_info.name, ts_min_y, ts_max_y)

        c.ts_min = ts_min_m

        c.ts_max = ts_max_y

        stats = self.sa.query(Statistics)\

            .filter(Statistics.repository == c.repo_info.dbrepo)\

            .scalar()

        if stats and stats.languages:

            lang_stats = json.loads(stats.languages)

            c.commit_data = stats.commit_activity

            c.overview_data = stats.commit_activity_combined

            c.trending_languages = json.dumps(OrderedDict(

                                       sorted(lang_stats.items(), reverse=True,

                                            key=lambda k: k[1])[:2]

                                        )

                                    )

        else:

            c.commit_data = json.dumps({})

            c.overview_data = json.dumps([[ts_min_y, 0], [ts_max_y, 0] ])

            c.trending_languages = json.dumps({})

        return render('summary/summary.html')

def pygmentize_annotation(filenode, **kwargs):

    """

    pygmentize function for annotation

    :param filenode:

    """

    color_dict = {}

    def gen_color():

        """generator for getting 10k of evenly distibuted colors using hsv color

        and golden ratio.

        """

        import colorsys

        n = 10000

        golden_ratio = 0.618033988749895

        h = 0.22717784590367374

        #generate 10k nice web friendly colors in the same order

        for c in xrange(n):

            h += golden_ratio

            h %= 1

            HSV_tuple = [h, 0.95, 0.95]

            RGB_tuple = colorsys.hsv_to_rgb(*HSV_tuple)

            yield map(lambda x:str(int(x * 256)), RGB_tuple)

    cgenerator = gen_color()

    def get_color_string(cs):

        if color_dict.has_key(cs):

            col = color_dict[cs]

        else:

            col = color_dict[cs] = cgenerator.next()

        return "color: rgb(%s)! important;" % (', '.join(col))

    def url_func(changeset):

        tooltip_html = "<div style='font-size:0.8em'><b>Author:</b>" + \

        " %s<br/><b>Date:</b> %s</b><br/><b>Message:</b> %s<br/></div>"

        tooltip_html = tooltip_html % (changeset.author,

                                               changeset.date,

                                               tooltip(changeset.message))

        lnk_format = 'r%-5s:%s' % (changeset.revision,

                                 changeset.raw_id)

        uri = link_to(

                lnk_format,

                url('changeset_home', repo_name=changeset.repository.name,

                    revision=changeset.raw_id),

                style=get_color_string(changeset.raw_id),

                class_='tooltip',

                tooltip_title=tooltip_html

              )

        uri += '\n'

        return uri

    return literal(annotate_highlight(filenode, url_func, **kwargs))

def repo_name_slug(value):

    """Return slug of name of repository

    This function is called on each creation/modification

    of repository to prevent bad names in repo

    """

    slug = remove_formatting(value)

    slug = strip_tags(slug)

    for c in """=[]\;'"<>,/~!@#$%^&*()+{}|: """:

        slug = slug.replace(c, '-')

    slug = recursive_replace(slug, '-')

    slug = collapse(slug, '-')

    return slug

def get_changeset_safe(repo, rev):

    from vcs.backends.base import BaseRepository

    from vcs.exceptions import RepositoryError

    if not isinstance(repo, BaseRepository):

        raise Exception('You must pass an Repository '

                        'object as first argument got %s', type(repo))

    try:

        cs = repo.get_changeset(rev)

    except RepositoryError:

        from rhodecode.lib.utils import EmptyChangeset

        cs = EmptyChangeset()

    return cs

flash = _Flash()

#==============================================================================

# MERCURIAL FILTERS available via h.

#==============================================================================

from mercurial import util

from mercurial.templatefilters import person as _person

def _age(curdate):

    """turns a datetime into an age string."""

    from datetime import timedelta, datetime

    agescales = [("year", 3600 * 24 * 365),

             ("month", 3600 * 24 * 30),

             #("week", 3600 * 24 * 7),

             ("day", 3600 * 24),

             ("hour", 3600),

             ("minute", 60),

             ("second", 1)]

    age = datetime.now() - curdate

    age_seconds = (age.days * agescales[2][1]) + age.seconds

    pos = 1

    for scale in agescales:

        if scale[1] <= age_seconds:

            return time_ago_in_words(curdate, agescales[pos][0])

        pos += 1

age = lambda  x:_age(x)

capitalize = lambda x: x.capitalize()

email = util.email

email_or_none = lambda x: util.email(x) if util.email(x) != x else None

person = lambda x: _person(x)

short_id = lambda x: x[:12]

#==============================================================================

# PERMS

#==============================================================================

from rhodecode.lib.auth import HasPermissionAny, HasPermissionAll, \

HasRepoPermissionAny, HasRepoPermissionAll

#==============================================================================

# GRAVATAR URL

#==============================================================================

import hashlib

import urllib

from pylons import request

def gravatar_url(email_address, size=30):

    ssl_enabled = 'https' == request.environ.get('HTTP_X_URL_SCHEME')

    default = 'identicon'

    baseurl_nossl = "http://www.gravatar.com/avatar/"

    baseurl_ssl = "https://secure.gravatar.com/avatar/"

    baseurl = baseurl_ssl if ssl_enabled else baseurl_nossl

    # construct the url

    gravatar_url = baseurl + hashlib.md5(email_address.lower()).hexdigest() + "?"

    gravatar_url += urllib.urlencode({'d':default, 's':str(size)})

    return gravatar_url

def safe_unicode(str):

    """safe unicode function. In case of UnicodeDecode error we try to return

    unicode with errors replace, if this failes we return unicode with 

    string_escape decoding """

    try:

        u_str = unicode(str)

    except UnicodeDecodeError:

        try:

            u_str = unicode(str, 'utf-8', 'replace')

        except UnicodeDecodeError:

            #incase we have a decode error just represent as byte string

            u_str = unicode(str(str).encode('string_escape'))

    return u_str

  'git-receive-pack': dulserver.ReceivePackHandler,

}

from dulwich.repo import Repo

from dulwich.web import HTTPGitApplication

from paste.auth.basic import AuthBasicAuthenticator

from paste.httpheaders import REMOTE_USER, AUTH_TYPE

from rhodecode.lib.auth import authfunc, HasPermissionAnyMiddleware

from rhodecode.lib.utils import action_logger, is_git, invalidate_cache, \

    check_repo_fast

from rhodecode.model.user import UserModel

from webob.exc import HTTPNotFound, HTTPForbidden, HTTPInternalServerError

import logging

import os

import traceback

log = logging.getLogger(__name__)

class SimpleGit(object):

    def __init__(self, application, config):

        self.application = application

        self.config = config

        #authenticate this git request using 

        self.authenticate = AuthBasicAuthenticator('', authfunc)

    def __call__(self, environ, start_response):

        if not is_git(environ):

            return self.application(environ, start_response)

        #===================================================================

        # AUTHENTICATE THIS GIT REQUEST

        #===================================================================

        username = REMOTE_USER(environ)

        if not username:

            self.authenticate.realm = self.config['rhodecode_realm']

            result = self.authenticate(environ)

            if isinstance(result, str):

                AUTH_TYPE.update(environ, 'basic')

                REMOTE_USER.update(environ, result)

            else:

                return result.wsgi_application(environ, start_response)

        try:

            self.repo_name = environ['PATH_INFO'].split('/')[1]

            if self.repo_name.endswith('/'):

                self.repo_name = self.repo_name.rstrip('/')

        except:

            log.error(traceback.format_exc())

            return HTTPInternalServerError()(environ, start_response)

        #===================================================================

        # CHECK PERMISSIONS FOR THIS REQUEST

        #===================================================================

        action = self.__get_action(environ)

        if action:

            username = self.__get_environ_user(environ)

            try:

                user = self.__get_user(username)

            except:

                log.error(traceback.format_exc())

                return HTTPInternalServerError()(environ, start_response)

            #check permissions for this repository

            if action == 'push':

                if not HasPermissionAnyMiddleware('repository.write',

                                                  'repository.admin')\

                                                    (user, self.repo_name):

                    return HTTPForbidden()(environ, start_response)

            else:

                #any other action need at least read permission

                if not HasPermissionAnyMiddleware('repository.read',

                                                  'repository.write',

                                                  'repository.admin')\

                                                    (user, self.repo_name):

                    return HTTPForbidden()(environ, start_response)

            #log action

            if action in ('push', 'pull', 'clone'):

                proxy_key = 'HTTP_X_REAL_IP'

                def_key = 'REMOTE_ADDR'

                ipaddr = environ.get(proxy_key, environ.get(def_key, '0.0.0.0'))

                self.__log_user_action(user, action, self.repo_name, ipaddr)

        #===================================================================

        # GIT REQUEST HANDLING

        #===================================================================

        self.basepath = self.config['base_path']

        self.repo_path = os.path.join(self.basepath, self.repo_name)

        #quick check if that dir exists...

        if check_repo_fast(self.repo_name, self.basepath):

            return HTTPNotFound()(environ, start_response)

        try:

            app = self.__make_app()

            log.error(traceback.format_exc())

            return HTTPInternalServerError()(environ, start_response)

        #invalidate cache on push

        if action == 'push':

            self.__invalidate_cache(self.repo_name)

            messages = []

            messages.append('thank you for using rhodecode')

            return app(environ, start_response)

        else:

            return app(environ, start_response)

    def __make_app(self):

        backend = dulserver.DictBackend({'/' + self.repo_name: Repo(self.repo_path)})

        gitserve = HTTPGitApplication(backend)

        return gitserve

    def __get_environ_user(self, environ):

        return environ.get('REMOTE_USER')

    def __get_user(self, username):

        return UserModel().get_by_username(username, cache=True)

    def __get_action(self, environ):

        """

        Maps git request commands into a pull or push command.

        :param environ:

        """

        service = environ['QUERY_STRING'].split('=')

        if len(service) > 1:

            service_cmd = service[1]

            mapping = {'git-receive-pack': 'push',

                       'git-upload-pack': 'pull',

                       }

            return mapping.get(service_cmd, service_cmd if service_cmd else 'other')

        else:

            return 'other'

    def __log_user_action(self, user, action, repo, ipaddr):

        action_logger(user, action, repo, ipaddr)

    def __invalidate_cache(self, repo_name):

        """we know that some change was made to repositories and we should

        invalidate the cache to see the changes right away but only for

        push requests"""

        invalidate_cache('cached_repo_list')

        invalidate_cache('full_changelog', repo_name)

    if not ret:

        raise Exception('Could not get application ui settings !')

    settings = {}

    for each in ret:

        k = each.ui_key

        v = each.ui_value

        if k == '/':

            k = 'root_path'

        if k.find('.') != -1:

            k = k.replace('.', '_')

        if each.ui_section == 'hooks':

            v = each.ui_active

        settings[each.ui_section + '_' + k] = v

    return settings

#propagated from mercurial documentation

ui_sections = ['alias', 'auth',

                'decode/encode', 'defaults',

                'diff', 'email',

                'extensions', 'format',

                'merge-patterns', 'merge-tools',

                'hooks', 'http_proxy',

                'smtp', 'patch',

                'paths', 'profiling',

                'server', 'trusted',

                'ui', 'web', ]

def make_ui(read_from='file', path=None, checkpaths=True):

    """

    A function that will read python rc files or database

    and make an mercurial ui object from read options

    :param path: path to mercurial config file

    :param checkpaths: check the path

    :param read_from: read from 'file' or 'db'

    """

    baseui = ui.ui()

    if read_from == 'file':

        if not os.path.isfile(path):

            log.warning('Unable to read config file %s' % path)

            return False

        log.debug('reading hgrc from %s', path)

        cfg = config.config()

        cfg.read(path)

        for section in ui_sections:

            for k, v in cfg.items(section):

                baseui.setconfig(section, k, v)

                log.debug('settings ui from file[%s]%s:%s', section, k, v)

    elif read_from == 'db':

        hg_ui = get_hg_ui_cached()

        for ui_ in hg_ui:

            if ui_.ui_active:

                log.debug('settings ui from db[%s]%s:%s', ui_.ui_section, ui_.ui_key, ui_.ui_value)

                baseui.setconfig(ui_.ui_section, ui_.ui_key, ui_.ui_value)

    return baseui

def set_rhodecode_config(config):

    hgsettings = get_hg_settings()

    for k, v in hgsettings.items():

        config[k] = v

def invalidate_cache(name, *args):

    """Invalidates given name cache"""

    from beaker.cache import region_invalidate

    log.info('INVALIDATING CACHE FOR %s', name)

    """propagate our arguments to make sure invalidation works. First

    argument has to be the name of cached func name give to cache decorator

    without that the invalidation would not work"""

    tmp = [name]

    tmp.extend(args)

    args = tuple(tmp)

    if name == 'cached_repo_list':

        from rhodecode.model.hg import _get_repos_cached

        region_invalidate(_get_repos_cached, None, *args)

    if name == 'full_changelog':

        from rhodecode.model.hg import _full_changelog_cached

        region_invalidate(_full_changelog_cached, None, *args)

class EmptyChangeset(BaseChangeset):

    """

    """

    @LazyProperty

    def raw_id(self):

        """

        Returns raw string identifying this changeset, useful for web

        representation.

        """

    @LazyProperty

    def short_id(self):

        return self.raw_id[:12]

    def get_file_changeset(self, path):

        return self

    def get_file_content(self, path):

        return u''

    def get_file_size(self, path):

        return 0

def repo2db_mapper(initial_repo_list, remove_obsolete=False):

    """

    maps all found repositories into db

    """

    sa = meta.Session()

    rm = RepoModel(sa)

    user = sa.query(User).filter(User.admin == True).first()

    for name, repo in initial_repo_list.items():

        if not rm.get(name, cache=False):

            log.info('repository %s not found creating default', name)

            if isinstance(repo, MercurialRepository):

                repo_type = 'hg'

            if isinstance(repo, GitRepository):

                repo_type = 'git'

            form_data = {

                         'repo_name':name,

                         'repo_type':repo_type,

                         'description':repo.description if repo.description != 'unknown' else \

                                        'auto description for %s' % name,

                         'private':False

                         }

            rm.create(form_data, user, just_db=True)

    if remove_obsolete:

        #remove from database those repositories that are not in the filesystem

        for repo in sa.query(Repository).all():

            if repo.repo_name not in initial_repo_list.keys():

                sa.delete(repo)

                sa.commit()

    meta.Session.remove()

class OrderedDict(dict, DictMixin):

    def __init__(self, *args, **kwds):

        if len(args) > 1:

            raise TypeError('expected at most 1 arguments, got %d' % len(args))

        try:

            self.__end

        except AttributeError:

            self.clear()

        self.update(*args, **kwds)

    def clear(self):

        self.__end = end = []

        end += [None, end, end]         # sentinel node for doubly linked list

        self.__map = {}                 # key --> [key, prev, next]

        dict.clear(self)

    def __setitem__(self, key, value):

        if key not in self:

            end = self.__end

            curr = end[1]

            curr[2] = end[1] = self.__map[key] = [key, curr, end]

        dict.__setitem__(self, key, value)

    def __delitem__(self, key):

        dict.__delitem__(self, key)

        key, prev, next = self.__map.pop(key)

        prev[2] = next

        next[1] = prev

    def __iter__(self):

        end = self.__end

        curr = end[2]

        while curr is not end:

            yield curr[0]

            curr = curr[2]

    def __reversed__(self):

        end = self.__end

        curr = end[1]

        while curr is not end:

            yield curr[0]

            curr = curr[1]

from rhodecode import get_version

import sys

py_version = sys.version_info

requirements = [

        "Pylons>=1.0.0",

        "SQLAlchemy>=0.6.4",

        "Mako>=0.3.5",

        "pygments>=1.3.0",

        "mercurial==1.6.4",

        "py-bcrypt",

        "babel",

    ]

classifiers = ['Development Status :: 4 - Beta',

                   'Environment :: Web Environment',

                   'Framework :: Pylons',

                   'Intended Audience :: Developers',

                   'License :: OSI Approved :: BSD License',

                   'Operating System :: OS Independent',

                   'Programming Language :: Python', ]

if sys.version_info < (2, 6):

    requirements.append("simplejson")

    requirements.append("pysqlite")

#additional files from project that goes somewhere in the filesystem

#relative to sys.prefix

data_files = []

#additional files that goes into package itself

package_data = {'rhodecode': ['i18n/*/LC_MESSAGES/*.mo', ], }

description = 'Mercurial repository serving and browsing app'

#long description

try:

    readme_file = 'README.rst'

    long_description = open(readme_file).read()

except IOError, err:

    sys.stderr.write("[WARNING] Cannot find file specified as "

        "long_description (%s)\n skipping that file" % readme_file)

    long_description = description

try:

    from setuptools import setup, find_packages

except ImportError:

    from ez_setup import use_setuptools

    use_setuptools()

    from setuptools import setup, find_packages

#packages

packages = find_packages(exclude=['ez_setup'])

setup(

    name='RhodeCode',

    version=get_version(),

    description=description,

    long_description=long_description,

    keywords='rhodiumcode mercurial web hgwebdir replacement serving hgweb rhodecode',

    license='BSD',

    author='Marcin Kuzminski',

    author_email='marcin@python-works.com',

    url='http://hg.python-works.com',

    install_requires=requirements,

    classifiers=classifiers,

    setup_requires=["PasteScript>=1.6.3"],

    data_files=data_files,

    packages=packages,

    include_package_data=True,

    test_suite='nose.collector',

    package_data=package_data,

    message_extractors={'rhodecode': [

            ('**.py', 'python', None),

            ('templates/**.mako', 'mako', {'input_encoding': 'utf-8'}),

            ('public/**', 'ignore', None)]},

    zip_safe=False,

    paster_plugins=['PasteScript', 'Pylons'],

    entry_points="""

    [paste.app_factory]

    main = rhodecode.config.middleware:make_app

    [paste.app_install]

    main = pylons.util:PylonsInstaller