Evaluation, Selection, And Implementation Of Free/libre/open-source Software

Overview Process Criteria Resources

Evaluating, Selecting, and Implementing Free/Libre/Open-Source Software Marko Schütz Department of Mathematical Sciences University of Puerto Rico at Mayagüez Mayagüez, PR

07 May 2009

Marko Schütz

Evaluating, Selecting, and Implementing Free/Libre/Open-Source

Overview Process Criteria Resources

Outline

1

Overview

2

Process

3

Criteria

4

Resources

Marko Schütz

Evaluating, Selecting, and Implementing Free/Libre/Open-Source

Overview Process Criteria Resources

Workshop Structure

–

presentation of background

–

independent work

–

review

Marko Schütz

Evaluating, Selecting, and Implementing Free/Libre/Open-Source

Overview Process Criteria Resources

Introduction –

the problem > >

–

FLOSS sought to fill some need wealth of FLOSS available: 100,000s of projects

stages of the process requirements implement

identify candidates

analyze

review compare

– –

criteria resources supporting the process > >

existing evaluation frameworks project documentation templates Marko Schütz

Evaluating, Selecting, and Implementing Free/Libre/Open-Source

Overview Process Criteria Resources

Stages

involve user- and developer community –

clarify and document requirements

–

identify candidates

–

review candidates

–

shortlist and prepare comparison

–

analyze results

–

implement

Marko Schütz

Evaluating, Selecting, and Implementing Free/Libre/Open-Source

Overview Process Criteria Resources

Clarify and Document Requirements –

elicit user interviews user surveys > brainstorming sessions

>

>

–

FLOSS development differs from proprietary development FLOSS: do one thing right and inter-operate proprietary: cover many features, use lock-in (data formats) to make users stay within application > formulate neutral requirements

> >

By e.g. requiring a single application for all identified needs many possible FLOSS combinations will be ruled out –

document use cases Marko Schütz

Evaluating, Selecting, and Implementing Free/Libre/Open-Source

Overview Process Criteria Resources

Identify Candidates

–

perform search on the Internet, including specialized sites: http://sourceforge.net, http://directory.fsf.org, http://freshmeat.net, http://debian.org, http://icewalkers.com, http://cpan.org.

–

research individual features/requirements to find candidates

–

use name of well-known proprietary software with the term “open source” on google

Marko Schütz

Evaluating, Selecting, and Implementing Free/Libre/Open-Source

Overview Process Criteria Resources

Identify Candidates (cont.)

–

use existing lists > > > > > >

A MITRE 2003 report lists 115 FLOSS “generally recognized as safe” The ”Generally Recognized As Mature” (GRAM) contains 39 FLOSS Les Trophées du Libre Qualification and Selection of Open Source Software sourceforge.net project of the month, community choice awards problem with all these lists: outdated - can only be used as indicators

Marko Schütz

Evaluating, Selecting, and Implementing Free/Libre/Open-Source

Overview Process Criteria Resources

Review Candidates

–

use existing reviews and comparisons

–

mailing lists

–

user and developer documentation

–

project statistics

–

code review

–

collect findings

Marko Schütz

Evaluating, Selecting, and Implementing Free/Libre/Open-Source

Overview Process Criteria Resources

Shortlist and Prepare Comparison

–

weed out using must-have, must-not-have requirements, e.g. must have unencrypted, standards-based backup format

–

feed back findings to user community

–

feed back findings to developer community

–

goal: less than a handful solutions (might consist of several programs)

Marko Schütz

Evaluating, Selecting, and Implementing Free/Libre/Open-Source

Overview Process Criteria Resources

Analyze Results

–

document lessons learned in previous stages

–

weigh and evaluate criteria > >

–

absolute relative ranking

obtain approval from intended users, administration, management, “customer”

Marko Schütz

Evaluating, Selecting, and Implementing Free/Libre/Open-Source

Overview Process Criteria Resources

Implement

–

invite some early adopters

–

prototype alongside current solution

–

collect and maintain automated black-box tests for most/all use cases

–

allow for ample transition time

–

encourage constant feedback from users/stake-holders

Marko Schütz

Evaluating, Selecting, and Implementing Free/Libre/Open-Source

Overview Process Criteria Resources

Criteria

–

often criteria are competing >

–

example: code size vs number of features

kinds of criteria > > > > > >

requirements project community project source code project evolution deliverables trials

Marko Schütz

Evaluating, Selecting, and Implementing Free/Libre/Open-Source

Overview Process Criteria Resources

Community –

developers (number of, background of)

–

conferences

–

project activity (recent/sustained) some projects have activity bursts and times where they fall asleep

–

size, organization

–

leadership

–

mentoring

–

GSoC, etc

Marko Schütz

Evaluating, Selecting, and Implementing Free/Libre/Open-Source

Overview Process Criteria Resources

Source Code –

technologies portability hidden issues: Java may be less portable than e.g. C > teams existing competence in those technologies > teams desire, capability to learn these technologies > extends to e.g.

>

-

programming language database system desktop environment communications protocols

–

How well is the implementation documented?

–

code size > >

is an expenditure!!! bigger is worse, not better all that code has to be maintained Marko Schütz

Evaluating, Selecting, and Implementing Free/Libre/Open-Source

Overview Process Criteria Resources

Project Evolution –

How long does the project exist?

–

history How has it proceeded since then? Have there been prolonged phases of inactivity? > How has developer community evolved? > Major obstacles/challenges faced?

> >

–

maturity Is it based on a tried concept, architecture? What are the revisions it went through? > Will be different for different kinds of applications > Installations

> >

- What are the showcase deployments/installations? - How many installations are there? Marko Schütz

Evaluating, Selecting, and Implementing Free/Libre/Open-Source

Overview Process Criteria Resources

Deliverables

–

usability

–

quality

–

security

–

feature list: breadth/focus

–

documentation

Marko Schütz

Evaluating, Selecting, and Implementing Free/Libre/Open-Source

Overview Process Criteria Resources

Deliverables: Usability –

often misused: ease of installation, initial startup: valuable for casual, new users > frequent/”power” users need growth path, e.g.:

>

- scripting - external control - plug-ins –

setup prerequisites which are shared with other software that is likely to be used? e.g. requires KDE, requires PostgreSQL. Amortization. > done once, updates should be rare. Amortize over user number, usage time. > estimated time to update to future versions

>

Marko Schütz

Evaluating, Selecting, and Implementing Free/Libre/Open-Source

Overview Process Criteria Resources

Deliverables: Quality –

minor releases & point/patch releases > >

–

criteria for version number increase developer oriented might rely of code repository

open bugs > > > > > > > > > >

bugs need to be reported how many reporters? what kind of bugs are reported? assignment of appropriate priority bug-fixing sprints held avoidance of regressions reactivity to security relevant bugs security relevant bugs relative to software class what is considered critical? as with most criteria here: absolute numbers mean little Marko Schütz

Evaluating, Selecting, and Implementing Free/Libre/Open-Source

Overview Process Criteria Resources

Deliverables: Security

–

security vulnerabilities how many how critical > over what period

>

>

–

out-of-box configuration secure?

Marko Schütz

Evaluating, Selecting, and Implementing Free/Libre/Open-Source

Overview Process Criteria Resources

Deliverables: Feature List

–

the project might be too broad then risk the team doea not develop sufficient expertise in one the individual features > one of the most successful and longlived approaches has been the UNIX approach of small tools that do one thing, one thing only and that do that one thing right. >

Marko Schütz

Evaluating, Selecting, and Implementing Free/Libre/Open-Source

Overview Process Criteria Resources

Trials

–

form an early adoption team: a group of users who start using the software in real productive work: must be users who want to use the new software

–

early adoption team needs to feed back as much as possible on their experiences, issues, problems, questions, surprises, etc

–

very carfully listen to early adoption team

Marko Schütz

Evaluating, Selecting, and Implementing Free/Libre/Open-Source

Overview Process Criteria Resources

Business Readiness Rating (BRR)

–

http://www.openbrr.org

–

evaluation methodology

–

provides > >

template some sample evaluations

Marko Schütz

Evaluating, Selecting, and Implementing Free/Libre/Open-Source

Overview Process Criteria Resources

Qualification and Selection of Open Source Software (QSOS)

–

http://www.qsos.org

–

evaluation methodology

–

provides template browser-based software to work with templates > sample evaluations

> >

Marko Schütz

Evaluating, Selecting, and Implementing Free/Libre/Open-Source

Overview Process Criteria Resources

ReadySET

–

http://readyset.tigris.org

–

project documentation library

–

provides >

XHTML-based forms for software-engineering projects

Marko Schütz

Evaluating, Selecting, and Implementing Free/Libre/Open-Source

Overview Process Criteria Resources

Discussion

–

BRR and QSOS are often too rigid, e.g.: > >

QSOS only allows three values per criterion: no, partial, full BRR and QSOS strive for evaluation of criteria independent deployment context - e.g. QSOS’s books available - cannot be compensated with assignment of weights

>

–

both can be applied with regard to deployment context worst case: loss of comparability

ReadySET sometimes too software-development oriented strip out parts too software-development oriented

Marko Schütz

Evaluating, Selecting, and Implementing Free/Libre/Open-Source

Overview Process Criteria Resources

Independent Work

–

pick domain you know something about e.g. > > > > > >

database systems photo management software course management software statistics software data visualization enterprise resource planning

–

develop a realistic scenario

–

use the process described

Marko Schütz

Evaluating, Selecting, and Implementing Free/Libre/Open-Source