The tasks will be designed to attempt to force the users to ask specific, relatively difficult types of questions. We expect that they will also ask a number of "easier" questions in the course of answering the more complex questions.

I will pilot the study with two pair-programming teams from cohort A. In addition to the videotaping and Mylar Monitor, I will do a semi-structured interview with the participants to gauge the effectiveness of the study.

• H4: Which technique people use to pull information out of the source code via the IDE correlates with how much experience they have programming.
• H5: Which technique people use to pull information out of the source code via the IDE correlates with how much experience they have with the IDE.
• H6: We can figure out when people get stuck by the patterns of their interactions with the IDE.

• PPIG (NB: good stuff, fertile ground)

• Group A will be collaborators -- graduate students in CS specializing in software practices or human-computer interfaces.
• Group B will be novice programmers -- undergraduate students in CS.
• Group C will be experienced programmers -- professional code developers.

Each of the groups will be videotaped while they use Eclipse to perform three to five different (small) programming assignments on a large code base. Their interactions with Eclipse will be logged using the Mylar Monitor, and the Mylar Monitor logs and the videotape will be synchronized.

Two studies of cohort A will use pair-programming. In all five cases, I will do a semi-structured interview with the participants to gauge the effectiveness of the study. Afterwards, I will run two pilots with people drawn from the Group A pool, where they do the same tasks, but as a pair-programming team. (In addition to the data gathered from the team, this ensures that I will be competent to run the subsequent study.)

Finally, I will run the study with at least three pair-programming teams from cohort B and three from cohort C.

• Anneliese von Mayrhauser/Andrews tbd

• the source code that they add/modify/delete

• P2: We do not know how to recognize when people are stuck.
• P3: We do not know which techniques are the most productive.
• P4: We do not know how to teach/train developers how to be more productive.

• H0.2: Developers will ask a lot of easier questions in the pursuit of answering more difficult questions. (This means that we don't have to put effort into guiding the coders to asking those easier questions.)

• Andrew Ko ISCE05, ICSE07-tbd, CHI07-tbd, other Ko papers
• Emerson Murphy Hill tbd
• Analise Von Meyerhauser/Andrews tbd
• Robert DeLine tbd
• Gina Venolia (at least check out again) Note to Gail: Gina has done some interesting email stuff AND a friend at Microsoft insists that I should meet her because I would like her; I can easily imagine going down to Seattle sometime and having coffee with her.

I will do so via a user study that captures information about several assigned tasks:

• the questions that developers ask about the code for that task
• which IDE interactions they do to answer them
• the source code that they add/modify/delete
• the time that it takes to do the task
• the quality of their code, as measured by the number of unit tests which pass

Givens:

Jonathan Sillito showed:

• G1: Questions that developers ask in the pursuit of code can be classified.

• H0.0: We can design a study that will force people ask (generally) the same questions.
• H0.1: We can figure out what questions they are asking.
• H0.2: When guiding people to ask difficult questions, they will also by
• H1: There are a few (and only a few) different techniques for pulling answers to a given question out of a code base via an IDE.
• H2: Which technique people use to pull information out of the source code via the IDE correlates with how fast they can at answering the question.
• H3: Which technique people use to pull information out of the source code via the IDE correlates with the quality of their solution.
• H4: We can figure out when people get stuck by the patterns of their interactions with the IDE.

Literature Review

• Robillard et al

• (programmer variability studies: Sackman/Erikson/Grant, Dickey, Curtis, DeMarco and Lister)

I will run pilots of the study with collaborators in Software Practices and HCI. Three pilots will be talk-aloud done by individual collaborators (who are sophisticated enough to give good talk-aloud data). Two will be pair-programming. In all five cases, I will do a semi-structured interview with the participants to gauge the effectiveness of the study.

I will then run three pilots with people who are not familiar with this project and its goals.

I will examine the Mylar Monitor logs "by hand" to see which techniques people use to answer the questions that arise. Both successful techniques -- those which lead to an answer to the question -- and unsuccessful techniques will be noted.

I will also note points where the participants are "stuck".

I will also use data mining/machine learning techniques to find patterns that correlate with

• speed
• code quality
• being "stuck"

• Set ALLOWTOPICCHANGE = DuckySherwood, GailMurphy

• P4: Coding video is expensive, annoying, and it is difficult to be consistent in coding (especially when one doesn't know what one is looking for)

• H6: Patterns are similar across tools.
• H7: Patterns are similar across languages.

• Jonathan Sillito's questions
• Andrew Ko ISCE05

• from the Mylar bugzilla: many individual compressed logs of traces of a small number of developers either fixing one well-described bug or adding a well-defined feature, with the context
• from the glob: many individual logs of traces of a large number of developers working on unknown material, without the code available

1. (potentially, tho hopefully not) write my own algorithm and/or modify an existing algorithm

• recognize those patterns
• (optionally?) to generate reports with
  • what patterns were found
  • description of the pattern distribution compared with the broader population's distribution
  • ?proscriptive advice on more effective techniques for coding?

Problems:

• P1: We (software engineering researchers) do not know what different low-level techniques developers use when developing code using an IDE.
  • P1.1: We do not have a shared vocabulary for discussing different techniques that developers use when developing code with IDEs. (?)
• P2: We do not know which techniques are the most productive.
• P3: We do not know how to teach/train developers how to be more productive.

Givens (right word?):

• G1: Different people use different techniques for locating relevant pieces of code.
• G2: Charactaristic interaction patterns reflecting those techniques can be discovered by analyzing coded transcripts of video of users navigating code.
• G3: Success at finding relevant pieces of code correlates with what technique(s) the developer uses.

Hypotheses:

• H1: These characteristic interaction patterns can be discovered by analyzing interaction telemetry of navigation tasks.
• H2: Software can recognize those patterns in navigation tasks.
• H3: Data mining software can discover interesting interaction patterns in navigation tasks.
• H4: Data mining software can discover interesting interaction patterns in more general code-development tasks.
• H5: Success in coding tasks correlates with which interaction patterns the developer uses.

• Robillard et al
• Murphy/Kersten/Findlater
• BSD et al (unpublished)
• ?

• many individual logs of traces of a small number of developers either fixing one well-described bug or adding a well-defined feature, with the code available
• many individual logs of traces of a large number of developers working on unknown material, without the code available

Proposed analysis methods

1. eyeballs (better word?) -- I will examine the data visually, with filters as appropriate to change how the data is visualized
2. protein-motif finding algorithm -- I will use a modified protein motif-finding algorithm to search for common patterns, and judgement to select interesting ones.
3. data-visualization and mining tools, e.g. YALE -- I will use data mining and visualization tools to search through the patterns.

• V.R. Basili, R.W. Selby, and D.H. Hutchens, “Experimentation in Software Engineering,” IEEE Trans. Software Eng., vol. 12, no. 7, pp. 733-743, July 1986. Martin says it's a good foundational doc.

• Jonathan's paper

• Grades

• http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=295895
• evolution metrics
• http://charm.cs.uiuc.edu/papers/ProductivityPPHECatHPCA04.pdf
• pair programming
• critique of cyclomatic metric
• IBM thing
• measuring team productivity

• Vik's talk -- uses Timed Markov Models

• • Table 2 lists the metrics evaluated in the study, including a short description and a reference to the definition of the metric. All of the metrics are proposed by Chidamber and Kimerer [CK94] or by Lorenz & Kidd [LK94] (?) However, we rule out some of the proposed metrics because they received serious critique in the literature (LCOM and RFC [CK94]), because the definition isn’t clear (MCX, CCO, CCP, CRE [LK94]; LCOM [CK94, EDL98]), because the lack of static typing in Smalltalk prohibits the computation of the metric (CBO [CK94]), because the metric is too similar with another metric included in the list (NIM, NCM and PIM in [LK94] resemble WMC-NOM in [CK94]), or simply because the metric is deemed inappropriate (NAC, SIX, MUI, FFU, FOC, CLM, PCM, PRC [LK94])

Follow-ons

• early students vs. later students
• students vs. professionals
• single vs pair-programming
• Eclipse vs other IDEs
• Java vs other languages

• • Software MEtrics: Roadmap
  • space agency metrics
  • taxonomy for programming style prolly not useful

• • impact of group and individ factors on prog productivity

Publishable papers

• time spent vs. grade vs. metrics -- whole boatload of papers possible from that!

How evaluate

• Grade
• FindBugs
• Software metrics
  • Third time charm: Stronger prediction of programmer performance by software complexity metrics, 1979. Evidence that software complexity metrics developed by Halstead and McCabe are related to the difficulty of finding bugs in the code.
  • Eclipse metrics
  • Vocabulary: afferent coupling, cohesion

We want to investigate work practices of highly productive programmers and less-productive programmers. To do so, we will

• Recruit test subjects from students in computer science classes where all students tackle the same assignments.
• Have the students install logging software.
• Log interactions that developers have with a Java integrated development environment called Eclipse.
• Have the students submit the logs with the submitted assignments.
• Have the instructor deliver the logs, the submissions, and the grade for the coding portion of the assignment.
• Assign a score to each submission based on both mechanically-derived metrics (like how tangled@@@ the code is or how many unit tests it passed)and the grade.
• Use data mining techniques to look for patterns in the data that correlate with the quality of the submissions.

• sw for doing acceptance tests on traces
• some tool/mechanism for organizing/collecting all the user data

Tools needed

• Logging sw
• visualization sw
  • something that replays the session
  • Mylog
• data mining sw
• something that checks that the trace is complete -- replays the session and makes sure that replaying the trace creates the handin

Problem Statement

Literature Review

Proposed data-gathering methods

Proposed analysis methods

Unsorted junk

Need academic ref

• reference to Stanley Eisenstat yale cs 323 time vs. outcome studies

Interesting references for me to chase down

• Jeffrey Bonar and Elliot Soloway. Uncovering principles of novice programming. In 10th ACM POPL, pages 10–13, 1983.

Ducky Thesis Proposal Notes

Interesting references to chase

• Cross, E. The behavioral styles of computer programmers. in Proc 8th Annual SIGCPR Conference. 1970. Maryland, WA, USA.

• Mayer, D.B. and A.W. Stalnaker. Selection and Evaluation of Computer Personnel – the Research History of SIG/CPR. in Proc 1968 23rd ACM National Conference,. 1968. Las Vegas, NV, USA.

• Michael McCracken, Vicki Almstrum, Danny Diaz, Mark Guzdial, Dianne Hagan, Yifat Ben- David Kolikant, Cary Laxer, Lynda Thomas, Ian Utting, and Tadeusz Wilusz. A multinational, multi-institutional study of assessment of programming skills of first-year CS students. In Working group reports from ITiCSE on Innovation and technology in computer science education, Canterbury, UK, 2001. ACM Press.

• B Adelson and E Soloway. The role of domain experience in software design. IEEE Transactions on Software Engineering, 11(November):1351–1360, 1985.
• Jeffrey Bonar and Elliot Soloway. Uncovering principles of novice programming. In 10th ACM

and other references from This Camel Has Two Humps and Testing Programming Aptitude

follow-on to the camel