Category: Dissertation

Here is my presentation for the 2014 AAEEBL conference, complete with text. Thank you to those who came, to Macaulay Honors College for their support, and to those reading this for their constructive feedback.

Abstract:

After over a decade of integrating eportfolio technology into the post-secondary classroom, where do we stand? The pedagogical practice of asking students to compose in open, online, multi-user spaces has grown rapidly in recent years. There are a host of advantages that support this practice, including that writing in public venues cultivates digital literacy through broader audience awareness, facilitates interactivity and collaboration among peers, and supports the creation and integration of multimedia artifacts into the writing process. Addressing the lack of systematic study of students’ preparedness to write in online spaces, and evidence that these practices foster the development of long-tail, real-world skills, this presentation will demonstrate qualitative and quantitative approaches to investigating these assertions. Rather than focusing on administrative measures of success, this investigation focuses on the learning process of students.

Full Text (note slide advance prompts are included):

When this call for papers came out, it was if I created it to match my dissertation project, since the research track – “Data-driven Evaluation of ePortfolios in an Age of Increased Accountability” – expresses my topic exactly. In fact, I am going to use the prompts provided by the call to outline my project in order to present my hypothesis, data, methodology, and initial findings.

Prompts:

1. What are the best practices for evaluating the “value-added” of a particular course or program?
2. Which types of eportfolios are more successful in measuring learning outcomes?
3. Effective methods of analysis and evaluation
4. What do we know so far from research? What are the important questions still ahead?

SLIDE

What are the best practices for evaluating the “value-added” of a particular course or program?

From blog posts, to scholarly journals, and of course the rising interest of popular media outlets, everyone seems to have an opinion of the integration of blogging technology in higher education. Even a cursory Google search produces a host of constituent assertions that support the use of online writing platforms, such as eportfolios, in college-level courses. Claims in favor of this integration include that writing in public venues cultivates digital literacy through broader audience awareness, facilitates interactivity and collaboration between peers, and supports the creation and integration of multimedia artifacts into the writing process. However, most of these assertions are based on anecdotal narratives or survey results that focus on the experience of the faculty and administrators involved.

SLIDE

What I am seeking is evidence derived from the content of the compositions created by students in online, open spaces, and the value of this experience as articulated by the students themselves. Therefore, this project seeks to address the lack of systematic study of students’ writing in online spaces, the multimodal aspects of digital composition, and evidence that these practices foster the development of long-tail,^[1] real-world skills.

This study is an attempt to investigate the assertions made about the integration of digital writing in higher education through a combination of qualitative and quantitative research. By applying digital humanities methods and composition theory to almost a decade of student writing produced in an online, open eportfolio system, I will look for evidence of the “value added” through the adoption of a well-supported, cross-curricular implementation of eportfolio technology.

SLIDE

Which types of eportfolios are more successful in measuring learning outcomes?

The research is drawn from a case study of six consecutive years of eportfolios, culled from the Macaulay Honors College (Macaulay), a unique honors program within the City University of New York system that spans eight public university campuses. This group was chosen for study for a number of reasons, involving the particular set of benefits afforded to these students, as well as the demographics of the population itself. Each student is provided with a new laptop computer, dedicated advisors, and full tuition, theoretically eliminating some variables with regard to access and availability of tools and support. The Macaulay student population is notably diverse, consisting of students from a wide range of ethnic, racial, and economic backgrounds, with a significant portion being first-generation college students. These students take the same four seminars in their first two years of the program, and learn the same software in the course of their studies. The program is supported with Instructional Technology Fellows (such as myself) who run workshops, immersion events, and are available for consultation throughout their coursework. Therefore, although the test group is made up of a diverse sample of students, since they are provided with equal – and exemplary – resources in the pursuit of their studies, the Macaulay students represent a strong case study.

SLIDE

Just as this is a particularly strong sample of students to study, this eportfolio system also presents significant advantages. Not all eportfolio platforms are created equal; many proprietary programs are walled gardens available only to those within the university (often only in the class) and do not allow the students to access the backend in order to experiment with functionality and design elements of the site. These skills are in high demand,and selecting a system that bars this level of engagement in digital literacy is a missed opportunity for long-tail education.

SLIDE

The eportfolio system this investigation focuses on is run on WordPress, a blogging platform with BuddyPress, a social networking function, built-in. As of the end of 2013, an estimated 20% of the Internet was built on WordPress, so working on this platform in an educational setting provides students with the opportunity to develop real-world skills.

From a usability standpoint, WYSIWYG blogging platforms are a desirable content management system for use in higher education. With lower barriers to entry, and the familiarity of the basic composing functions, users comfortable with desktop publishing can transition to the online space with minimal instruction. In WordPress, the capabilities of the platform that extend beyond composing deal mostly with the design of the front-end: the choice of theme, the information architecture of the site, and the ability to draw in outside information to be displayed in the widget areas of the site. The skills needed to control these elements can be outsourced to a site administrator, which in the case of Macaulay Honors College is typically a combination of the instructor and the Instructional Technology Fellow. But, both the platform and the mediation by the administrators distance the student from understanding how the technology works, and denies them unimpeded control of their compositions. Essentially these mediators are doing the work that makes online publishing different than composing on paper or in a word processor for the students. This is a missed opportunity currently being addressed by many innovative instructors in higher education.

SLIDE

For example, Karl Stolley has his students compose entirely in code, setting up their own servers and designing their own sites from the metaphorical ground up. As Stolley argued in his keynote address at the 2013 Computers & Writing conference:

 Given the opportunity for extended encounters with difficulty (rather than the software   tools that route around it), digital writers can become specific intellectuals: people whose deep technological expertise rivals that of their command of rhetoric–who are                     therefore able to learn, teach, and build things that scare the living crap out of others.      (http://talks.karlstolley.com/cwcon-keynote/#/9)

Stolley’s assertion includes two supporting points worth mentioning here: first, that even though all digital writing is difficult, if we can avoid “excessive mediation” (an example of which would be a content management system that does not grant access to the backend, like Blackboard) then we can avoid the second pitfall, which is the need to “keep up” with impediments such as platform upgrades that can delay progress. The remedy for these common pitfalls in digital writing pedagogy for Stolley remains command line level learning.

Although Stolley’s practice represents one extreme, albiet admirable approach, it is representative of a turn away from remediation back toward the fundamentals of computer programming (“In Search of Troublesome Digital Writing: A Meditation on Difficulty”). That the future of rhetoric is an ability to communicate with computers is at the heart of this movement.

SLIDE

Returning to Vygotsky’s theory of cognitive development, I believe a digital writing platform can serve as a “zone of proximal” development for the student of digital writing. In fact, a site created for the purpose of playing and learning is called a “sandbox” by the WordPress community. The middle ground between the command line and the word processor represented by a WordPress Dashboard functions as a learning environment in which student can develop skills that could be applied to more advanced system engineering. At Macaulay the curriculum committee committed to extend programmatic learning objectives beyond this first stage of development by having students create WordPress sites as a class for the final project in the second seminar. Positioning this project in the second semester of their first year gives students a chance to acclimate to the WordPress interface before embarking in the advanced work of designing a site. The website project teaches the students to understand several critical digital literacy skills, encouraging them to see the relationship between the content and design of their site by working through information architecture choices and usability design decisions as a group. This collaborative engagement mimics a professional environment, and allows students with technical aptitude or design proclivities to guide those whose strength may be in research and writing. Since over 20% of the web is built on WordPress[1], knowing the difference between a page and post, widget and plugin, or understanding how to choose and customize a theme, are “real-world” skills attractive to employers. This curriculum-wide scaffolding prepares Macaulay students to create their own personal portfolios as well, an option many students embrace in order to build their online presence in preparation for the next stage of their career. This study aims to explore the connection between the course sites and the personal sites in order to identify the transference of skills from the teacher-directed content to the student-directed content.

SLIDE

By inviting Macaulay students to participate in the design of their course sites, create web-based projects in conjunction with traditional research and fieldwork, and by further encouraging and supporting them in building their own eportfolios that exist outside of their formal coursework, the college aligns their education philosophy with the increasingly wide-spread “Maker” movement happening both within and around the academy. “Critical Making,” as defined by Matt Ratto and Stephen Hockema, “is an elision of two typically disconnected modes of engagement in the world — ‘critical thinking,’ often considered as abstract, explicit, linguistically-based, internal and cognitively individualistic; and ‘making,’ typically understood as material, tacit, embodied, external, and community-oriented” (52)[Ratto, Matt and Megan Boler, eds. DIY Citizenship: Critical Making and Social Media. Cambridge, MA: MIT Press. (In press; forthcoming January 2014)]. The claim then, is that an education based on building and making will lead to a combination of practical proficiencies, experience working with others, and critical thinking skills that can be adapted to meet a variety of complex tasks. It is a future-thinking mode of learning, that positions students not just to accomplish the mission at hand, but be prepared to envision the next problem, and solve it.

As Roger Whitson writes, “making enables us to rethink how a different combination of methods and practices could create different gadgets, experiences, and histories.[…] Making is not simply a way of understanding; it is also an investigation of what could have been” (“Steampunk”).

Hosting the eportfolio system on WordPress not only allows the undergraduate students to take on an active role as makers, it also enabled me, as a graduate student, to access the data they produced and engage in the act of making new as well. Since Macaulay owns and operates these sites on their own server, and because the large majority of these sites are public, I was able to download the content of over 3000 sites in a MySQL database, which can then be manipulated and reformatted in multiple ways.

SLIDE

Why would I want to delve into thousands of lines of messy data? Because I strongly believe that we, as educators, should reclaim ownership of the digital material we produce. Rather than allowing corporations to mine own data in order to sell it back to us through advertising and expensive long-term contracts that lock us into products that limit us in a myriad of ways – we should be building, maintaining, and mining our own sites. This study is grounded in the philosophical belief that if we view writing as data, and use it as such through the use of digital humanities tools (distant reading, data mining, and data visualization), we can improve our pedagogy based on what we extract from this data. This meta-approach to research is experimental and experiential – as Kathleen Fitzpatrick put it, I am “doing the risky thing” by attempting a methodology that requires a high degree of technical difficulty as is still met with skepticism in the humanities. Perhaps, if I can show what is possible to discover when we control our own course management systems and use them for research and assessment, my study can be used by others to argue to the move to open source platforms in higher education.

SLIDE

Effective methods of analysis and evaluation

The data for this research is being collected in three phases, with each phase investigating a stage of development in the learning process of Macaulay students exposed to the eportfolio system as part of their undergraduate coursework. This investigation utilizes both data triangulation, and method triangulation. The first stage is a survey of incoming students to Macaulay Honors College, examining their online reading and writing habits prior to enrollment. The second is a data analysis of student work on teacher-directed course sites (with particular attention on student writing, tagging, and citation practices), and the third, a series of interviews with the winners of an institution-wide eportfolio contest, with in-depth analysis of the corresponding student-directed sites.

The first stage of this study is a voluntary online survey presented to first year students focused on understanding their online reading and writing habits prior to entering MHC. The survey was distributed in Spring 2014, and will be disturbed again to the incoming class in Fall 2014. The purpose of this survey is to assess how prepared the students are to compose in open, online spaces when they begin the honors program by revealing how often and to what extent they have engaged in online writing practices in their personal, professional, and educational lives before entering college. The survey asks the students to identify the sites they use to communicate on the web and their degree of interaction within these online writing spaces; for example whether they used the Internet to gather information, create original material on a website, or to design and manipulate the infrastructure of a website.

SLIDE

*survey findings*

SLIDE

The second stage of this study involves data mining material from six years of teacher-directed eportfolio sites. These course sites consist of content generated from the four seminars required by MHC: Seminar 1, The Arts in New York City; Seminar 2, Science and Technology in New York City; Seminar 3, The Peopling of New York City; and Seminar 4, The Future of New York City. The Associate Dean of Teaching, Learning and Technology, Dr. Joseph Ugoretz, worked with me to extract any of the data that was marked private, therefore providing data from sites on the network that are completely open to the public. This is the stage I am working on now, and this is why I am attending this conference. Thus far, I have been working with my Micki Kaufman, a Digital Fellow at the Graduate Center, to clean this data in usable chunks. First, we selected specific classes that contained robust data, such as content-rich posts. Uses this test cases, we were able to look at relationships represented in the data – such as this chart showing frequency and length of post by each contributor. This graph is done in Excel as a proof of concept, but for large-scale visualizations this work will be done using Gephi and D3. We are also removing extraneous material such as the html markup present in the text so that I can filter it through concordance software such as Voyant.

Much of my work distant reading the sites is informed by a combination of composition and rhetoric and new media theory. I am drawing from process theory and cognitive psychology, as well as case studies that implement scientific methods and the history of portfolio-based instruction, to structure and define my study of student writing in online open spaces.

SLIDE

For instance, I am using case studies that “code” student writing, such as those performed by Janet Emig and Sondra Perl. In “The Composing Processes of Twelfth Graders,” Emig identifies the “two dominant modes of composing” are the reflexive and the extensive:

The reflexive focuses upon the writer’s thoughts and feelings concerning his        experiences; the chief audience is the writer himself; the domain explored is often the affective; the style is tentative, personal, and exploratory. The extensive mode is defined here as the mode that focuses upon the writer’s  conveying a message or a communication to another; the domain explored is usually the cognitive; the style is assured, impersonal and often reportorial. (4)

I wish to make use of Emig’s terms while complicating the idea of public versus private as applied in the digital space. The description of the reflexive mode as “affective” and “exploratory” are hallmarks of many of the assignments found in portfolio based classes, however in the case of online, open eportfolios the intended audience is not the self, but an external audience of their instructors, peers, and anyone searching for the topic on the Internet. Emig defines the reflexive mode as internal, what she calls “self-sponsored” writing, whereas writing in the extensive mode, or “school-sponsored” writing, is intended for an outside audience. However, when considering writing in a public online space this designation falls apart, since the blogosphere invites the writer to mesh both of these modes. The personal, confessional style^⁠6 of the blogging genre matches Emig’s description of “reflexive” writing, but the public audience not only suggests, but goes beyond the concept of extensive writing.

SLIDE

Similarly, the digital space complicates the notion – coined by Peter Elbow – of “low” and “high” stakes writing. Now fairly ubiquitous across the education system, low stakes writing assignments tend to be short exploratory exercises that build toward formal “high stakes” assignments, which carry a large percentage of the course grade. On paper, this practice can take many forms, such as in-class free writing, out-of-class journal entries, or ink shedding – a timed pre-writing exercise. This writing can be private or public; the key is that it is not graded or assessed in terms of error. Online, low stakes assignments manifest in familiar forms, such as short reviews and reflections on a course site, but in new formats such as blog posts, discussion forum threads, comments, or as contributions to a social media site (such as twitter, facebook, tumblr, etc.). In the digital realm, low stakes writing is almost always public, even if just to a limited audience. From my work examining student writing in the Macaulay setting, blogging assignments tend toward reflection and self-evaluation. In many of the course sites examined for this study, student posts were meant to express the individual’s response to a text, event, or experience. While some contain the marks of scholarly work, such as research and citations, the content conveys a personal context, rather than an academic one. However, while these short written responses may only be assessed in terms of completion, many would consider the venue to be a high stakes environment. Since the writing is public, and it is often read by their peers, teacher, and potentially the general public, the pressure of an audience raises the stakes.

SLIDE

What do we know so far from research? What are the important questions still ahead?

As Collin Brooke points out in Lingua Fracta: Toward a Rhetoric of New Media, we look to the rhetoric of old media to answer our questions about new media practices. So, when asking about the drawbacks and benefits of blogging, we cannot define the results through the language of old forms, such as the academic essay, journal article, or scholarly monograph. Instead, there should be an effort to examine the new technology in order to determine how its form and function shape the writing process. For that we must look at how people use the technology, and how the technology uses us.

While building on the process theory can provide a context for identifying the stylistic content of student writing on online, open sites, this ignores the other essential element of composition in digital spaces: the space itself. When investigating the writing process in digital spaces, the interface design must also be considered as an active agent. As Doug Rushkoff said in a 2014 presentation at the CUNY Digital Humanities Initiative “I could teach more by through analyzing the design of the Blackboard interface than by teaching with Blackboard.” As research in the area increases it becomes increasingly evident that design mediates our composition process in significant ways that need to be accounted for and articulated. In their 2005 case study presented in “Movement in the Interface,” Synne Skjulstad and Andrew Morrison work through the difficulties of articulating interface design in their attempt to describe the process of building a multimodal site (BallectroWeb). They write:

SLIDE

 Studied in terms of human-computer interaction (HCI), interfaces have been  thought of as intermediary to communication. However, interfaces have   come to be understood as more than a static, graphical layer lying between system and user. They exist as devices for shaping and spatialising the  organization, selection and articulation of what is to be communicated electronically. As a result, interfaces are now an integral and dynamic part of communication design as a whole. (Skjulstad 415)

Drawing from Lev Vygotsky’s concept of the mediating artifact from Activity Theory, Skjulstad and Morrison conclude that the “constructedness” of the interface mediates the content. If taken as true, the content management systems on which eportfolios are built and managed affect the content itself, and therefore no two systems can be taken as equivalent. The decisions made for the writer by the interface design are as important to the final product as the choices made by the authors themselves.

SLIDE

Students compose in WordPress using the “backend” of the platform, an area not visible to a viewer who does not have editing privileges. Known as the “Dashboard,” the control panel for the site obviously resembles a word processor, with icon based action buttons that represent common tasks. The remediation at work in the iconography of word processing programs, such as a floppy disk image for the save function, has been discussed elsewhere but the transference of these structures to the blogging interface carries further implications. Designing the backend of the blog to look like a blank page to be filled with text signals to the composer that words should be the primary mode of creation. Most of the icons offer options to manipulate the text, including font styles, font sizes, and color options, along side functions that directly apply to the delivery of text such as spell check, line spacing, and paragraph formatting. The majority of the elements that encourage the composer to experiment with multimedia also match those found in word processors – such as the ability to add hyperlinks to other webpages, internal bookmarking features, and a WYSIWYG insert media function which uploads images, info-graphics, or videos from files on your computer. All of these align themselves with the word processor rather than with the practice of original bloggers who used the command line to write – requiring code. This move away from composing with mark-up languages such as HTML and style sheets such as CSS is an interesting one, with long-tail benefits and drawbacks.

An amalgamation of classical rhetoric, new media theory, and critical pedagogy, participatory design proponents argue that by developing an understanding of the mode of delivery through which we communicate, we are better able to craft our message and reach our audience(s). In their article “Toward a Public Rhetoric Through Participatory Design: Critical Engagements and Creative Expression in the Neighborhood Networks Project,”
Carl DiSalvo, et al write:

SLIDE

Taken together, critical engagements with technology and the creative expression of issues through technology begin to form a public rhetoric: They constitute the activity of discovering, inventing, and delivering arguments about how we could or should live in the world. The artifacts or systems conceived or created become rhetorical by their persuasive   intentions and capabilities, and by the way they inform and/or provoke a response from or dialogue with others. (48-49)

In its ideal form, an eportfolio system built on an open platform enables learners to make sophisticated design choices; in the process of conceptualizing, implementing, critiquing, and revising the digital space, students develop a deeper comprehension of the relationship between content and delivery. Scholars such as Collin Brooke and Ben McCorkle have already made the connection between design in digital publication and delivery as a canonical rhetorical mode. Both scholars claim the field of writing studies has neglected the rhetorical modes in recent years, and call for a return to theorizing particularly the role of delivery in the age of digital publication. This call is echoed by DiSilvo et al, who argue,

Positioning design as rhetoric does not claim some essential or deterministic quality of technological artifacts or systems. Nor does it suggest that design is fundamentally duplicitous, as contemporary pejorative notions of rhetoric might imply. Rather, positioning design as rhetoric calls attention to the ways in which the built environment reflects and tries to influence values and behavior and explicitly recognizes the capacity of people to design artifacts or systems that promote or thwart certain perspectives and agendas (DiSalvo, 48-49).

SLIDE

Last stage: understanding student-directed design and delivery choices

The third phase involves a detailed study of five or six (based on availability) student-created eportfolios that were selected as Eportfolio Expo^[1] winners. The Eportfolio Expo is a self-nominated contest of student-directed sites judged by a panel of professionals who rank the submissions based on preset criteria. In the Fall of this year, I will conduct an interview of each of the student winners and perform a “close reading” of their sites, to assess how the use of eportfolios in their coursework influenced their personal sites.^[2] What I am looking for here is evidence of transfer – or the application of skills learned in the classroom to activities performed outside the boundaries of assigned coursework.

Lines of further inquiry:

– Interface design/remediation

– Rhetorical mode of delivery

– Students as Designers

– Re-evaluate collaborative work

– New form of the blog – static vs dynamic