Summerschool2

EUN evaluation report from the MELT summer school in Mechelen

by Riina Vuorikari based on the feedback from Sylvia Hartinger, Thomas Maier and Quentin Tremerie

The MELT Summer school took place in Mechelen, Belgium on June 9 to 12 2008.

EUN's main task at the Summer School was:

• To collect sufficient feedback from the teachers on the portal use and development to be able to apply social tagging approach to enrich the content.

• To gain better understanding of the properties of digital content that teachers choose to use, even if it was of foreign origin and in a different language from what the user speaks.

This report will give first impressions on:

1. The session where teachers were asked to bring their own "travel well" resource to the Summer School, and

2. The sessions on the portal use where the feedback that was gathered from participants. The feedback on the portal was also collected in other situations due to technical problems during the portal session.

Participants

The MELT pilot teachers participated in the Summer school from following countries:

• Austria, 3 ppl
• Belgium, 9 teachers and focus group members, plus a number of organisers
• Finland, 7 ppl
• Estonia, 6 ppl
• Hungary, 12 ppl
• Italy, 1 ppl

Teachers own resources that "travel well"

Each teacher was asked to bring a learning resource to the workshop that they think travels well, i.e. they were asked to present an educational resource that they thought other teachers would find useful in their teaching context. The special attention in this session was on the cross-border issues, such as what about

• the language (e.g. is language an obstacle of use)
• the different curriculum requirements (e.g. is this same topic or matter part of my curriculum?)
• the different scenarios on the use (e.g. if learners are the intended audience, will they use it in a foreign language or is it the teacher who uses the resource for demonstration purposes?)

Group work

Teachers were divided to groups according to their main topics: there were

• 2 secondary science groups,
• elementary teachers’ group, and
• language teachers’ group.

Each group had about 8 members, and a focus group member to facilitate the discussion. The goal was that teachers present these resources to each other and then choose two, which they think “travel best”. These resources were shown to all in a common session in the following morning.

Teachers were given supporting questions:

Tasks

• Is this resource suitable for many different purposes or tasks?
• Think what kind of different scenarios could there be around the use of this resource
• Who could use this resource: is it for teachers, learners or both? Who else?

Reuse

• Could you use this resource as such? If you needed to change something, how would it need to be modified to be used in your context? Could you do these modifications yourself? Or would you let the owner of the resource to know about these changes, maybe s/he could help you?
• If you were to use the resource, how important do you think was it to hear about its previous use from your colleague? Would you think knowing the strong points and obstacles for success be important? How could these be best made available for you?

Outcomes

Example of the group work: Science teachers group

In this group, all secondary science teachers had resources that were either simulations or animations.

• Some of them very more interactive, meaning that the learner could do an exercise using a web-based module on DNA
• one was somewhat a trial and error kind of resource that allowed users to simulate how different genes over-rule when breeding (e.g. the color of rabbit, or eye color)
• Interestingly, when asked, in this group most all teachers said that they would use this material themselves for a demonstration. They would not plan to have the learners interacting with the resource. The reason for this was that the teacher would know the scientific concept already and would not need the narrative from the learning resources to know what is happening.

That resources that were liked in this group were mostly animations.

• Interestingly, all the material they had brought was either simulation or animation with hardly or little text.
• These teachers knew right away that this type of material would be something that travels well, that could be used in another country despite of language.

• In the other groups, too, most resources were not modifiable, but used as such. When users wanted to change something, users appreciated the GeoGebra’s possibility to translate. Not a lot about mix-and-match, more about show and tell.

Tuesday presentation of the resources

• Secondary Science Teachers:
• Group1: GeoGebra 2 resources, one from a Belgian teacher and one from Finnish
• Group2: Hungarian and Estonian hearts; Moon faces, animations and simulations with hardly any text

• English language learning: “Get out of the room”, Cours pour apprendre Francais (French vocabulary, hearing words).

• Primary: Estonian bird watching film; Eduland (vocabulary; exercises, listen pronunciations, drag and drop)

We asked teachers to post their resources on delicious. They can be found at: http://del.icio.us/tag/melt_mechelen and also in the wiki at: http://wiki.oercommons.org/mediawiki/index.php/MELT_Summer_School_Project

Comments and feedback on the Portal and "Travel well" resources

In this session the users were divided into two groups:

• Beginner teachers group (10 people: 2 from Finland, 8 from Estonia)
• Pilot school teachers with experience in the Portal use

The session suffered from the load on the server which made the portal slow, thus most of the feedback on the portal was collected during other sessions in small interviews. During this session teachers used the list of "travel well" resources produced by Fernand.

In this part we collect some main concerns on the portal and present some "unintended" outcomes regarding teachers sharing, tagging and reusing resources outside of the MELT portal.

Comments from teachers during interviews (in or outside of session)

Portal feedback

1. Annotations:

• Teachers liked tagging and see the potential of the tag search.
• There were a few comments on rating. It was not clear to all that a click on the stars will lead to user's comments. At least one person would like to make the insertion of comments mandatory, those should be editable at a later stage. Several teachers mentioned the need to know how many people rated the resources when viewing the result list.
• One person commented on the "My favourties": very useful, easy to understand, should be possible to edit tags and have a grouping option for the favourites

2. Displaying resources:

• Primary teachers wanted a easy access section for themselves, but the way this could be realised did not become totally clear.
• Personalisation of the Travel well and Most bookmarked resources needed. This way teachers narrow down to the topic areas they are actually interested in.
• Hoover over descriptions of resource type would be useful.
• Teachers were confused by different order of search results.
• To get real use out of the thumbnails, they should be bigger.

3. Search

• One person commented on the filters: easy to use, words are not always clear (text free), helpful, after clicking on apply the search should take you to the top of the page
• Three people suggested to have a search by country
• Two people wanted to have the possibility to save their searches
• Language teachers are not content with the indexing keyword/browsing category of "Foreign languages". It is too broad and should have sub-categories like " French language", "English language", etc. This message should also be sent to expert indexers, it is insufficient to index something only as Foreign language without sub-category

4. Technical

• Technical problems and slowness ("..technical problems are too disturbing."
• I think it would be a nice portal with a lot of resources but it doesn't work
• I like the portal and I would take my students there too, if it would only work

5. Usability of the portal

• too much text on top of the page 1
• simple and nice
• "melt selection" as a tag was not self-explanatory
• Users were mostly familiar with the term "travel well”, but it should explained somewhere on the page
• “most bookmarked” resources were of interest to 4 out of 5

6. Resources

• One Be teacher was surprised to find very little Dutch resources.
• One Be teacher asked why there are resources in Melt that do not travel well.
• One Be teacher was interested in SCORM packages.
• Material was found useful by most people, A Finnish teacher told us he liked the Chemistry and Physics resources, he found on the portal.
• An Italian teacher expressed the need for more flexible (smaller) resources that could be assembled in various ways.
• In interviews and presentations it became clear that teacher activities are needed, small annotations to a resource would be enough.

7. Other

• Some teachers expressed the interest in uploading resources. One teacher expressed the interest to have feedback on his created material by other people.

An unintended outcome: a lot of sharing happens elsewhere, Delicious usage

• Interestingly, many teachers started bookmarking resources that they found interesting from the "Travel well" list into their delicious account.
• 22 teachers said to have an account. After last year's summer school teachers said that they've used it, some more some less.
• 5 people said not to have an account at all, we do not know from 15 participants
• A quick check on those accounts reveal that they have collectively made more than 5000 posts
• When asked about the delicious use, many said that they use it for teaching related material, but also for private, for example to share some websites among their family members
• This clearly calls for a better integration of delicious service to the MELT portal, in order to make MELT portal useful and more central in the community building aspect.
  • We should allow users to post their MELT favourites in delicious whenever possible
  • We should get the users tags and bookmarks as a feed or using APML from delicious so that they could share them with other MELT users
  • This would make MELT more central in the community building part, which is to a certain extent lacking in delicious
  • Moreover, delicious has an API to access the data. We could investigate how many resources overlap with MELT or Fernand's selection and input those tags to MELT system, if needed.

• API http://del.icio.us/help/api/posts
• It would be interesting to look what is the overlap of resources in teachers' delicious and on the portal, can some of the tags be imported?

Study related to Travel well resources

To better understand what type of resources teachers find useful for their everyday teaching tasks, we had an online survey for teachers. Our aim was to study whether teachers would find educational resources that come from different countries and are in different languages, of any use for them. The questionnaire was online before the MELT summer school, but since the results are relevant with the summer school observations, we report on this study here.

You can see the list of the resources below, and the links to these resources are found here: http://del.icio.us/vuorikari/travelwell+survey

The list gives the name of the resource, it's subject area, country of origin, language, resources type and number of expertise. We also calculated a "travel well" value for these resources based on the answers, we explain that below.

The methodology

We used a Web-based questionnaire that had 28 respondents, more than half of them were MELT pilot teachers.

We showed to 28 teachers 18 digital learning resources that were

• in different subject areas
• in different languages
• originated from different countries
• different types of resources, e.g., animation, reference text, drill and practice

We asked them to answer four (4) questions regarding each resource and the answers were given on the scale of 1 to 5 (1=no - 5=for sure):

• Would you use this resource, or part of it, for planning and preparation of your lesson?
• Would you use this resource, or part of it, for your teaching?
• Would you share this resource with your colleagues?
• What is your expertise in this topic?

We only considered the votes from users who:

• had at least some expertise in the subject area of the resource, and who
• came from a different country than the resource did. For example, if there was a Hungarian learning resource and the respondent came from Hungary, we did not qualify that vote.

Results

• After processing the data with the above constraints we were left with 254 votes.

• 43% of respondents found resources, which came from a different country than they did, of use for preparation purposes.
• 41% of respondents found resources, which came from a different country than they did, of use for teaching purposes.
• 65% of respondents said that they would share these resources, or parts of them, with their colleagues and friends.
• Even 35% of respondents, who said they did not have expertise in the given subject area, thought that they would share the resource with their colleagues

In what languages were those resources? (Note, 30% of the resources were in English, see the list above)

• en 57% 60%
• de 15% 13%
• pl 10% 10%
• No text 6% 9%
• et 5% 4%
• sl 4% 3%
• ca 2% 1%
• hu 1% 1%

What type of resources were they?

• Out of the resources that users said they would use, most resources used for preparation and teaching purposes had properties like being drill and practices (22%), simple websites (27%) or demonstration (17%). There were hardly any differences between the types of resources that were used in teaching and those respondents were willing to share with colleagues and friends. Note: check what was the distribution of resources to start with.

In what subject areas were those resources? (Note, that this might rather reflect the subject expertise of the respondents than the topics that are "desired")

• Out of the resources that users said they would use, most resources used for preparation and teaching purposes were in Geography, Biology and Citizenship. As usual, educational resources geared towards Science teaching seem to be better understood in other languages than the native one. Note: check what was the distribution of resources to start with.

List of Resources that teachers selected

Calculating a "Travel well" value for resources

We calculate a hypothetical number as a value for each resource to measure how well it travels. We took the number of cross-border votes from users who voted for this resource and said to have expertise in this field. We divided the sum by the number of experts in the whole survey.

In the future, we will be interested in testing whether the actions from previous users can be used to predict the potential usefulness of a resource for other users, who come from a different country than the resources does and whose mother tongue is different from that of the resource.

Reflections on answers

a) If teachers are looking for digital resources on-line, they most likely use them both for preparatory purposes and for teaching purposes. We do not know, though, whether they would use the resource in their teaching themselves or let pupils interact with this resource. It would be interesting to know if a resource, that is in another language than that of the user, would be used by pupils or students. Some answers in a group interview indicated that teachers would most often use the resource themselves to demonstrate the concept, rather than letting their pupils to use it. The answers vary on this when the resource was for language learning, of course.

b) Teachers are good filters. More teachers say that they would be willing to share resources with their colleagues than actually use them. It might be that they find a resource, which they think is interesting, but does not match to their curriculum goals for the year. They might say, "Hey, my colleague would love this, I'll send it to her!" This is the basic mechanism of viral marketing, how can we leverage this on a learning portal?

Feedback on the programme and organisation

• It would be great if the timetable would allow more time for teachers to work on their ideas, give us real feedback, show and share their practices.
• more time really needed for teachers to share and reuse
• research type activities to understand what teachers do and why need to be better integrated into the programme so that we can guarantee to have "hard data" (e.g. document teachers own "travel well" LOs was not done properly, ..)
• more mingling activities needed, e.g. instruct teachers not hang out with their own country folks, seat people differently during dinner,

Links

• movies: Chris (http://www.youtube.com/watch?v=lltFO-GrL3s&feature=related), other teachers
• delicious link:. http://del.icio.us/tag/melt_mechelen
• Flickr: http://www.flickr.com/search/?w=all&q=melt_mechelen&m=text