> [!summary]+ Summary > This page describes a survey that was created and conducted "to glean some understanding of actual student use and opinions regarding generative Artificial Intelligence (genAI) in learning environments and at GW." Background is provided, as are the questions, responses, and a presentation of the results to the GWU Faculty Senate. # genAI Undergraduate Student Survey (Pilot) **Delivery details:** <u>Survey date</u>: December 12-16, 2023 <u>Presentation date</u>: January 19, 2024 <u>Audience</u>: Faculty Senate – Education, Policy, and Technology Committee <u>Delivery format</u>: Remote/Zoom <u>Duration</u>: 20 minutes ## About the survey This survey was designed in November 2023 and was a collaboration between myself, Irene Foster (Professor of Economics), and Patty Dinneen (Director of Faculty Development). We each had very different views of the potential role of generative Artificial Intelligence (genAI) in teaching and learning, but were equally curious what undergraduate students were actually doing with the technology. By this time, we had some ideas from conversations and observations, but nothing that really told us what was happening with students at George Washington University (GWU) and their use of genAI. To that end, the purpose of the survey was "*to glean some understanding of actual student use and opinions regarding generative Artificial Intelligence (genAI) in learning environments and at GW.*" ### Survey participation The survey was provided to sections of an introductory macroeconomics course for undergraduate students at GWU. Each section was taught by Professor Foster. The course met in-person and used Blackboard as the course LMS. Blackboard was used for all communications with students and was the selected method for communicating about this survey. Participation in the survey was completely voluntary and anonymous. The survey itself was created and run through Google Forms. To access the survey, students only needed to login with their GWU email address, and their email address was not logged as part of data collection. ### Survey invitation/instructions The following message was sent to potential student participants via Blackboard: > Dear class: > > I am working with the GW Office of Academic Innovation to understand student experience with GAI at GW. Your feedback about how you, your professors and other students use GAI (or not!) would greatly inform our review process. > > Please fill out this anonymous survey, which should take you between 5-10 minutes to complete. There are 14 main questions in the survey. The survey will be live until Saturday, December 16th 11:59pm. If 75% of the class fills out the survey, all students will receive 1 EC point towards the final. If 95% of the class fills out the survey, all students will receive 2 EC points towards the final. > > Thank you for your help with this pilot study! > Prof. Foster > > **Definition of GAI** > Generative Artificial Intelligence (GAI) refers to a subset of AI techniques that focus on creating or generating new content, such as images, text, audio, or even video, using algorithms and models. Unlike traditional AI systems that are designed for specific tasks, generative AI aims to produce novel and creative outputs that resemble human-generated content. ### Survey questions 1. **(Q1) Which of the following GAI tools have you used during the Fall 2023 term? (Choose all that apply.)** 1. Adobe Firefly (provided by GW) 2. Anthropic Claude 3. Google Bard 4. OpenAI ChatGPT (free version) 5. OpenAI ChatGPT (paid version) 6. None of the above 7. *Participants could add their unique response.* 2. **(Q2) In which of the following ways do GAI tools, such as ChatGPT, act as a partner to help you with your school work? (Choose all that apply.)** 1. Coach – reflection and learning regulation 2. Mentor – provide feedback 3. Teammate – alternative viewpoints 4. Tool – accomplish tasks 5. Tutor – receive instruction 6. Simulator – practice applying knowledge 7. None of the above 3. **(Q3) For which of the following reasons have you turned to GAI tools to help you with your school work? (Choose all that apply.)** 1. To boost my creativity on an assignment 2. To help me better understand difficult course content 3. When I have multiple outside commitments and just need to get it done 4. When I have multiple assignments and just need to get it done 5. When I am scared about failing an assignment 6. When I am scared about failing a course 4. **(Q4a) How often do you use GAI tools (e.g. OpenAI ChatGPT) ChatGPT or other GAI to help with your school work? Please choose the best answer.** 1. Use almost daily 2. Once or twice a week 3. Once or twice a month 4. Never 5. **(Q4b) How often would you say your friends use GAI tools (e.g. OpenAI ChatGPT) to help with their school work? Please choose the best answer.** 1. Almost daily 2. Once or twice a week 3. Once or twice a monthNever 4. I do not know 6. **(Q4c) I am using GAI:** 1. of my own volition 2. because a professor asked me to 7. **(Q5a) Have any of the professors, in classes you are taking this semester, explicitly allowed the use ChatGPT or other GAI tools for the following:** 1. To get started with writing: 1. Yes 2. No 3. Don't know 2. To help clarify concepts: 1. Yes 2. No 3. Don't know 3. To teach writing styles: 1. Yes 2. No 3. Don't know 4. To see a problem solved step-by-step: 1. Yes 2. No 3. Don't know 5. To be an out-of-class tutor: 1. Yes 2. No 3. Don't know 6. Propose practice questions: 1. Yes 2. No 3. Don't know 8. **(Q5b) If you answered “Yes” to anything in 5a, can you please specify in which classes they were (choose all that apply):** 1. University Writing 2. Math 3. Science 4. Social Science 5. Humanities 9. **(Q5c) Anything you'd like to add that wasn't listed in 5a?** 1. Open-ended 10. **(Q6) I would understand if a faculty member did not want me to use GAI tools in their course, and clearly explained why.** 1. Strongly disagree 2. Disagree 3. Neutral 4. Agree 5. Strongly agree 11. **(Q7a) I expect to be able to use GAI tools in any course at GW.** 1. Strongly disagree 2. Disagree 3. Neutral 4. Agree 5. Strongly agree 12. **(Q7b) My friends expect to be able to use GAI tools in any course at GW.** 1. Strongly disagree 2. Disagree 3. Neutral 4. Agree 5. Strongly agree 13. **(Q8) Which of the following best describes your attitude toward the adoption of new technologies?** 1. I love new technology and am always the first among my friends to try out a new product. 2. I like new technology and am one of the first among my friends to try out a new technology. 3. I will only try a new technology after a lot of other people have tried it first. 4. Technology scares me and I won’t adopt new technology unless I am forced to 14. **(Q9) How important do you think it is for you to learn how to effectively use GAI tools?** 1. Not at all important 2. Slightly important 3. Important 4. Fairly important 5. Very important 15. **(Q10) I believe GAI is an accurate source of information for my school work.** 1. All the time 2. Most of the time 3. Some of the time 4. Never 16. **(Q11) How would you rate your ability to use GAI effectively?** 1. Expert 2. Intermediate 3. Novice 4. No ability 17. **(Q12) What approach would you like to see GW take regarding the use of GAI tools in classes?** 1. Open-ended 18. **(Q13) Is paying for a subscription to a ‘better’ version of GAI tools a barrier for you?** 1. Yes 2. No 19. **(Q14a) Your GW School affiliation:** 1. CCAS 2. ESIA 3. GWSB 4. SEAS 5. SPH 6. Elliot (ESIA) 20. **(Q14b) Your Class level:** 1. Freshman 2. Sophomore 3. Junior 4. Senior ## Post survey data The survey was run 1 time with the idea of serving as a pilot for potential runs with other faculty after being presented to the GW Faculty Senate. For this run, it was offered to multiple sections of an undergraduate economics course between **December 12-16, 2024**. During that time, 470 students were invited and of those 336 students responded. This was a **71.5% response rate**. ### Survey analysis All data was collected from Google Sheets and analysis was conducted by me during the 2024 Winter Break. I used Google Sheets to organize responses to quantitative questions, and I used Microsoft Copilot and OpenAI ChatGPT-4 (a paid model) with the Data Analyst GPT to assist with the qualitative analysis of question 12 responses. Ultimately, attempts to work with Microsoft Copilot were futile. This may have been because I was attempting to use the *protected version* offered by the university. Relative to the consumer version, it did not perform very well. Due to the unhelpful performance, I chose not to work with Microsoft Copilot. #### A note on Q12 (open-ended) At the time of the survey, GWU's genAI guidance on its allowance in courses was entirely instructor discretion. All instructors were asked to make a declaration in their syllabus, but if they did not, then some default language neither allowing or accepting would apply. Because a single policy did not exist, faculty expectations were random. This made many students frustrated with the university, the faculty, and with their peers. Having a strong sense this was happening from various engagements with faculty and students, question 12 was added and considered very important to us. We wanted to know what students wanted the university and faculty to allow regarding the use of genAI in coursework. In analyzing this question, I read through a total of 290 responses. I did not fully trust an LLM to understand certain words, phrases, or emotion in response structure. I decided to code the responses on a 5-point scale as follows: 1. **Use** — clearly in favor of genAI tool use with a wide statement 2. **Lean-use** — in favor of genAI tool use, but identifying a specific learning use 3. **Undecided** — states they are unsure or offers positions of use and disuse 4. **Lean-disuse** — not in favor of genAI tool use, but without an absolute 'no' 5. **Disuse** — clearly not in favor of genAI tool use with a wide statement After coding the responses, I created a clean spreadsheet to upload and share with ChatGPT-4. No identifiable data was included. There were only 2 columns: 1) item number and 2) question response. The goal of this collaboration was to try to identify themes in the responses. After a number of tries, and after figuring out that the only format it would understand would be a CSV file, 7 themes were identified. Generally, it did not go very well at the time of this analysis. ### Summary of findings The following is a high-level summary of findings. For a more detailed view of responses, including our summary interpretation, please see the embedded presentation below. - **Q1: Which of the following GAI tools have you used during the Fall 2023 term? (Choose all that apply.)** - 414 selections recorded. - The most common tools used were **ChatGPT free version** (71.1%), **ChatGPT paid version** (23.8%), and **Adobe Firefly** (5.4%). 23.8% noted that they used something different. - **Q2: In which of the following ways do GAI tools, such as ChatGPT, act as a partner to help you with your school work? (Choose all that apply.)** - 682 selections were recorded. - The most common usage of GAI tools in partnership was as a **mentor** to provide feedback (40.2%). In a very close second, participants identified their second most frequent partnership as a **tutor** providing instruction (38.7%). - **Q3: For which of the following reasons have you turned to GAI tools to help you with your school work? (Choose all that apply.)** - 595 selections were recorded. - The common reason selected for GAI use was "**to help me better understand difficult course content (72.9%)**". In a distant second, participants identified "**to boost my creativity on an assignment (48.2%)**". - **Q4a: How often do you use GAI tools (e.g. OpenAI ChatGPT) ChatGPT or other GAI to help with your school work? Please choose the best answer.** - 336 responses recorded. - The most common response was "**once or twice a week (37.2%)**". However, "**once or twice a month (33.9%)**" was a very close second. - **Q4b: How often would you say your friends use GAI tools (e.g. OpenAI ChatGPT) to help with their school work? Please choose the best answer.** - 336 responses recorded. - The most common response was "**once or twice a week (38.4%)**". There was no close second answer. - **Q4c: I am using GAI:** - 336 responses recorded. - The vast majority identified that they were using GAI "**of my own volition (79.2%)**". - **Q5a: Have any of the professors, in classes you are taking this semester, explicitly allowed the use ChatGPT or other GAI tools for the following:** - This question was broken into 6 sub-questions with possible answers as "yes", "no", or "don't know". The answer with the highest response is provided. - To get started with writing — 340 responses; No (37.4%) - To help clarify concepts — 339 responses; Yes (51.6%) - To teach writing styles — 338 responses; No (51.8%) - To see a problem solved step-by-step — 341 responses; No (37.0%) - Be an out-of-class tutor — 341 responses; No (41.6%) - Propose practice questions — 343 responses; Yes (40.0%) - **Q5b: If you answered “Yes” to anything in 5a, can you please specify in which classes they were (choose all that apply):** - 346 selections were recorded. - The most selected response was **social science (107)** and the next highest response was **humanities (91)**. - **Q5c: Anything you'd like to add that wasn't listed in 5a?** - 47 responses were given. - One of the longest responses was, "In my comparative politics class, my teacher had us do a whole essay using chatgpt in order to analyze the shortcomings of it and then in the next class, she showed us how to use chatgpt to our advantage." - **Q6: I would understand if a faculty member did not want me to use GAI tools in their course, and clearly explained why.** - 289 responses recorded. - Respondents strongly leaned towards understanding with **39.9% marking agree** and **37.8% marking strongly agree**. - **Q7a: I expect to be able to use GAI tools in any course at GW.** - 262 responses recorded. - Respondents were largely **neutral (37.5%)**, but overall there was a slight lean towards disagreement. - **Q7b: My friends expect to be able to use GAI tools in any course at GW.** - 336 responses recorded. - Respondents were largely **neutral (40.8%)**, but overall there was a slight lean towards agreement. - **Q8: Which of the following best describes your attitude toward the adoption of new technologies?** - 336 responses recorded. - The most selected response was **early majority (50.0%)** and the next highest response was **early adapters (31.8%)**. - **Q9: How important do you think it is for you to learn how to effectively use GAI tools?** - 336 responses recorded. - The spread of responses was rather decent, but the most selected responses was **very important (28.6%)**. - **Q10: I believe GAI is an accurate source of information for my school work.** - 336 responses recorded. - The most selected response was **some of the time (58.6%)**. There was no close second response. - **Q11: How would you rate your ability to use GAI effectively?** - 336 responses recorded. - The most selected response was **intermediate (49.4%)**. There was no close second response. - **Q12: What approach would you like to see GW take regarding the use of GAI tools in classes?** - 336 responses recorded. - Major themes identified included: 1) Regulation and policy; 2) Educational integration; 3) Learning enhancement; 4) Access and usage limits; 5) Technology literacy; and 6) Ethical and responsible use. - A few highlighted responses: - "Prohibit them for courses that require creativity, allow them on courses where they can be helpful to understand concepts or practice, and have special sessions/events that teach people how to best use GAI tools." - "GAI is apart of the future, learning how to use it and understand it is a vaulable part of our education." - "I would like to see professors more openly discuss it's beneficial use and not be it such a taboo thing as if it's only meant for cheating." - "Strict guidelines. I'm tired of seeing AI being used during exams." - "I would like GW to teach us how to use it as a tool for learning, not as a cop-out, because its not something that's going to go away and it could make us even stronger students so I think educated use of it could prove very positive in the future." - "I believe that GWU should embrace the use of GAI tools in classes going forward. It is understandable that there are reservations regarding the usage of GAI tools through forms of cheating, plagiarizing, and slacking off; however, GAI can also be helpful to check your work, understand concepts more completely, and minimize time spent on research." - "It's the technology of the future, so I would like to see GW offer either a class/crash course on effective techniques for using it." - "I like when my professors aren't strictly against it because it is obviously something we have at our disposal. I think most of us use it as something to help us generate ideas or summarize something that may be a bit dense or confusing." - "Since we are still in school, I think it is more important that we learn fundamental skills for ourselves." - "Acknowledgement that AI is not just a way of cheating but can be a supplement to studying, especially in large lecture classes where one on one assistance is hard to get." - **Q13: Is paying for a subscription to a ‘better’ version of GAI tools a barrier for you?** - 336 responses recorded. - Overall the responses were very close between **yes (52.1%)** and **no (47.9%)**. - **Q14a: Your GW School affiliation:** - 336 responses recorded. - The two most common GW colleges/schools the respondents came from were **Elliot School of International Affairs (ESIA) (45.5%)** and **GW School of Business (GWSB) (39.9%)**. - **Q14b: Your class level:** - 336 responses recorded. - The vast majority of respondents were **freshman/first-year (93.2%)**. ## Survey presentation Results of the survey were presented to the [GW Faculty Senate](https://facultysenate.gwu.edu) [Education, Technology, and Policy Committee](https://facultysenate.gwu.edu/committees/educational-policy) on **January 19, 2024**. Slides 1-15 were covered in that time. Slides 16-41 were provided for reference to the full survey to the EPT Committee. <div class="container"><iframe class="responsive-iframe-sd" src="https://1drv.ms/b/c/13829E5D2EB238DE/IQT1CFCDltXdTZSGu9tIndJlAcUAz6bLETZc1g_4NCmB9PI" width="100%" height="400" frameborder="0" scrolling="no"></iframe></div> *Note: These slides were built with a custom slide deck that I made using Microsoft PowerPoint. Generative Artificial Intelligence (genAI) was not used to create these slide. All stock images were provided by [Adobe Stock](https://stock.adobe.com) or [Getty Images](https://www.gettyimages.com). All charts were produced by Google Sheets.* ## Resources - Hadidi, R., & Power, D. (2020). Technology adoption and disruption – Organizational implications for the future of work. Journal of the Midwest Association for Information Systems (JMWAIS), 2020(1), 1. [DOI:1017705/3jmwa.000053](https://jmwais.org/wp-content/uploads/sites/8/2020/01/A1-2020-jan.pdf) - Mollick, E. R. and Mollick, L. *Assigning AI: Seven Approaches for Students, with Prompts* (September 23, 2023). The Wharton School Research Paper, Available at SSRN: [https://ssrn.com/abstract=4475995](https://ssrn.com/abstract=4475995) or [http://dx.doi.org/10.2139/ssrn.4475995](https://dx.doi.org/10.2139/ssrn.4475995)