VR Headset for Endoscopy and Microsurgery Client: Dr. Azam Ahmed Advisor: Mr. Willis Tompkins, Dr. John Puccinelli Team Members: Tom Geissler geissler2@wisc.edu Team Co-Leader Sam Peters speters9@wisc.edu Team Co-Leader Sam Simon ssimon8@wisc.edu Communicator Sam Schini schini@wisc.edu BSAC Jake Cohn jmcohn2@wisc.edu BPAG Josh Niesen jsniesen@wisc.edu BWIG Reporting Period: November 2nd through November 8th, 2018 Problem Statement: Endoscopic surgeries have become increasingly prevalent in the operating room along with the visualization techniques used to perform them. Traditionally, large monitors have been used to display the images from the surgical tools, however, these are bulky and don t provide as immersive of an experience as other methods do. This problem has been partially addressed by using virtual reality headsets. One major limitation of these however, is that they do not allow the surgeon to see anything outside of the operative view. This is a problem as the surgeon will have to remove the headset everytime he or she has to change instruments, or perform macroscopically. The goal of this project is to design a VR headset that would allow the surgeon to transition from the operative view to normal view without the use of the surgeon s hands. Summary of Team Role Accomplishments: - Team: The group again divided into two subgroups: one focusing on the video transfer from the endoscope to Unity, and the other on the view transition from VR to environmental. The subgroups each met at the Makerspace to test code segments on the HTC Vive Pro. The video transfer group succeeded in displaying a 2D video in Unity without lag. - Leaders: Continued to lead subteams in accomplishing our task in the project. - Communicator: Filled out the main paragraphs of the weekly progress report and distributed the document to our client and advisor. - BWIG: The project page was updated with team s progress report and the meeting notes were posted in lab archives. - BPAG: Continued to wait on our proof of concept before we purchase thousands of dollars of equipment for our project. - BSAC: Prepared to attend and contribute opinion to this week s meeting. Design Accomplishments: Each team member got the opportunity to watch the Unity/SteamVR tutorials suggested by Taylor. The entire group again met at the makerspace to test out segments of code we had developed from
researching similar projects and Unity syntax online. The video transfer group had some success in displaying a 2D video and running it without lag in Unity. The video transition group also had success in getting the headset to display a continual feed of its gyroscopic rotation values. Activities: Date Name Task Time (hours) Weekly Total Semester Total 11/7/2018 Team Meet with Taylor to discuss current Unity project status and current display issues 3 3 29.5 10/31/2018 Sam Schini 10/31/2018 Sam Schini Worked with Unity and researched video conversion. Communicated with potential experts to assist with video streaming obstacles. 1 1 20 1 1 5 11/3/2018 Jake Continued watching VR videos and doing general research 1 1 20.5 11/7/2018 Jake Show and Tell Brainstorming 1 2 21.5 11/5/18 Sam Simon Watched VRTK Videos 1 1 26 11/6/2018-11/7/2018 Sam Simon Worked with the Google Cardboard to make a hands-on demonstration for the Show-and-Tell 3 4 29 10/30/2018 Sam Peters Sam Peters Unity development. 2 2 18.5 Unity development. 2 4 20.5 11/2//2018 Josh Worked at the Makerspace and discussed the gyroscope and display development that need to take place with Taylor. 11/8//2018 Josh Worked on progress report updates and updating website 3 3 32.5 0.5 2.5 33 11/3/18 Tom Watched VRTK videos 1 1 24.5 11/8/18 Tom Worked on getting rotation values 2 3 26.5
Statement of Team Goals: Both subgroups will to continue to grow off of this week's accomplishments to bring the headsets functionality up to our client s standards. Although the video transfer group was able to display a video feed, this subgroup will need to find a way to format the video in the Vive Pro headset so the image immersive but not distorted. It is one thing to display a video on the headset that was received from the internet or even one s laptop camera, buth this subgroup will also need to work closely with the information we received about the endoscope to discover how we will display the endoscopic feed in real time. The video transition subgroup, alternatively, will need to craft some code in Unity that will allow the headset to play a Camera-to-endoscope/endoscope-to-camera function when a certain x (vertical) rotation value is read. Individual Goals: - Sam Simon: I will continue working with Tom and Josh in coding the headset to read real type gyroscopic rotation value and initiate a video transition function when a certain x value is read. If we can accomplish this by next week, I will help the rest of the team merge the two subgroups work into one seamless program. - Sam Schini: Next week, I will transition to help the other half of our team find a video transition solution. I will continue to familiarize myself with Unity and research different video formats and compatibility while filling my team in on the discussions of BSAC. - Sam Peters: Next week I will be assisting the transition team with their subproblem. Now that we have completed the video stream problem, we will working on completing the transition and integrating the two sub problems. With the two rough solutions combined, I can start to work on quality optimizations. - Tom Geissler: I spent this past week trying to get the rotation values of the headset in Unity. Next week I will continue on this track and try to incorporate different actions based on these rotation values into our unity code. - Jake Cohn: This week, I want to confirm that we found the solution to our subteam s objective, then we can move on to all start working on the other team s transition mechanism. In addition to helping with the transition mechanism, I want to start working on the Final Report so we can stay ahead of the game. - Josh Niesen: By working this week with Taylor and watching videos, we were able to display the gyroscope values that are being registered from an external camera in a long list from each frame. Moving forward, I hope that more research and development will lead to being able to display the gyroscopic values continuously in time at a single location rather than a list. I also aim to begin working on determining the angle that will serve as the minimum to switch views. Difficulties: After returning to the Makerspace again, our group has come to conclusion that Taylor may not be as useful of a resource as we initially hoped. It has been difficult to persuade Taylor to remain by our while working on the project. He will usually join us for 5 minutes before
recommending a video tutorial that does not help our cause and sprinting back upstairs. Our group will need to compensate by either reaching out to our professional on campus or conducting an even more extensive search on the internet for our information. While this change doesn t leave us without options, it has majorly slowed down our progress. Project Schedule/Timeline: Task September October November December 7 14 21 28 5 12 19 26 2 9 16 23 30 7 14 Project R & D Current Products X X X Scope Based Surgery X X X X Virtual Reality X X X X Mechanical Solutions X X X X X Materials X X X X X Cost Estimation X X X Prototyping X X Documentation Progress Reports X X X X X X X X Preliminary Presentation X X X Preliminary Deliverables X X X Final Poster Final Deliverables Meetings Team X X X X X X X X X Client X X X Advisor X X X X X X X X
Website Update X X X X X X X X X Filled boxes = Projected timeline X = Task was worked on or completed Expenses: We have not incurred any expenses yet.