There is a considerable amount of interest amongst students and residents to do research, publish and be productive, however less than half actually do. While taking a research project from inception to publication can seem like a daunting task it is certainly something almost any student or resident can do. I find that one of the major reasons for someone who is interested in research but is unable to do so is the inertia associated with pursuing something unfamiliar.
The goal of this blog is to provide a framework on how to perform an outcomes research project with further details available in the ‘further reading’ section at the end.
I will use a large dataset analysis as an example. Once you go through one of these projects you will have the skills and knowledge to do a vast array of research projects. Also, large dataset research is ideally suited for students and trainees with limited time. Extensive chart reviews are not required and for the most part you can work on your own time.
First and foremost, as a medical student, resident or fellow further formal training in research is not a necessity to be able to conduct and publish a decent and valuable outcomes study. Do not undervalue what you have already learned (consciously or subconsciously) through classes, seminars, grand rounds, talks, journal clubs etc. The only pre-requisites are the three Ps – Patience, Perseverance, and Persistence. You either gain the rest as you go along or you utilize help from people around you.
Step 1 – The research idea
Even though this often seems like the hardest part, it really is not. Keep your eyes open to your daily observations and your mind open to curiosity and research ideas will appear on their own. The research question often comes while studying, reading articles, discussions with colleagues and faculty and very commonly while preparing M&M slides. The real problem is in figuring out whether it is feasible and will add to the body of literature. Here is where experience is helpful and in lieu of that – talking to a colleague or mentor will often lead you in the right direction. One general principle is to refine your idea in to something that is comparative rather than descriptive, for example do not ask what the incidence of ureteral injury is after a laparoscopic sigmoid colectomy, ask if ureteral injury is more likely if the surgery is done laparoscopically or open, among obese versus non-obese patients, or for inflammatory conditions versus malignant ones etcetera.
- At this stage also begin thinking about which meeting you want to submit your abstract to. Unfortunately, with all our clinical responsibilities it is very easy to keep putting things off. A hard deadline like an abstract submission deadline will goad your project to completion. You can find out about abstract deadlines by talking to your mentors or searching for important meetings in that specialty and looking at when the last years abstract deadline was. Almost invariably the deadline is about the same time every year. (The Academic Surgical Congress is an excellent place to submit your research to – abstract deadline is in near the end of August of every year). Your deadline for the abstract first draft should be one month before abstract deadline.
Step 2 – Literature search
This often goes along with step 1 and not really after it. While you are pondering over your research question the first thing you need to do is to try and answer the question yourself. Read your books and search the literature.
- Make sure your search is exhaustive – you do not want to spend substantial time and effort only to realize that your research was already done last year, in fact done better. A proper literature search should take substantial time, it is the single most important aspect of your research and this is not the time to cut corners. More often than not you will find that your idea is already answered. But do not be disheartened- you know the process- just keep doing it till you find a nice doable, publishable idea! Additionally, no effort goes wasted – you will learn a lot from the process and also come up with further research ideas. Often you can tweak your research idea just a little to make it novel.
- Once done, take a break and then try searching again. Involving others in the literature search can often prevent fatal errors.
- During the literature search, refine your question further – think of the population, the operation(s), exposures, and outcomes.
- Sometimes your exact question is not done, or was done differently or inadequately – do not give up easily – there is often much room for investigation.
- Use a reference manager. Once in the meat of the literature search using a reference manager like Endnote or Reworks will help keep you organized – and will save much time at the manuscript writing phase. If you have not used these software before- have no fear! The software is easier to understand than the quirks of your attending. There are numerous free tutorials online. Talk to your librarian – they love helping with this kind of stuff.
Step 3 – Choosing your dataset
If you are at an institution with a high volume of cases in your proposed research area you may benefit from a chart review type of study design. Your institution may have already kept a prospective dataset of what you want to study so talking to colleagues and mentors will help. For many outcomes based research projects, large database analysis is very fruitful – especially for infrequent conditions or outcomes. These datasets are essentially spreadsheets of information collected by various organization for administrative, quality improvement, or research purposes. Often, they are aggregated data collected from several institutions.
- First make a list of variables you want to study. This should include your outcome of interest as well as your predictors and confounders – the variables you want to control for. This list is generated from your research idea and from your literature search. This list can be refined and revised as you go along.
- Find your dataset. There are a number of datasets commonly used for research pertaining to surgical outcomes. You can find out more about these datasets and what they are best used for from the literature or from speaking with your mentors and collaborators.
- Each dataset has a user manual that is freely available and downloadable. Read through this document to see what the dataset is about and what it can be used for. The document also contains a list of variables with their description. Familiarize yourself with these and use this to decide whether this is the most appropriate dataset for your research.
- Familiarize yourself with codes – ICD 9 diagnosis codes, procedure codes, and CPT codes hold a tremendous amount of information, and can be very useful to your research. Descriptions of these codes can be found online.
- Can it be done? You may not be able to find a dataset that has all the variables you want – in fact this is the more likely scenario. This is not fatal to the study and may just be a limitation. Needless to say, your outcome variable and predictor(s) of interest are critical. You and your mentor will have to decide whether the study can still go one without the missing variables.
Step 4 – IRB.
Almost all dataset are stripped of identifying information and HIPAA compliant. IRB approval is never an issue, however, the process is necessary and requires some paperwork. The only tip here is to keep the IRB application broad so that multiple future studies can be performed under the same approval. For example, something like ‘the national trauma database will be used to study outcomes in trauma patients. Multiple research projects will be conducted using this database around this theme’. IRB approval can take a week to a month – but there is much work that can be done while waiting for the IRB. You can complete your literature search, begin drafting an introduction, start planning your analysis, plan your next study, or begin acquiring the dataset itself.
Step 5 – Acquiring the dataset
To perform the analysis you will need to obtain the dataset or obtain access to it. Different datasets require different resources. Again, the user manual will direct you on how to acquire the dataset. Some are publicly available with a short online form, while some require more paperwork. Some require you to buy the dataset, others are free. Talk to your chair and convince him/her to buy the dataset for the department. Ideally the dataset should be downloaded onto a departmental server that can be accessed securely by other members of the department, or by you when needed again. Many datasets are relatively small files and easy to manage, however some datasets come with multiple large files that require larger storage space and computational power. The user file usually gives this information. But also talking to people who have used the dataset before will give you an idea of the ‘user friendliness’ of the dataset. Another often easier way to acquire the dataset is to collaborate with someone in your vicinity that has access to it. Discuss your idea with them and ask them if they would be interested in being a part of the research. Often their expertise in the field and familiarity with the dataset will contribute immensely to your research project.
Step 6- Analysis
Great! so now you have the data, the IRB is done and you know what you to do. The analysis part can be tricky depending on your prior level of skill. Definitely seek statistical help – find a biostatistician you have access to via your department/hospital/university/institution etc, or a colleague who is proficient at analysis. Even though there are many easy to understand online tutorials freely available – you definitely do not have time to learn this during the clinical years of residency. If you are taking time off for research then certainly learn to do the analysis by yourself – your first project may take eons but the time and effort investment upfront will pay off and your subsequent research papers will be much quicker.
- Sit down with your statistician and discuss with them what you want to do. They will suggest the best way to do it. You may have to tweak the variables a few times and the analysis can sometimes be an iterative process. This is where your expert knowledge and your literature search helps.
- The analysis is key part of an outcomes study. It really is the meat of it. Try to understand the analysis, ask questions.
- This part may also be time consuming if you do not do it yourself, however you can increase the efficiency by scheduling regular meetings with your statistician and responding quickly to the requests. Try to have a plan ready in the beginning and stick to it so there is not much back and forth. Follow up and follow through – don’t waste your time and don’t waste your statisticians time.
- The usual analysis steps are: (i) Data cleaning -getting your variables ready for analysis. (ii) Selecting your patient population – rarely is the entire dataset used for research, you have to decide on who to include and exclude (ages, operations, diagnoses etc ) (iii) Descriptive analysis- basic proportions and summary statisitcs of your variables. (iv) Univariate analysis (v) Multivariate analysis, and (vi) Post model testing if required (some tests that measure the appropriateness of your analysis). In general the statistician will do the analysis and should suggest the most appropriate statistical tests
Step 7 – Understanding the results
This step is more crucial than the analysis itself. The onus really is on you to look carefully at your results and understand what the output tables are saying. For the most part this will be very easy for you. For more complicated analyses you can meet with your statisticians who are generally very helpful at explaining what it all means (make notes, one forgets this very easily).
- A careful review of the descriptive tables is very prudent. You may also pick up some issues with some variables that you will need to help the statistician fix, for example while looking at the descriptive tables you notice that 85% of your laparoscopic cholecystectomies had length of hospital stays of 15 days or more. If you think that this cannot be right you should ask the statistician to go back and double check the coding. Something must have been done incorrectly (in this case choosing only patients with acute acalculous cholecystitis instead of acute calculous cholecystitis). At this stage also take a look at how much missing information is present in each of the variables and decide on how to handle it in the analysis – discuss with your statistician.
- Ideally you want to go step by step i.e. discuss the descriptive analysis and fix any issues before proceeding with the comparative analyses.
Step 8 – Writing the abstract
So now you have enough material to write up your abstract. I always find it helpful to first write an outline and then focus on drafting each section. For the outline just write (or scribble), in bullets, what you want to say in your introduction, what was your objective, the important parts of the methods, and then the main results and conclusions. You can add in the numbers, percentages etc. as you go along or later. The actual sentences come with the first draft and can be revised and polished until the submission deadline.
- While you can write the manuscript first and directly submit to a journal, I do highly encourage you to submit and present your work at a meeting. Presenting at a meeting means you will attend at least part of the program. This has so much added value. You will interact with like-minded people, hear inspirational talks, can network and find mentors. All of that helps tremendously in your path to becoming an academic surgeon.
- Manuscript writing texts will say that the abstract should generally be written after the manuscript. Either approach is fine but if you are targeting a meeting all they want is the abstract to begin with. The manuscript can come afterwards.
Step 9 – Manuscript writing
Do not underestimate the transition from step 8 to step 9. This is where several people fall off the boat. It is a matter of commitment. You have to be patient, persistent and persevere through it. Just remember that you have done all this work and you just have one final step before you can submit. And yes, just submitting an abstract and presenting is not enough. Publish everything you do to make it really count! There are several useful articles on how to write a manuscript.
- Writing, like anything else is a skill which improves with practice. So, if you sit down in front of your computer and find yourself struggling to write, rewrite, refine, polish the same sentence for the past hour – just know that this is normal. Some tips include; Start with the easy stuff -you can either start writing the methods section first or start making your results tables. Do not pay too much attention to sentences themselves, those can always be refined later. If you have coauthors and collaborators you can divide up the sections, work simultaneously and review each other’s work.
- Writing your first manuscript takes time- chip away at it slowly and steadily. If you feel you are not making progress with one section or paragraph – take a break, move to something else and then come back to it later. For the most part you can write any paragraph first. For example, you may find it easier to just write out the limitations paragraphs. It is like placing the pieces of a puzzle on the board. You can link up the pieces later. The important thing is to make progress (get the pieces down)
- Reading aids writing! To write a good introduction- read 10 good introductions before you start to write. For example, choose a good journal and read the introduction section of 10 random articles that are of interest to you. Similarly for each section.
Step 10 – Reviewing
I find this part to be the most tedious and by now really just want to submit the manuscript and be done with it. However, the reviewing stage is an important step that should not be omitted. Remember the 3 Ps (Patience, Persistence, and Perseverance) and chug along. At this stage you have to read, revise, re-write and sometimes (painfully) even redo the analysis if a co-author picked up on something you had missed before. However, it all works out. Every little enhancement of the manuscript helps. Have someone else copyedit the manuscript for you – by now your eyes will be blind to some typographical errors.
Step 11 – Publication
If you are at this stage you have already won! Pat yourself on the back and do the Makarina. The publication process is much simpler than the prior 10 steps, however it is a process. First you have to choose an appropriate journal to submit to. Then look at that journals ‘instructions to authors’ document and format your manuscript perfectly. You have to write a cover letter and sometimes get conflict of interest statements from your co-authors. The submission process itself is easy to understand and self-explanatory. After you submit your manuscript it will be reviewed by 2-4 reviewers and will either be accepted, rejected, or accepted with revisions. If accepted there may be some minor revisions – do them quickly and resubmit. If rejected don’t be disheartened – go for a run or do something that soothes you and then plan to submit to another journal –usually there is some aspect you can revise to strengthen the manuscript. The comments from the reviewers will help. If your article gets accepted with revisions. Take a break then read the reviewer comments again. Break it down and start revising one point at a time. Keep track of all the changes you make in a separate document and send this with your revised manuscript. The review process can take from 3 to 6 months and the actual publication can also take another 6 months. But be patient. Soon enough you will see your work published in a scientific journal. The appropriate dance here is the Fred Astaire (https://img.buzzfeed.com/buzzfeed-static/static/2014-12/5/16/enhanced/webdr01/anigif_original-grid-image-24797-1417815671-31.gif?crop=323:488;0,0)
With that I hope you have an overview of the whole process from start to finish. If you are feeling overwhelmed, that is ok. Looking at whole picture does that – take it one step at a time. Remember that this is something you can totally do with the skills you have, you just have to be persistent, have patience and persevere through it. Research is a really fun process and is often very interesting to see how your idea unfold and matures in to a manuscript. Along the way you learn so much and develop a variety of relationships.
Some Further tips on efficiency
- While formal research training is not necessary it certainly increases efficiency and speed. If you can do some or all of your own analysis-that really does save a lot of time.
- Attending courses like the AAS Fundamentals of Research Course can give you a jump start.
- Other courses offered by your library (Refworks, SPSS, Manuscript writing etc) will also increase your efficiency and quality.
- Everything you need to know is online and free – you just have to look for it.
- Stay focused and keep your team focused – it is very easy to deviate as once you are in the meat of it good ideas will start pouring in. Branching and exploring other related research ideas is great – but finish one then do the next.
- Great mentorship trumps all! -actively find good mentors, but don’t be stagnant while looking. Your mentor does not have to be at your primary institution.
- Collaborating adds quality, increases productivity and makes it all the more fun. Include more people and you will find it easier to complete projects. Although, chose your collaborators wisely.
Best of luck,
Stay curious
“It isn’t the mountains ahead to climb that wear you out; it’s the pebble in your shoe.” ~ Mohammad Ali
Further Readings
Success in Academic Surgery: Health Services Research
Editors: Dimick, Justin B., Greenberg, Caprice C. (Eds.)
http://www.springer.com/us/book/9781447147176
Research methodology, a step by step guide for beginners
Ranjit Kumar
https://www.amazon.com/Research-Methodology-Step-Step-Beginners/dp/1446269973/ref=dp_ob_title_bk?dpID=51qwBiZDKwL&preST=_SX218_BO1,204,203,200_QL40_&dpSrc=detail
Writing a manuscript
https://www.elsevier.com/connect/11-steps-to-structuring-a-science-paper-editors-will-take-seriously
Surgical meetings and deadlines
https://ww2.mc.vanderbilt.edu/gsr/28495
The 11 Steps to Conducting Outcomes Research
