Announcing CovidMine – analyse integrated COVID genomic and geographical distribution data

We’re excited to announce that a project we’ve been working on for the last few weeks is ready for public consumption: CovidMine, an InterMine dedicated to COVID-19 / SARS-CoV-2 data. Data is updated on a daily basis Monday-Saturday at 6PM UK time. You can try CovidMine out now, or read more about it below. 

So, what’s it all about, and why another COVID resource? 

This is something we thought about a lot, initially – there have been a massive number of initiatives going into making data available and visualising it already. In the end it came down to a couple of reasonably simple facts: InterMine already has tools to draw data from a lot of sources and integrate it, but it also offers a familiar interface if you’ve used any of the other InterMines out there, and we have API language bindings for multiple programming languages, including R, Python, Perl, and Javascript

Data sources include confirmed Covid-19 cases, deaths, new confirmed cases and new deaths for countries from Our World In Data1, data separately for individual states (for the United States only) from the COVID Tracking Project2, Sars-CoV-2 reference genome3 and nucleotide sequences from isolates deposited in Genbank4.

If you’re aware of other data sets that might make this more useful please contact us to suggest them.

Jump straight in

We’ve prepared a few template queries to help you get started with your analysis –

What’s still missing and how can I help? 

We’re officially focusing our efforts on developing tools for CovidMine in our new user interface, BlueGenes, rather than the legacy JSP interface. 

A few things we’d like to add to the UI:

  • A data visualisation showing all results on a map.
  • A visualisation that shows change over time in countries or regions, for known cases, recovered, and deaths. 
  • A genome browser (JBrowse 1)

These visualisations would update based on the filters in the table showing in your data

Data updates: 

  • Find and integrate a data source which provides China data separately for individual states

Bioschemas Markup

We have applied structured data in JSON-LD format, using the Bioschemas.org profiles DataSet, Gene and Protein. It’s available in the legacy JSP interface only, but it will be integrated in the new interface soon.

If you’re aware of other data sets that might make this more useful, or other visualisations that might be exciting, please contact us to suggest them! 

References:

  1. https://covidtracking.com
  2. https://covid.ourworldindata.org
  3. https://www.ncbi.nlm.nih.gov/genbank/sars-cov-2-seqs/#reference-genome
  4. https://www.ncbi.nlm.nih.gov/labs/virus/vssi/#/virus?SeqType_s=Nucleotide&VirusLineage_ss=SARS-CoV-2,%20taxid:2697049

Roshni Prajapati on BlueGenes UX, user research, and saving people from bad design

This is our blog series interviewing our 2020 Outreachy interns, who are working remotely for InterMine for 3 months on a variety of projects. We’ve interviewed Roshni Prajapati, who will be working on UX research and recommendations for BlueGenes.

Hi Roshni! We’re really excited to have you on board as part of the team this summer. Can you introduce yourself? 

Hi team, can’t deny the wait for a totally new experience is driving me crazy here! I’m an IT undergraduate pursuing bachelors of Technology at IIIT Allahabad and will be onboarded to IIIrd year from August onwards. Primarily my interest lies in interaction design, user research, product thinking and a bit of graphics design still I don’t mind banging lines of code to build stuff that interest me. Few of my works could be seen here & here.

Some days I try to solve user issues by merging aesthetics, a bit mathematics & data in unequal proportions while other days I can be spotted preparing for my upcoming hackathon, lying all day watching cartoons or enjoying 70s-80s classical playlist. 

    Other than this I’m a wanderlust person, a guitarist, a painter, an intermediate football and Table Tennis player and a coffee addict 😛 

What interested you about Outreachy with InterMine?

I have this craving of improving things to redefine work for living breathing humans i.e,  to save them from bad design. Case with InterMine is that while surfing through the mine-sites I noticed it mostly comprises analytical data and their representation. The current website has several user issues & pain points, also naive look and presentation of the data is not apt and even violates some design rules. This made me dive deeper into the real world biodata and their visualization for better usability of the website added the fact that the organization itself registered a design issue (driving me more to work).

    One of the facts is that design analysis needs views from users and developers and it becomes important that the community interacts. So I needed a better understanding of the real world bio data (new to me) and mentors willingly helped, this everready response brought an optimistic vibe to work for the team and organization.

Tell us about the project you’re planning to do for InterMine this summer.

The content layout in the current website design needs to be strategically placed in order to make it easier for users to go through. Since the site contains heavy analytical and a variety of biological data, my task will be to organize the website content such that users can find the things at ease, improving overall user experience. So basically I would try to carry out my process in following phases-

Discover & Define: Carry out questionnaire sessions and meetings for collecting user experience observations then interpret the observations and define insights. I will try to convey my ideas through user personas & stories and finally set my design challenges.

Develop & Deliver: and further will discuss ideas and through sketching and experimenting and prototyping by working on feedback iteratively.

Are there any challenges you anticipate for your project? How do you plan to overcome them?

Previously I have worked on several projects of my own and this would be the first time I would be working with a community. So collecting user experience observations remotely through unit testing and other methods is gonna be quite a challenge for me. One of the major tasks also includes my contribution in implementation of design of which I’m concerned. Since this is gonna take some time, it could be counted as another challenge still I’m pretty much sure that work would get done under the time duration provided 🙂 

Share a meme or gif that represents your project

Outreachy Interview: Qian on the InterMine Training Portal

This is our blog series interviewing our 2020 Outreachy interns, who are working remotely for InterMine for 3 months on a variety of projects. We’ve interviewed Qian, who will be working on the InterMine Training Portal.

Hi Qian! We’re really excited to have you on board as part of the team this summer. Can you introduce yourself? 

Hi InterMine team! I feel so excited to be part of the team! I am a Computer Science undergraduate from National University of Singapore. Next semester I will be a sophomore. I transferred from Shanghai Jiao Tong University to NUS last year. Previously, I majored in chemistry and biology. So I have some biology background. I feel so happy to take advantage of both my major backgrounds to contribute to InterMine!

I did an HTML-related project last semester. This is the link. This is the pr I cooperated with my partner. We dealt with generating an HTML textbook in seconds which is our introductory book to programming. I learned Java and JavaScript this year. I also helped a professor to deal with huge data using Jupyter Notebooks last semester, from which I learned python. (To be frank, python is much easier than Java. I was so frustrated by Java final :(. )

I am a newbie in computer science. I feel very lucky there are many open CS resources  to help me learn basic concepts. And, open source platforms are good places to find people with similar interests. I learned much from InterMine discord chat!

Apart from studying, I love playing the piano. My biggest goal is to be able to play La campanella fluently. This is my favorite version.

What interested you about Outreachy with InterMine?

Among many projects, this project almost does not restrict anything. I can design the portal according to my own ideas (of course I get advice from mentors). This is my first internship so I want to try to be more flexible.

Another reason is InterMine is about biology data processing. I have a biology background and I am interested in genes. I think cooperating with computer science, especially data analysis, is the future of biology.

Further, I love the atmosphere in this community. Yo is a good mentor as she is so helpful and kind. Members in InterMine are all warm-hearted and enthusiastic about new ideas. I learned much during the application period, especially during this hard period. I got mental relaxation when communicating with people in InterMine!

Tell us about the project you’re planning to do for InterMine this summer.

  1. Change the layout of the training portal page to be more useful and beautiful. 
  2. Make text and video tutorials for different languages.
  3. Add some features to the page.
  4. Combine different tutorials together. 

Are there any challenges you anticipate for your project? How do you plan to overcome them?

I have to rewrite tutorials in different languages which I am not very familiar with. So I am learning Perl and R these days! 

Another challenge I think I will meet is I am not sure about the time arrangement. As this is my first internship, I don’t have experience in arranging a schedule by myself previously. I hope to finish as I wrote in my plan. To overcome it, I am going to get advice from my mentors and volunteers. As this is a 3-month internship, I think I can have better anticipation of productivity with the guidance of mentors after 2 or 3 weeks. Then I will adjust my plan timely.

Share a meme or gif that represents your project!

Announcing the InterMine 2020 Interns 🚀

Announcing the InterMine 2020 Interns! 🚀

As we mentioned in an earlier blog post, this year InterMine is participating in Outreachy for our May-August (northern hemisphere summer) internships. This year we’ll have five Outreachy interns, as well as a couple of in-house interns working with us. Please give them all a huge round of congratulations! 

Our interns and their projects 👩‍💻👨‍💻

InterMine training portal – Qian

Qian will be working on the InterMine training resources, creating new programmer and UI-oriented tutorials and updating/organising existing tutorials. 

Qian will be mentored by Yo and Asher. 

CLI tool for managing InterMine instances – Pooja Gaur

Contributing to the InterMine cloud project, Pooja will be orchestrating better automated builds for InterMines that will also work nicely in Travis or other CI. 

Pooja will be mentored by Kevin and Ank.

Improving the UX and UI of BlueGenes – Roshni Prajapati

BlueGenes, the new InterMine UI has improved drastically since Kevin joined the team and began adding and improving features. Roshni will conduct research into the usability and UX of BlueGenes, and redesign / improve interfaces where needs are identified. 

Roshni will be mentored by Yo and Kevin. 

Improving the InterMine Data Browser – John Mendez

The InterMine Data Browser was a 2018 GSoC project led by Adrián, designed to make it easier for people new to Intermine to learn and explore what types of data are present in an InterMine. John will be helping update the Data Browser with new features and migrate it to React. 

John will be mentored by Adrián, Nikhil, and Aman. 

Javascript Data Visualisations – Sakshi Srivastava

Last year, Akshat Bhargava created a suite of data visualisation tools for BlueGenes. Sakshi will be extending this work, refining some of the Tool API specifications and adding new data visualisations. 

Sakshi will be mentored by Akshat,  Kevin, and Asher. 

With thanks to our sponsors

We’re grateful to the Wellcome Trust for awarding a Diversity Enrichment grant that enabled us to sponsor three Outreachy interns, as well as Outreachy themselves who were able to sponsor another two internship positions.  

In-house interns  

We’ll also be working with Ank Kumar (working on InterMine Cloud related projects) and Celia Sanchez Laorden (InterMineR). 

What about GSoC? 

In previous years we also participated in Google Summer of Code – this year InterMine wasn’t accepted as a mentoring organisation. It is common for GSoC to “rest” applicants occasionally, so we’ll continue to apply and hopefully we will be back in GSoC next year.

Once we knew we weren’t accepted into GSoC for 2020 we joined the Open Bioinformatics Foundation GSoC org, who act as an umbrella organisation for GSoC mentoring organisations, and we had several strong applicants apply via the OBF’s organisation. Unfortunately the number of slots awarded to the OBF was lower than the number of promising students the OBF was ready to accept. This meant that InterMine was unable to take on any GSoC students, even though there were high-quality applications.

New FlyMine build: FlyMine 49, now with added JBrowse!

FlyMine has been updated to the latest version of FlyBase. All other data sets have also been updated to the newest versions, we have fixed a few bugs and made the JBrowse displayer available again. See the data sources page for a full list of the current datasets and their versions.

All data can be accessed through our comprehensive library of template searches or by building your own queries using the query builder.

If you have any questions, please see our docs and do not hesitate
to contact us should you require any further assistance. For help and
feedback please email support@intermine.org or visit our contact page.

[Post originally drafted by Sergio Contrino]

Routine Maintenance: FlyMine, 10:00-12:00 GMT 18 Feb

Update, 13:00 GMT 18 Feb: All up and working – if you spot any issues please let us know (but we hope there are none! 🚀)

FlyMine will be offline on Tuesday 18th February 2020 for server maintenance work.
Estimated downtime is 2 hours, starting at 10.00 GMT (check your timezone) until around noon (check in your timezone).

We’ll update this post when we are back up as normal.

Routine Maintenance for HumanMine – Wednesday 8 Jan 2020

Update, 11 AM GMT January 8: The migration has been completed successfully!

Original post:

On Wednesday the 8th of January, we’ll be upgrading the server HumanMine lives on, as well as updating it with a fresh new data build – all the same data sources as before, but more recent versions where they are available.

We anticipate downtime for around two hours, from 10:00 GMT to 12:00 noon GMT, although it is possible the downtime will be longer or shorter than this.

The new dataset versions will be available on the HumanMine dataset page after the upgrade is complete.

InterMine internships plans for 2020: GSoC, Outreachy, other?

TL;DR: InterMine has been participating in Google Summer of Code (GSoC) for three fantastic years now, and we’re hoping to participate in a fourth, with a few improvements. We’ll be applying for GSoC again, but probably taking on fewer students than previous years if we are accepted as an org. This is because we also plan to participate in Outreachy, another mentoring program that runs on the same timeline. We’ll probably take on around six interns in total between both programs. 

What is Outreachy? Why the change? 

In short, it’s a chance to include a more diverse pool of InterMine interns.

It’s no secret that there’s poor diversity in tech generally and particularly in open source: open source software developers self-report to be over 90% male and 15% or less self-report as BME (https://osf.io/preprints/socarxiv/qps53). Similar problems are present for staff and students in higher education, both in gender balance and ethnic balance (https://www.ecu.ac.uk/about-us/he-equality-challenges).

It’s not surprising, then, that while we are pleased to have welcomed interns from India, USA, Ecuador, Sri Lanka, Greece, Spain, and Canada in our last three years of GSoC, only two of our seventeen interns were female.

Outreachy is a program similar to Google Summer of Code, but with a focus on getting more underrepresented folks into tech. The timeline is broadly the same, and our commitment to supporting internships has not changed, so we hope this will help us include interns from a broader range of backgrounds.

Looking to help out? One of the biggest ways you can help us is by spreading the word – share this blog post with anyone you know who is interested in tech and comes from an under-represented community, and/or is looking for experience in tech projects that don’t fall under the umbrella of “code” projects (see the note about Outreachy-only internship topics in the next section). 

Potential student/intern Frequently Asked Questions:

Q: Should I choose Outreachy or GSoC? 

If you were already considering participating in GSoC, we’d suggest staying with GSoC. You can read about eligibility for Outreachy in the Outreachy application guide, too. Outreachy interns don’t have to be students, and could instead be recent graduates, parents returning to work after a break, people who know how to code looking for early experience, or perhaps something else. Another difference is that Outreachy allows tech-related projects that aren’t code – e.g. design, user experience, documentation, or accessibility. Ultimately, your choice should depend on which of the two programs you’re eligible for and which projects you’re interested in, since some projects will be Outreachy-only. 

Q: I’m keen to get started! What do I do?

🎉 Amazing! Take a look at the InterMine GSoC site (we’ll update it to cover Outreachy soon) and InterMine contributing guide and spend a little while learning about InterMine. You can pick up an issue or two if you’d like, but it’s not mandatory for GSoC students. (Outreachy interns need to make at least one small contribution during the contribution period).

It’s also a nice idea to read some successful project proposals from previous years, like this one from Nupur Gunwant

Q: Do I need to get started right away?

A: It’s great to see your enthusiasm! We won’t know for sure until February 20th 2020 whether or not we’re accepted to GSoC as a mentor organisation, and Outreachy applications won’t open until late January. Feel free to sit back with a nice drink and relax until then – seriously! Our project idea list isn’t out yet, and we don’t grade project proposals based on the number of contributions.

Read through our criteria for proposal grading here: intermine.org/gsoc/guidance/grading-criteria-2019/#experience – some evidence of coding is necessary, and you’ll need to understand InterMine well enough to write a sensible project proposal, but beyond that, we don’t want you to work to exhaustion! Well-rested and happy is much more fun 😉

One thing that is really helpful is if you like to hang out in chat (chat.intermine.org) – you can welcome newbies into the #gsoc channel, and answer some of their basic questions. Teamworking and community skills are highly valuable! 

Levelling up: From GSoC student to mentor

We’re really proud of our ongoing engagement with GSoC students from previous years, and we always encourage our students to stay involved in any way that suits them, from writing papers about their work, summer internships in the office, and even joining the team. Here, we’ve interviewed Aman Dwivedi, Arunan Sugunakumar, and Adrián Rodríguez-Bazaga, all of whom were mentors in 2019, but came from the special perspective of having been InterMine students in 2018. It’s not long at all until we’ll be thinking about GSoC 2020! 

Hi all – thanks for volunteering to be interviewed! What motivated you to return as a mentor after having been a student?

Adrián: As a result of being a student under InterMine umbrella during GSoC 2018, I got invaluable skills that contributed towards my professional career, and eventually to getting a job at the mentor organization itself. One of these skills is the ability to communicate, cooperate, and in general terms, to work with a software development organization in an international setting. This is a highly demanded skill – both in industry and in academia – that I couldn’t really get anywhere else before GSoC. 

On the second hand, the opportunity to learn how to contribute back into an open-source with a (huge) codebase and a decent number of contributors, both with code and ideas, was a unique chance to add this top-tier ability to my skill-stack. For this reason, since the high impact that GSoC had in my career, I wanted to go back and help other prospective students by mentoring them and sharing my experience, something that my current position at InterMine helped to contribute positively.

Aman: As a developer, I think we often use open source software and we don’t really get a chance to give back to the community. It becomes difficult to keep contributing to open source in our day to day professional work. Being a part of GSoC in the past, I have realised the importance of open source projects and the communities running them. Returning as a mentor for GSoC this year gave me a reason and a chance to contribute again. I always wanted to be a part of the GSoC journey again and this gave me an opportunity to welcome new contributors to the community.

Arunan: Being part of an organisation which is on the other side of the planet is always an exciting thing to do. I understood the full meaning of the term ‘Globalization’ when I was a student at InterMine last year, thanks to GSOC. I loved our meetings, guidance I received, the project outcome and the level of satisfaction I got. I wanted to have the same experience again this year as a mentor with the organisation I am familiar with.

Did you feel like you had any special insights into what students were going through, having been in the same position in previous years?

Adrián: Having been in the same situation as the students were during GSoC, was indeed very helpful to find and understand the potential needs that they might have. As a matter of illustration, one of the difficulties that is common within already-accepted GSoC students, is that when they face issues in terms of how to continue their progress through the program – either in terms of how to fix obstacles that they might find or contributing with new features – they often don’t feel “brave” enough to communicate with the mentor in order to ask about those problems directly, but instead prefer to find their way through independently, as maybe some of them feel that asking on how to proceed/fix something is a “signal of  lack of knowledge”, and in my opinion this is totally wrong, as mentors are there precisely to help you get around these situations!

Aman: From being a GSoC student to stepping into the shoes of a GSoC mentor, I already was aware of the problems faced by a student. Being a first time contributor in an open source organisation is just like entering a room full of unknown people. Sometimes the student might not know when to ask for help or feedback. Communication becomes the main barrier in such cases.

Arunan: As a student, the hardest part was selecting an organisation and working with them before submitting a proposal. GSoC has gained more and more popularity over the years and the competition is very tough. This might discourage many students and they might postpone their idea of participating in GSoC to the following year. Students should learn to overcome this fear and start trying. Once you have passed a threshold point of getting to know the organisation, the path becomes clear and easy. Once you reach this point, you get all the motivation in the world to start and complete the project because it is an exciting journey.

What advice would you give to a student who is applying for GSoC? Is there something you’d go back and tell yourself when you were a student? 

Adrián: In my view, and re-iterating what I’ve stated in my answer to the previous question, I encourage students to communicate with mentors constantly, and ask about any issue that may arise during the program, while still keeping a high degree of independence.

Aman: GSoC is about open source communities. The student should keep in mind that his/her code would be used by a lot of people all over the world. Each and every aspect of the student’s work has a great impact on a lot of people and a lot of dependent projects. With this thought, comes a great responsibility of ownership. The student should work passionately and should ask for feedback and suggestions from other community members to enhance his/her work.

What tips would you give to first-time mentors? 

Adrián: For first-time mentors, I strongly advise to be proficient enough with the tech stack and have a clear idea of what the desired output from the project is – especially if the project has not been proposed by you, so that you are able to guide the student through the program. In addition to that, make sure to continuously be in close communication with at least one senior mentor in the organization, so that any arising matters can be cleared.

Aman: Mentors should understand the project thoroughly. Understanding the various components of the project is extremely necessary. One should be in sync with the core team of the organisation and should discuss about the expectations from the project. Selection of students is the most important part of GSoC. It is always better to discuss about the various students with the other team members before coming on to the final selection.

Arunan: Mentoring might seem hard especially if you are not part of the internal InterMine team. But if you are comfortable with the project and the tech stack, then mentoring wouldn’t be a problem. Mentors needs to be up-to-date on the project all the time and should have some patience when the student struggles. If you are a first time mentor, it is better to co-mentor with a person who is in the internal InterMine team so that decision making can be easy and aligns with the future work of the organisation.

Interested in participating as a mentor or student yourself?

Mentoring: If you’re interested in mentoring, please email yo@intermine.org to discuss your project ideas. Generally we expect mentors to be known to us and/or have had some involvement in the InterMine community before participating as a mentor. You can also read through our Guidance for Mentors.

Interested student / intern: Check out our guide for students applicants. In 2020 we may well be participating in Outreachy as well as GSoC – so you don’t have to be a student to apply!

Call recording available: GSoC 2019 Final Presentations

Our Google Summer of Code students presented their work at a special edition of the community call yesterday. You can catch up on the entire recording on YouTube – or scroll down to see individual presentations. The agenda and notes accompanying the call (including code and slides links) is in Google Docs.

Prabodh Kotasthane – Spring Migration

Prabodh’s presentations starts at 3:54: https://youtu.be/ZzV6JmVRQmA?t=234

Slides

Ankur Kumar – InterMine Cloud

Ank’s presentation starts at 13:12: https://youtu.be/ZzV6JmVRQmA?t=792

Laksh Singla – Upgrading imjs & im-tables

Laksh’s presentation starts at 21:08: https://youtu.be/ZzV6JmVRQmA?t=1268

Rahul Yadav – Single Sign-In

Rahul’s presentation starts at 27:39 https://youtu.be/ZzV6JmVRQmA?t=1659

Deepak Kumar – InterMine Schema Validator

Deepak’s presentation starts at 24:11 https://youtu.be/ZzV6JmVRQmA?t=2051

Akshat Bhargava – Data Visualisations

Akshat’s presentation starts at 41:30 https://youtu.be/ZzV6JmVRQmA?t=2490