InterMine 2.0

We are excited to announce the official release of InterMine 2.0!

InterMine 2.0 includes some model updates, a big change in how InterMine itself is built, lots of new features, like a new UI, and a long list bug fixes. See the full list of updates here.

This release represents a large milestone for the InterMine team! Not only because we made big fundamental changes to the core InterMine data model and build system, but also because this release represents a major shift in philosophy for us. Previously InterMine was a big, monolithic, single piece of software. You downloaded the whole InterMine, you compiled the whole InterMine, you got the whole of InterMine. Instead, we are moving towards this idea of modularity and responsiveness. Smaller, independent libraries that are interconnected but can be used for tools and features separately or linked together.

Smaller decoupled InterMine packages will allow us to develop more features faster with less errors. InterMine maintainers might then have the flexibility to include (or not) the features in their mine, plug in their own tools, etc.

Version 2.0 represents a big step towards this goal!

A New Interface

A new feature in InterMine 2.0 is the ability to run our new UI, nicknamed “Blue Genes”. This app is in addition to the current webapp and offers a new and responsive search environment for your InterMine data.

Blue genes is a modern UI built in Clojure and provides a modern user experience.

  • Super fast response times
  • Interactive list upload
  • Redesigned “My account” section
  • Search autocomplete
  • Template and query builder result previews
  • .. lots more!

Once you have your InterMine updated to InterMine 2.0, there is a single command that will launch Blue Genes for your mine.

We are actively seeking feedback on Blue Genes, it’s still very much in the beta phase still, so please get in touch once you have some opinions!

Special Thanks

Thanks to everyone who helped test this release! Thanks Howie Motenko at MGI for your alpha testing and model insights. And a BIG thank you goes to Sam Hokin from the NCGR who spent a lot of time and effort helping improve InterMine! Thanks Sam and Howie! You are much appreciated.

Helpful Links

What exactly we changed (blog post)

Full list of GitHub tickets included this release

Docs on how to upgrade to 2.0

 

As always, please contact us if you have any questions or comments! We have an active twitter account, a discord server at chat.intermine.org, and a low traffic mailing list.

Advertisements

FlyMine 46.0 Released!

FlyMine has been updated to the latest version of FlyBase. All other data sets have also been updated to the newest versions and we have fixed a few bugs. See the data sources page for a full list of data and their versions. All data can be accessed through our comprehensive library of template searches or by building your own queries using the query builder.

 

Data model changes!

Our data model has changed slightly to make querying easier between mines.

  • Protein molecular weight is a float instead of an integer
  • We’ve added URLs for GO term evidence codes
  • Sequence Ontology (the basis for the InterMine data model) is updated, so lots of new data types added.

See our previous blog post for complete details of updates.

No more Anopheles

After listening to community feedback, we have decided to stop loading Anopheles data into FlyMine. However, as always, if there is a specific data set you are interested in, please contact us!

 

We have docs and videos, and for a full list of data sources available in FlyMine see the data sources list.

However, please do not hesitate to contact us should you require any further assistance. For all types of help and feedback email info@intermine.org.

HumanMine 5.0 released!

HumanMine has been updated to the latest version of NCBI Entrez Gene. All other data sets have also been updated to the newest versions and we have fixed a few bugs. See the data sources page for a full list of data and their versions. All data can be accessed through our comprehensive library of template searches or by building your own queries using the query builder.

 

New Data Source: GTEx

We added a new expression data set, GTEx. Here’s an example search:

Gene –> Tissue Expression

Tissue –> Gene expression

Data model changes!

Our data model has changed slightly to make querying easier between mines.

  • Protein molecular weight is a float instead of an integer
  • We’ve added URLs for GO term evidence codes
  • Sequence Ontology (the basis for the InterMine data model) is updated, so lots of new data types added.

See our previous blog post for complete details of updates.

 

We have docs and videos, and for a full list of data sources available in HumanMine see the data sources list.

However, please do not hesitate to contact us should you require any further assistance. For all types of help and feedback email info@intermine.org.

Coming up soon: InterMine 2.0 release webinar, community calls, and GSoC presentations

What’s coming up soon in InterMineLand? Here are a few of the highlights:

Upgrading to 2.0 – Thursday 2nd August

With the release of InterMine 2.0 RC1, we’ll be dedicating the InterMine Developer call to an InterMine 2.0 Upgrade Webinar, spending around 20 minutes discussing how one upgrades an InterMine 1.x installation to use the newer (and much more easygoing) Gradle dependency management system. Q&A afterwards so you can learn everything you’ve been burning to know. [Call in information]

This call will be recorded so anyone who couldn’t make it can catch up.

GSoC Student project presentations – Thursday 16th August

Six students, six awesome projects. Our students have been blogging prolifically while working over the last three months, and they’ll be presenting their work on the developer call, with five minutes slots per student + time for Q&A afterwards. [Agenda here]

This call will be recorded so anyone who couldn’t make it can catch up.

Community Outreach Call – 6th September 

Once a quarter we host non-techie calls where we focus on interesting things the community has been doing as well as community engagement in general. This time we’ll be featuring Kevin Macpherson, who runs some fantastic community outreach at SGD, including amazing webinar video use-cases.  [Agenda, still work in progress]

Previous featured speakers include Jacqueline Campbell talk about her approach to community engagement, Wayne Decateur demonstrating InterMine code in Jupyter notebooks, and Abby Cabunoc Mayes, Mozilla’s Working Open Practice Lead.

We’re still looking for speakers for this call and the next one, in December – If you have a topic you’d like to share about InterMine, open science/source, or bioinformatics in general, ping yo@intermine.org to pitch the idea.

GSoC’18- Cross InterMine Search Tool (Progress so far)


It has been three weeks since the commencement of the GSoC’18 Coding Phase and a lot of progress has been made till now. The project has been deployed here.

This week has been really very productive. I have worked upon several parts of the application this week. My main focus for this week was to add filters into the application. But I ended up with some more cool stuff here 😉
Let’s have a look at all of them one by one.

  1. Search Rating/Relevance Score
The application knows what you are searching for 😉

Relevance score is a value which is calculated by the InterMine QuickSearch API endpoint for every keyword which is searched. This rating determines how much relevant is the result item with the search keyword. The QuickSearch API response provides a floating value of ‘relevance’ parameter. InterMine uses a formula for determining the Search score in the specific InterMine Search Portals. I have used the same formula to convert the floating relevance score into a score out of 5. The formula is here:
=> Math.round(Math.max(0.1, Math.min(1, relevance)) * 5)

2. Different colors for different Categories in Results

The world is colorful and hence our app too 😉

This was a feedback received from the InterMine community. Having different category results shaded in different colors helps in exploring the results easily. At times, there may be a long list of result items returned by the application. Then having separate colors for separate categories helps us to explore faster. Our eyes are meant to perceive colors more quickly than text.

3. External links to reports

Every result item has a link which opens the result report on its particular InterMine portal. This report page contains more detailed information about the result item. On clicking the icon, a new tab with the given result report opens in the browser. The result link is generated dynamically:

4. Metadata about search results

Metadata for BMAP mine (search: ‘brca1′)

Having metadata about the results returned is always handy at work. Every tab in the application loads a set of metadata as attached above. This can certainly help in understanding the presence of a search term in the mine in a unified way.

5. Search/Relevance Score Filter

Score filter on sidebar of the application

With addition of score/relevance in the application, it was very much necessary to add an option to filter the results based on that. This section provides radio buttons to filter out the results based on the relevance score of the result item. I hope this feature will be extremely beneficial for the community members out there. 🙂

6. Category Filter

Every mine contains data from various diverse categories. So, this feature too was a necessary requirement of a full fledged searching tool. This section is loaded dynamically based on the types of categories returned by the API. The application uses the search result metadata received from the API to generate these category checkboxes dynamically. So, we need not worry about hard-coding any of these.


So, this was all about progress so far. Almost everything in the application is mobile optimized and ready to use. Most probably, I will also be extending the scope of this project and add a REST API service for searching multiple mines. So, the project will be a full fledged Cross InterMine Search Tool in future, i.e. a package of, a client driven search interface & a back-end REST API service. You can find the project repository here. Please have a look at the application here. I would love to have more feedback from the community. (For providing your feedback/suggestions, kindly email me at dwivedi.aman96@gmail.com) Thanks for reading. Happy Coding! 🙂

 

GSoC 2018 Students Announced! 🌞☀️

After last year’s great success, we’re really excited to welcome six Google Summer of Code students to work with us again this year:

Aman Dwivedi will be working on a Cross-InterMine search tool. This will use the registry to allow users to search multiple InterMines at once, and should be a good way to figure out which mine has the data you’re looking for. Aman will be mentored by Nadia Yudina, herself a graduate of one of last year’s InterMine+GSoC program.

Adrián Rodríguez Bazaga will be working on something we’ve always wanted: an InterMine data browser – hopefully a tool that will allow users to learn a bit more about data inside an InterMine without having to know the data model. Yay for easier learning curves! Adrian’s mentor will be Yo Yehudi.

Arunan Sugunakumar is going to explore hooking InterMine up to a more modern search package, probably Solr or ElasticSearch. Our current version of Lucene is very old, and we know there are better options out there!  Daniela Butano will mentor this project.

Jake Macneal is going to work on a prototype to convert natural language questions into InterMine PathQuery – it would be exciting to have a user type “Show me all the genes associated with diabetes” into an InterMine, and get a sensible set of results back! Aaron Golden will mentor Jake.

Nupur Gunwant will be adding additional features to our python client, such as registry communication, a query manager, and visualisations. Julie Sullivan will be Nupur’s mentor for this project.

Ankit Kumar Lohani will be working on Buzzbang – a search engine to crawl multiple biological sources including, but not exclusively, InterMine instances. Justin Clark-Casey will be Ankit’s mentor.

We’re also planning to post a short interview series highlighting each student and their plans for the summer. We can’t wait to get started!!

 

 

 

 

 

 

 

How did InterMine determine its FAIR milestones?

life-sciences-fair

(Cross-posted from my blog here)

At InterMine, a life sciences data integration platform, we’re working on a BBSRC grant to make data available through InterMine ‘FAIR’. What does this mean? Well, firstly  FAIR is an initiative to make dataFindable, Accessible, Interoperable and Reusable (I’ve written a lot more about this here).

Taken on its face this is a bit woolly – isn’t InterMine data already FAIR? You can find data (type some text in its general search box or perform a structured query), access it (click the web link), interoperate with it (run a live query on its API) and reuse it (hey the data’s there, download it). Well, one of the great things about FAIR is that it has specific principles and recommendations on how to make data findable, accessible, interoperable and reusable. These place a heavy emphasis on uniformity so that software can much more easily use and combine data across the countless distinct data sources hosted by different organizations across the planet.

So in applying for the grant, how did we propose to apply these recommendations to InterMine? Essentially, we performed a gap analysis between the 15 guiding principles documented in the original FAIR paper and InterMine’s current capabilities, coming up with a plan for how we would bridge this gap.

Let’s take the first findability and accessibility FAIR guiding principles as an example

F1. (meta)data are assigned a globally unique and 
persistent identifier

A1. (meta)data are retrievable by their identifier 
using a standardized communications protocol

One way to fulfil these principles, and something popular in the semantic web world,
is to make identifiers be URLs. So great, InterMine already has URLs that have a 1-to-1 mapping to biological data objects! Search for the gene MYH7 in HumanMine for instance, and the report page you get back has this URL (stripping away some non-essential tracking information).

http://www.humanmine.org/humanmine/report.do?id=1157771

Look at another biological object and that ID number will change, since this is the internal ID used to track objects within an InterMine database.

But there’s a problem here. These ID numbers are not persistent, as required by principle F1. When the data in an InterMine installation like HumanMine is updated, this is not done additively, but rather than entire database is rebuilt since data sources need to be integrated anew. And on this rebuild, MYH7 is no longer guaranteed to have the
internal InterMine ID 1157771. In fact, it’s very likely to be different.

So part of our proposal was to implement a resolution to this problem. For InterMine as a data integration platform rather than a primary data provider it’s a very complex topic, particularly as we’re generic and model driven (so in principle you could host something completely different like a company database in InterMine!). I won’t delve into the possible solutions too much here, but at the moment it looks like a tradeoff between trying to make our internal ID persistent (e.g. by maintaining the mapping to biological objects between database rebuilds) and trying to incorporate external IDs such as MYH7 directly into the InterMine URL as specified by the InterMine instance operator, something like

http://www.humanmine.org/humanmine/gene/MYH7

We’ll be reporting more on this in the future.

This was a fairly straightforward example. Some of the other principles, such as

I3. (meta)data include qualified references to other 
(meta)data

required more interpretation, and in our proposal we related actions broadly to the principles (i.e. whether they addressed one or more of findability, accessibility, etc.) rather than specific FAIR clauses.

However, we wrote our proposal some time ago. Things are moving rapidly and many of the original FAIR paper authors are working on the FAIR metrics initiative, which will measure FAIRness with programattic and quantitative tests. I think this is a great step and now something for anybody looking to FAIRify their data resource to look at closely. We’ll be looking to apply these metrics to our own work as we continue development.