Blog: InterMine Cloud + ISATools: coming to a cloud near you

As many InterMiners may remember, due to some unlucky timing, we had a grant deadline that occurred during the InterMine Developer Workshop earlier this year, with seemingly countless group-work sessions like the one pictured here:

Thankfully, it looks like all our caffeine-fuelled hard work paid off the way we hoped: we are extremely excited to announce that the Wellcome Trust awarded us the grant! Here are a few highlights of what we plan to work on when the project starts in April next year:

Collaboration and metadata

This grant was written with Susanna-Assunta Sansone (Oxford e-Research Centre, University Of Oxford) to support a collaboration with the ISA-Tools group. ISA provides tools to structure metadata, covering the Investigation, Study and Assay of experiments; we will integrate the ISA format into InterMine and the ISA team will develop web-based tools to make metadata creation easier than ever.

Make your own InterMine with less sweat and toil

InterMines can be a challenge to set up right now unless you’re a developer. We’d like to do better. For simpler data formats, we’re hoping to create a UI-based wizard that allows you to drag and drop your files, select a few settings, and start a build – no need to touch a text editor. The more advanced / custom data formats will probably still require hands-on developer time.

InterMine in the cloud

If setting up your own InterMine becomes pleasantly easy, why not do it online? It’d be awesome if you could go to a website, click the “New InterMine” button, upload a few files (or paste their urls!), and end up with my-new-experiment.some-cloud.org for all your lab’s InterMine needs. Data will be merged with external supplementary data sources, and you’ll be able to analyse your data using visualisations (we’ll dedicate a chunk of time specifically to adding more datavis), our famous results tables, and familiar tools like gene set enrichment.

Enhanced import and export

We’d like to smooth out your way to sharing data with the community. Maybe you want to import data from Galaxy, or perhaps you want to export your InterMine as a virtual machine to be powered up and re-examined at a later date. Maybe you’d prefer to export data for publication as an ISA archive, bundled neatly with metadata, or maybe even generate the scaffold of a data paper automatically from your datasets. These are all things we’ll be working on with Susanna’s group.

Tell us what you think

As ever we are keen for input from the community and as we gear up for the next phase of development, now would be a particularly good time to hear from you! Tweet us, leave a comment, email the developer list, or pop by for a chat.

Advertisements

Google Analytics in BlueGenes: what should we track?

TL;DR: We’re implementing analytics tracking in BlueGenes. We can probably track anything you like, within reason. Leave a comment [comments now closed] or email us if you have anything you’d like to see! Must adhere our privacy policy.

Longer version:

InterMine’s JSP pages (the current, older UI) are set up with a couple of different types of tracking:

  1. Google Analytics, which currently anonymously records things like:
    1. Number of users and their locations
    2. Pages viewed
    3. With a bit of effort you can figure out what items were searched for by analysing query strings.
  2. InterMine home-brew internal analytics (to view in your own mine, log in as the super user and select the “usage” tab.) It tracks:
    1. Logins (anonymously)
    2. Keyword search terms
    3. Popular templates
    4. Count of custom queries executed
    5. List views by InterMine object type (but not list contents)
    6. Count of lists created, by type

So we have a couple of questions we’d love some feedback on, as we implement Google Analytics in BlueGenes:

  1. Do you use the current analytics? Which, or both?
  2. What would you *like* to record? Here’s a list of ideas

Things that are probably okay to track

  • Pageviews including counts and times – e.g. “17 views for /region-search on Monday the 13th at 10:pm”
  • Logins (anonymously)
  • Visitor location
  • Tools used (e.g. report page tools interacted with)
  • Popular templates
  • Mine used / switched to a different mine

Things we’re not sure about – what do you think?

  • Keyword search contents (anonymously). Pros: interesting analyses like this one. Cons: Could someone avoid InterMine out of fear someone would notice their gene is getting too much attention?
  • List contents (anon, as above).
  • What about mistyped identifier names in list upload?
  • Region search
  • Queries built in the query builder

I’m sure I’ve missed off quite a few things from both lists. We’d love to hear your input and feelings, both with regards to privacy and with ideas about useful trackable events and pages. Tweet us, comment on the web services tracking  github issue, email the dev group, or contact us some other way: http://intermine.readthedocs.io/en/latest/about/contact-us/

 

 

 

 

GSoC final month: testing, wrapping up, and live demos

We’re in to the final stretch of the three month Google Summer of Code period, and results are coming through thick and fast.

On August the 17th at 5PM UK time (you can check when it is in your local timezone) we’ll be doing short presentations for each of the projects as part of our community call – around 5 minutes per project. Come join in and see the great work our students have been doing!

Here’s a quick summary of projects to date:

InterMine Registry: The registry is up and running! You can view all known instances of InterMine in the registry front end, or browse the API docs to learn more about programmatic access. Tip: like the logos you see? Add yours with these handy tips from Julie in an earlier post.

intermine-registry
Snapshot of the registry front-end UI.

Leonardo also wrote a great blog post about his work on the registry.

InterMine iOS app: Several members of the InterMine community signed up to provide beta testing while the app was under development. Nadia’s been doing some great work on this – users can now use keyword search across multiple InterMines, browse templates, lists, and create sets of “favourite” InterMine objects – perhaps building up a literature search for future use. It also loads its mine list straight from the registry! Expect it in the app store soon.

Similarity Project: Samyadeep wrote up an in-depth technical project on the InterMine object similarity engine he’s been working on, using FlyMine sample data in Neo4J.

Neo4j: Yash will be demoing his InterMine Query <—>Cypher work on the call, or in the meantime, you can check out his blog posts on the subject.

R: Konstantinos updated our InterMine R client library to include new features such as enrichment visualisation – expect a blog post about it soon! It’s under review in Bioconductor but you can use the library now directly from GitHub.

 

InterMine Registry

At the beginning of the development of this project, there was no place from where all the up-to-date InterMine instances information like name, url, description, versions, organism, colors, logo, could be retrieved at once. This lead to hard-coded information, and inefficient processes in order to get these data. Motivated by these problems, InterMine Registry idea was conceived. InterMine Registry is a place where all the up-to-date instances information is stored and can be consumed by applications like Blue Genes, iOS, InterMine R, the friendly mine tool or available to everyone who needs it.

The core of InterMine Registry is its RESTful API (http://registry.intermine.org/api-docs/). Running over Node.js integrated with MongoDB, it contains methods (endpoints) to administer the instances on the registry (add, update & delete) and search among them. Maintaining the registry up-to-date is critical. In order to achieve this goal, the Registry provides automatic updates of all the instances every 24 hours. In addition to this, all or one instances can be manually updated by using the API  synchronization methods. It should be noted that in order to administer instances, an authentication process must be done.

To complement the API, a fully responsive front-end web application is being developed (http://registry.intermine.org/), from which everyone can see all the InterMine instances and search among them. Instances are presented in a list and grid view, both of them having the same purpose but with different aspect. Moreover, a world view is presented, from which the users can see the InterMine instances location on a world map. In addition to this, authenticated users can administer the instances (add, update & delete) with a nice user interface.

This project is part of Google Summer of Code, still under development by me, Leonardo Kuffó, undergraduate student at ESPOL university (Guayaquil, Ecuador), under the mentoring of Daniela Butano. The source code of the application can be found at https://github.com/intermine/intermine-registry

 

New Branding Parameters – Mine Update Needed

We have written several non-InterMine applications that require mine-specific displays. For example:

  • iOS app needs colour and logo to distinguish between mines
  • Blue genes app needs config from mine to brand the site
  • InterMine home page
  • Registry UI
  • InterMine R – shiny app
  • Friendly mines tool

And there may be more applications in the future!

image

To make your logo and mine colour available to these applications, please set these properties in web.properties file:

branding.images.logo This image should be 45px by 45px, defaults to InterMine logo
branding.colors.header.main Main colour for your mine, defaults to grey, #595455
branding.colors.header.text Text colour designed to be readable against your main colour, defaults to white, #fff.

You will have to restart your webapp for these to take effect. You can view these parameters at the /branding API end point, e.g. flymine.org/flymine/service/branding

Here are the docs on the web.properties file, and here is FlyMine’s web.properties file. There’s also an example on Codepen.

If you need help finding the right colour, we can help, or try a colour picker!

 

InterMine 2.0: Proposed Model Changes (II)

We have several new additions and changes to the InterMine core data model coming in InterMine 2.0 (due Fall 2017).

We had a great discussion on Thursday about the proposed changes. Below are the decisions we made.

Multiple Genome Versions

Many InterMine instances have several different genome versions.

Proposed addition to the InterMine core data model

  <class name="Organism" is-interface="true">
    <attribute name="annotationVersion" type="java.lang.String"/>
    <attribute name="assemblyVersion" type="java.lang.String"/>
  </class>

Multiple Varieties / Subspecies / Strains

We were going to add variety to the Organism data type to indicate subtypes that have the same taxon ID, however some people expressed a concern that this term wasn’t generic enough.

Proposed addition to the InterMine core data model

  <class name="Organism" is-interface="true">
    <attribute name="variety" type="java.lang.String"/>
  </class>

Other suggestions:

  1. Strain
  2. Subspecies
  3. Stock
  4. Line
  5. Accession
  6. Subtype
  7. Ecotype
  8. Isolate
  9. Others? …

It was suggested that we take a vote to choose the name. Please note that you can overwrite attribute names locally. But it would be better if we could all (mostly) agree!

User Interface

Both the above changes will require updates to the core InterMine code where it is assumed that Organism.taxonID is the unique field. This assumption will be replaced so that the new fields in Organism, where present, are used for the primary key.

For user friendliness, it will be necessary to assign unique organism names. Users will then be able to easily identify distinct versions in template queries and widgets.

Syntenic Regions

Proposed addition to the InterMine core data model

<class name="SyntenicRegion" extends="SequenceFeature" is-interface="true">
    <reference name="syntenyBlock" referenced-type="SyntenyBlock" reverse-reference="syntenicRegions"/>
  </class>
  
  <class name="SyntenyBlock" is-interface="true">    
   <collection name="syntenicRegions" referenced-type="SyntenicRegion" reverse-reference="syntenyBlock" />
   <reference name="dataSet" referenced-type="DataSet" />
   <reference name="publication" referenced-type="Publication" />
  </class>
  • We decided against making a SyntenyBlock a bio-entity, even though it would benefit from inheriting some references.
  • We also decided against the SyntenicRegion1 / SyntenicRegion1 format and instead they will be in a collection of regions.

GO Evidence Codes

Currently the GO evidence codes are only a controlled vocabulary and are limited to the code abreviation, e.g IEA. However UniProt and other data sources have started to use ECO ontology terms to represent the GO evidence codes instead.

We decided against changing the GO Evidence Code to be an ECO ontology term.

  • The ECO ontology is not comprehensive
  • Some mines have a specific data model for evidence terms

Instead we are going to add attributes to the GO Evidence Code:

  • Add full description of the GO Evidence Code
  • Add a link to more information on the GO evidence codes
  • (Optional) add a link to the ECO term, if available.
<class name="GOEvidenceCode" is-interface="true">
 <attribute name="code" type="java.lang.String" />
 <attribute name="description" type="java.lang.String" />
 <attribute name="URL" type="java.lang.String" />
</class>

IEA evidence code example

Ontology Annotations – Subject

Currently you can only reference BioEntities, e.g. Proteins and Genes, in an annotation. This is unsuitable as any object in InterMine can be annotated, e.g. Protein Domains. To solve this problem, we will add a new data type, Annotatable.

<class name="Annotatable" is-interface="true"> <collection name="ontologyAnnotations" referenced-type="OntologyAnnotation" reverse-reference="subject"/> </class> <class name="OntologyAnnotation" is-interface="true"> <reference name="subject" referenced-type="BioObject" reverse-reference="ontologyAnnotations"/> </class> <class name="BioEntity" is-interface="true" extends="Annotatable"/>

This will add complexity to the data model but this would be hidden from casual users with templates.


If you would like to be involved in these discussions, please do join our community calls or add your comments to the GitHub tickets. We want to hear from you!

Researchers connected in Berlin

researchersConnected.png

I really enjoyed attending the Neo4j Life & Health Sciences Workshop, organized in Berlin, this week, by Michael and Petra: a day rich with great presentations about the application and utility of graph technology in several research areas. Here are only few examples:

  • The Ontology Lookup Service, a repository for biomedical ontologies, implemented with the support of graph databases and Apache Solr for indexing, different technologies for different purposes.
  • In the Lamond lab (University of Dundee), they model proteomics data with graph databases in order to understand protein behaviour under different conditions and dimensions of analysis.
  • MetaProteomeAnalyzer (MPA), a tool for analyzing & visualizing metaproteomics, uses Neo4j as the backend for metaproteomics data analysis software.
  • Tabloid Proteome is a database of associated protein pairs, derived from mass-spectrometry based proteomics experiments, implemented using a graphdb, which can help also to discover proteins that are connected indirectly, or may have information that you are not looking for!
  • Reactome is a pathway database which has recently migrated from MySQL to Neo4j, with relevant performance improvement. You can access data via the GraphCore open source Java library, developed  with Spring Data Neo4j, or via Neo4j browser.

I’ve lost count of how many times I heard sentences like: “Biology systems are complex and growing and graphs are the native data model” or “Graph database technology is an effective tool for modelling highly connected data as we have in biology systems”. We already knew it, but it’s been very encouraging and promising hearing it again from so many researchers and practitioners with higher experience than us in graph technologies.

In the afternoon, I attended the workshops “Data modelling with Neo4j”; starting from the data sources we usually work with, we have tried to model the entities and the relationships in order to answer some relevant questions. Modelling can be very challenging and, in some cases, it might depend on the questions you have to answer!

Before the end, I had the chance to give a short presentation about our experience with Neo4j.

Thanks again Michael and Petra for organizing such a great event!