BlueGenes 0.10.0 release

This release was made to coincide with the InterMine 4.2.0 release, which included many updates to webservices important to BlueGenes. While BlueGenes aims to retain backward compatibility with InterMine instances all the way back to API version 27, (appropriate messages are displayed if your instance doesn’t support a feature) many new features are dependent on being up-to-date with InterMine releases.

We are still working towards the production release of BlueGenes, at which point we can recommend it for future deployments over the legacy user interface. This recent year has brought with it a plethora of necessary technical improvements and bug fixes, along with new additions to bring the user interface towards feature parity with the current webapp. The following details the most visible changes to BlueGenes in the last release, which you can explore by updating your local instance or using the public BlueGenes instance.

Visualization tools

  • New version of Tool API to allow list and query results page tools that use IDs from multiple classes
  • Tools on list and query results page should work properly for all classes now
  • Tools on list and query results page now update when editing im-table
  • Initialisation of tools has been made more performant
  • CovidMine visualization for Cases

im-tables

  • Better selection of constraint operation when creating filter
  • Filter manager for adding and modifying constraints and logic
  • Overly wide table contents are now hidden behind a scrollbar
  • Helpful messages and options when something goes wrong
  • Histogram in numeric column summary has been fixed and more features added
  • Calendar for Date type constraints
  • Searchable dropdown for single and multiple value constraints

Query builder page

  • Build queries with outer join and sorting
  • Save queries to your account
  • Load recently run queries from your current session
  • Data browser for selecting the root class
  • Import query from XML

Profile page (new)

  • Change your password or preferences
  • Delete your account
  • Register a new account for a mine

Lists

  • Folder hierarchy for your lists in My Data
  • Add and edit list descriptions

Interactive tool store

  • Currently placed in the developer page, but we intend to move it to an admin page in the future
  • Manage the installation, updating and removal of BlueGenes visualization tools using a web interface
  • Rich information on each tool, where they’ll be visible, and any compatibility issues with the currently active mine
  • All Tool API compliant npm packages with the bluegenes-intermine-tool tag are shown (only tools under the @intermine scope are installed by default)
  • Only superusers are allowed to make changes

Report page

  • Show FASTA information on report page when available (we intend to make drastic changes to the current report page in the near future)

Technical

  • Much improved handling of mines that are unresponsive or have erroneous web services
  • Java 11 support and a docker container

Previous minor releases

There have been some notable changes in prior minor releases. As they haven’t been mentioned in a blog post, we will include them here.

  • Dynamic page titles (the text displayed in the tab or window title) based on the current page and its contents
  • Improvements to the keyword search page
    • Filters should work as expected when applied
    • Multiple filter support
    • Endlessly display more results by scrolling down
    • Restoration of scroll position when returning to search page
  • Reworked routing
    • New and improved URL paths
    • Deep linking to pages of specific mines
  • Stability improvements to mine switching and initialising
  • HTTPS support

Announcing CovidMine – analyse integrated COVID genomic and geographical distribution data

We’re excited to announce that a project we’ve been working on for the last few weeks is ready for public consumption: CovidMine, an InterMine dedicated to COVID-19 / SARS-CoV-2 data. Data is updated on a daily basis Monday-Saturday at 6PM UK time. You can try CovidMine out now, or read more about it below. 

So, what’s it all about, and why another COVID resource? 

This is something we thought about a lot, initially – there have been a massive number of initiatives going into making data available and visualising it already. In the end it came down to a couple of reasonably simple facts: InterMine already has tools to draw data from a lot of sources and integrate it, but it also offers a familiar interface if you’ve used any of the other InterMines out there, and we have API language bindings for multiple programming languages, including R, Python, Perl, and Javascript

Data sources include confirmed Covid-19 cases, deaths, new confirmed cases and new deaths for countries from Our World In Data1, data separately for individual states (for the United States only) from the COVID Tracking Project2, Sars-CoV-2 reference genome3 and nucleotide sequences from isolates deposited in Genbank4.

If you’re aware of other data sets that might make this more useful please contact us to suggest them.

Jump straight in

We’ve prepared a few template queries to help you get started with your analysis –

What’s still missing and how can I help? 

We’re officially focusing our efforts on developing tools for CovidMine in our new user interface, BlueGenes, rather than the legacy JSP interface. 

A few things we’d like to add to the UI:

  • A data visualisation showing all results on a map.
  • A visualisation that shows change over time in countries or regions, for known cases, recovered, and deaths. 
  • A genome browser (JBrowse 1)

These visualisations would update based on the filters in the table showing in your data

Data updates: 

  • Find and integrate a data source which provides China data separately for individual states

Bioschemas Markup

We have applied structured data in JSON-LD format, using the Bioschemas.org profiles DataSet, Gene and Protein. It’s available in the legacy JSP interface only, but it will be integrated in the new interface soon.

If you’re aware of other data sets that might make this more useful, or other visualisations that might be exciting, please contact us to suggest them! 

References:

  1. https://covidtracking.com
  2. https://covid.ourworldindata.org
  3. https://www.ncbi.nlm.nih.gov/genbank/sars-cov-2-seqs/#reference-genome
  4. https://www.ncbi.nlm.nih.gov/labs/virus/vssi/#/virus?SeqType_s=Nucleotide&VirusLineage_ss=SARS-CoV-2,%20taxid:2697049

Roshni Prajapati on BlueGenes UX, user research, and saving people from bad design

This is our blog series interviewing our 2020 Outreachy interns, who are working remotely for InterMine for 3 months on a variety of projects. We’ve interviewed Roshni Prajapati, who will be working on UX research and recommendations for BlueGenes.

Hi Roshni! We’re really excited to have you on board as part of the team this summer. Can you introduce yourself? 

Hi team, can’t deny the wait for a totally new experience is driving me crazy here! I’m an IT undergraduate pursuing bachelors of Technology at IIIT Allahabad and will be onboarded to IIIrd year from August onwards. Primarily my interest lies in interaction design, user research, product thinking and a bit of graphics design still I don’t mind banging lines of code to build stuff that interest me. Few of my works could be seen here & here.

Some days I try to solve user issues by merging aesthetics, a bit mathematics & data in unequal proportions while other days I can be spotted preparing for my upcoming hackathon, lying all day watching cartoons or enjoying 70s-80s classical playlist. 

    Other than this I’m a wanderlust person, a guitarist, a painter, an intermediate football and Table Tennis player and a coffee addict 😛 

What interested you about Outreachy with InterMine?

I have this craving of improving things to redefine work for living breathing humans i.e,  to save them from bad design. Case with InterMine is that while surfing through the mine-sites I noticed it mostly comprises analytical data and their representation. The current website has several user issues & pain points, also naive look and presentation of the data is not apt and even violates some design rules. This made me dive deeper into the real world biodata and their visualization for better usability of the website added the fact that the organization itself registered a design issue (driving me more to work).

    One of the facts is that design analysis needs views from users and developers and it becomes important that the community interacts. So I needed a better understanding of the real world bio data (new to me) and mentors willingly helped, this everready response brought an optimistic vibe to work for the team and organization.

Tell us about the project you’re planning to do for InterMine this summer.

The content layout in the current website design needs to be strategically placed in order to make it easier for users to go through. Since the site contains heavy analytical and a variety of biological data, my task will be to organize the website content such that users can find the things at ease, improving overall user experience. So basically I would try to carry out my process in following phases-

Discover & Define: Carry out questionnaire sessions and meetings for collecting user experience observations then interpret the observations and define insights. I will try to convey my ideas through user personas & stories and finally set my design challenges.

Develop & Deliver: and further will discuss ideas and through sketching and experimenting and prototyping by working on feedback iteratively.

Are there any challenges you anticipate for your project? How do you plan to overcome them?

Previously I have worked on several projects of my own and this would be the first time I would be working with a community. So collecting user experience observations remotely through unit testing and other methods is gonna be quite a challenge for me. One of the major tasks also includes my contribution in implementation of design of which I’m concerned. Since this is gonna take some time, it could be counted as another challenge still I’m pretty much sure that work would get done under the time duration provided 🙂 

Share a meme or gif that represents your project

Status update for BlueGenes

It’s been a while since we posted our last (rather optimistic) update around BlueGenes, so we thought we’d share a quick update, starting with the basics.

As a reminder, the long-term goal of BlueGenes is to replace the existing JSP-based UI with a more modern interface – one that works well with mobiles, one that hopefully responds more quickly and is easier to use, and perhaps most importantly, is easy to update and customise.

Some of the questions we’ve had in the last few months:

Q: Will BlueGenes replace the current JSP UI?

A: Yes, eventually. Once we reach official beta/prod release (we’re currently in alpha), we anticipate running them concurrently for a couple of years, but we probably will only provide small fixes for the JSP UI during this period, focusing most of our development effort on BlueGenes.

Q: Do I have to run my own BlueGenes, or can I use the central one at apps.intermine.org?

A: Since BlueGenes is powered purely by web services, it will probably be possible to run your InterMine as a server/api-only service and use BlueGenes at bluegenes.apps.intermine.org/. You can also run your own BlueGenes on your servers and domains, allowing you to customise it so it’s suitable for your data, and not having to rely on our uptime. Either (or both) should work fine. There will be some version requirements related to what version of InterMine can access all the features of BlueGenes – see the next point.

Q: What version of InterMine do I need to have to run BlueGenes?

A: BlueGenes will require a minimum version of InterMine to run. The original release of InterMine web services focused primarily on providing a way to give JSP users access to their data programmatically, but at the time there wasn’t an anticipated need for application level services such as superuser actions. There are a few web services and authentication-layer services we still need to implement, so it’s likely BlueGenes will need API version 31+ or higher in order to be fully-featured. InterMines with API version 27 or higher can run a basic version of BlueGenes. You can check out this table to see if your InterMine is configured to work with BlueGenes.

Q: Ok, so what’s left to do before BlueGenes is released as a public beta?

A: Mostly authentication, superuser and MyMine features – things  like saving and updating personal templates, sorting lists in folders, updating preferences and passwords. Some of these features require updates to InterMine itself in order to work – hence the minimum version noted in the previous question. Once these are ready we’ll move to the public beta stage.

Your input here will be incredibly welcome, too – the more feedback we get early on, the more polished we hope BlueGenes can be.

Q: Will BlueGenes work nicely with HTTPS InterMines?

A: You will be able to run BlueGenes without HTTPS, but in order to avoid inadvertently exposing user passwords, the login button will only be available over HTTPS connections. We’re also working with a student over the next few months, to implement a pilot InterMine Single Sign On service. You can read about it in our interview with Rahul Yadav.

Q: Will I be able to customise the way BlueGenes looks?

A: Totally! There are two ways you can do this. One is to make sure you have your logo and colour settings configured in your web properties. We have a nice guide for that. This’ll tell us what your preferred highlight colours are – FlyMine is purple, HumanMine green, etc. If you’re really dedicated and would like to write your own CSS, you can do that too, if you’re running your own InterMine/BlueGenes combo.

Q: I have some nice custom visualisation tools in my InterMine. I don’t want to have to re-write them!

A: We don’t want you to re-write them either! It depends how they’re implemented in your mine, but we’ve designed the BlueGenes Tool API with you in mind, and many Javascript-powered tools will require only a few lines of code to become BlueGenes ready.

As an example, the Cytoscape interaction viewer currently used in some InterMines only requires 20 lines of code to import into BlueGenes, plus a few lines of config – all the other files (and most of the config too) is boilerplate that we auto-generated.

JavaScript everywhere – the BlueGenes Tool API version 1 is released!

If you attended the 2017 InterMine developer workshop, you may recall the discussion we had about embedding tools in InterMine’s new UI, BlueGenes. One of the biggest priorities was to make sure that it was easy and fun to create visualisations for your mine.

It’s taken a lot of sweat, toil, testing, and iteration, but I’m incredibly excited to announce that Version 1.0 of the Tool API is released today. This means that you’ll be able to view all of your favourite client-side tools in BlueGenes, hopefully with just a few quick tweaks. It’ll also be relatively straightforward to install tools that other people have created.

bluegenes_protein_report_page_for_A0A0B4KEJ0_DROME
Preview of a protein report page for A0A0B4KEJ0_DROME, with the InterMine Cytoscape interaction viewer and ProtVista protein feature viewer both embedded.

 

You can try it out yourself, at bluegenes.apps.intermine.org – just search for a gene or protein report page.

How does it work?

We didn’t want to re-invent the wheel, and JavaScript definitely has package managers out there already. We use npm (node package manager) to package and install all BlueGenes tools. You can find all BlueGenes compatible tools by browsing the tag bluegenes-intermine-tool, and once you’ve configured your InterMine to work with the tool API, installing a new tool is often as simple as typing npm install @intermine/my-new-tool-name --save into a terminal. Equally, updating tools is as simple as running npm update from your tools folder. BlueGenes then looks inside a file in your tool folder called package.json, and outputs all installed tools listed there. package.json is an npm configuration file which contains a manifest of all the installed packages.

Getting started

Running tools in BlueGenes

If you have an InterMine that’s at least at version 2.1, you can start bluegenes with the ./gradlew blueGenesStart task. See our docs for details.

It’s also possible to run tools on standalone BlueGenes.

Once tools are installed, you can see the entire list of them under the developer menu in BlueGenes (top right > click the cog > developer > tool “app store”).

Developing tools and converting your existing tools

So, you’ve gotten some tools installed and you’d like to add some more of your own? We have the full tool API specs, a tutorial to walk you through creating your first tool, and a nice tool-scaffolder yeoman generator that will create most of the boilerplate files you need automatically, so you can spend time on more important things like eating cake 🎂 and feeding the cat.

Some credits and thanks

A huge thanks to Vivek Krishnakumar for the very first draft of the Tool API specs, and to Josh Heimbach for further work on the spec. I’d also like to thank Julie, who patiently tested the Tool API installation process and helped me iron out a lot of the bugs.

Future plans

What’s next for BlueGenes and the Tool API? Well, we have some updates planned specifically for the Tool API, including: 

  • extending the tool support to list result pages working with legacy tools that aren’t packaged as Javascript modulesBetter integration with BioJS.

You can see a roadmap here for the Tool API. First, though, we’ll probably be thinking about some of the final bits of polish BlueGenes needs before it can be officially launched as a non-beta UI, including: 

  • Authentication. When the InterMine web services were initially implemented, it was with an eye to enable data scientists and bioinformaticians to access data from InterMine. Some authentication-related services had been implemented, such as token-based authentication, but given the fact that the web services weren’t designed with a full application layer in mind, we need to add some more, including as user registration.
  • A MyMine section. We started with some prototypes late last year / early this year, but ended up rolling them back due to unexpected complications.
  • Speedier and more configurable report pages. Is there anything you’ve always wished for in a report page that’s not a tool? Feel free to ping us with ideas.

 

Questions, concerns, confusion, ideas?

Drop by the InterMine chat, email the group at info@intermine.org, or drop me a mail directly – yo@intermine.org.