Status update for BlueGenes

It’s been a while since we posted our last (rather optimistic) update around BlueGenes, so we thought we’d share a quick update, starting with the basics.

As a reminder, the long-term goal of BlueGenes is to replace the existing JSP-based UI with a more modern interface – one that works well with mobiles, one that hopefully responds more quickly and is easier to use, and perhaps most importantly, is easy to update and customise.

Some of the questions we’ve had in the last few months:

Q: Will BlueGenes replace the current JSP UI?

A: Yes, eventually. Once we reach official beta/prod release (we’re currently in alpha), we anticipate running them concurrently for a couple of years, but we probably will only provide small fixes for the JSP UI during this period, focusing most of our development effort on BlueGenes.

Q: Do I have to run my own BlueGenes, or can I use the central one at apps.intermine.org?

A: Since BlueGenes is powered purely by web services, it will probably be possible to run your InterMine as a server/api-only service and use BlueGenes at bluegenes.apps.intermine.org/. You can also run your own BlueGenes on your servers and domains, allowing you to customise it so it’s suitable for your data, and not having to rely on our uptime. Either (or both) should work fine. There will be some version requirements related to what version of InterMine can access all the features of BlueGenes – see the next point.

Q: What version of InterMine do I need to have to run BlueGenes?

A: BlueGenes will require a minimum version of InterMine to run. The original release of InterMine web services focused primarily on providing a way to give JSP users access to their data programmatically, but at the time there wasn’t an anticipated need for application level services such as superuser actions. There are a few web services and authentication-layer services we still need to implement, so it’s likely BlueGenes will need API version 31+ or higher in order to be fully-featured. InterMines with API version 27 or higher can run a basic version of BlueGenes. You can check out this table to see if your InterMine is configured to work with BlueGenes.

Q: Ok, so what’s left to do before BlueGenes is released as a public beta?

A: Mostly authentication, superuser and MyMine features – things  like saving and updating personal templates, sorting lists in folders, updating preferences and passwords. Some of these features require updates to InterMine itself in order to work – hence the minimum version noted in the previous question. Once these are ready we’ll move to the public beta stage.

Your input here will be incredibly welcome, too – the more feedback we get early on, the more polished we hope BlueGenes can be.

Q: Will BlueGenes work nicely with HTTPS InterMines?

A: You will be able to run BlueGenes without HTTPS, but in order to avoid inadvertently exposing user passwords, the login button will only be available over HTTPS connections. We’re also working with a student over the next few months, to implement a pilot InterMine Single Sign On service. You can read about it in our interview with Rahul Yadav.

Q: Will I be able to customise the way BlueGenes looks?

A: Totally! There are two ways you can do this. One is to make sure you have your logo and colour settings configured in your web properties. We have a nice guide for that. This’ll tell us what your preferred highlight colours are – FlyMine is purple, HumanMine green, etc. If you’re really dedicated and would like to write your own CSS, you can do that too, if you’re running your own InterMine/BlueGenes combo.

Q: I have some nice custom visualisation tools in my InterMine. I don’t want to have to re-write them!

A: We don’t want you to re-write them either! It depends how they’re implemented in your mine, but we’ve designed the BlueGenes Tool API with you in mind, and many Javascript-powered tools will require only a few lines of code to become BlueGenes ready.

As an example, the Cytoscape interaction viewer currently used in some InterMines only requires 20 lines of code to import into BlueGenes, plus a few lines of config – all the other files (and most of the config too) is boilerplate that we auto-generated.

Advertisements

Community outreach call coming up – 6 June, all welcome!

Update: If you missed the call, the recording is available on YouTube

InterMine runs quarterly Community Outreach calls, targeted to interest people in the overlap between life sciences, open source, and open science, with a few InterMine specific updates sprinkled in as well. Most times we host one or two guest speakers who work in scientific outreach or have done an interesting InterMine-related project we’d like to spotlight. We aim to welcome anyone interested in community outreach and generally try to avoid overly techie themes.

The next outreach call coming up on June 6th, 5PM UK time, with two exciting guest speakers, plus we’ll briefly mention the six Google Summer of Code students working with us over the next few months. 

Please help to spread the word by sharing with colleagues and friends – the more the merrier!

Speakers:

  • Malvika Sharan is a computational biologist and a community outreach coordinator for EMBL Bio-IT, which fosters a community of bioinformaticians. She will be talking about inconclusiveness in Open Science Communities using examples from her work at EMBL and her involvement in The Carpentries, SSI, and Mozilla.
  • Emmy Tsang: I’m the new Innovation Community Manager at eLife and am now running the eLife Innovation Initiative. I’m going to talk about the #eLifeSprint– our effort to drive and support collaborations in developing open-source software for open science, and the latest developments of Reproducible Document Stack project, which include a roadmap towards sharing reproducible research.

Agenda and joining details: https://docs.google.com/document/d/1VhfFbOwx95UZP-sRDVib8HNGPCXCfnpDVGh8VAbeI0c/edit – we’ll be using Zoom for this call, rather than GotoMeeting.

When: 6 June, 2019 at 17:00 UK time / or in your timezone: http://arewemeetingyet.com/London/2019-06-06/17:00/InterMine%20Community%20Outreach%20Meeting

GSoC 2019 with InterMine is ON!

After the fabulous experience we’ve had with GSoC in 2017 and 2018, we’re delighted to announce that we’ll be mentoring again this year. It’s almost impossible to describe the breadth of experience, quality, and insight students bring us every year and we’re so excited to meet a whole new batch of students again in 2019.

Prospective student?

If you’re a student interested in working with us, your first port of call is our GSoC site. Most of our students hang out at chat.intermine.org too.

We have a Q&A webinar coming up on March 12, 2019 at 3PM UK time (when is it in your timezone?) where we’ll share tips for good applications, GSoC alumni from previous years will share their experiences, and we’ll briefly describe all of the project ideas and answer any questions. If you can’t make it, add your questions to the agenda before the call and we’ll answer them during the call anyway! Here’s the agenda and joining instructions.

Interested in mentoring?

Generally we expect mentors to come from our community – InterMine users, developers, or previous students. If you fit into one of those categories and want to help mentor, email yo@intermine.org. Not sure if you’d be a good fit? We’re still happy to discuss any ideas!

We’re adopting a code of conduct!

TL;DR:

Please read our Code of Conduct draft and comment if you need to.

Longer:

A growing and important movement in open source communities is to adopt a code of conduct, which generally governs behaviour amongst community members, and provides backing to enforce necessary actions if anyone within the community behaves in an unacceptable or unwelcoming manner. We haven’t had any problems, and we’d like things to stay that way in the future.

If the past is anything to go by, we’ll set this code of conduct up and rarely or never need to enforce anything, but it’s better to have clear guidelines in place and not need them, than vice versa. We’d also like to get this in place before anything happens, rather than as an obvious too-late response to an incident – not that we’re anticipating anything!

The draft we’ve put together is adapted quite closely from the Django code of conduct. We’re particularly grateful to them for licencing it under a creative commons attribution licence so we could re-use it.

Read the InterMine Code of Conduct draft here.

Questions?

If you’d like more info about codes of conduct – why they’re important, what topics they cover, etc., please see:

Comments, or questions that weren’t answered by the links above?

Feel free to comment on this post, tweet us, email yo@intermine.org, or info@intermine.org. Please comment by the 19th of March 2019.

 

Header image from flickr, taken by Mike McSharry and licenced under CC-BY-2.0 https://www.flickr.com/photos/mikemcsharry/5360225083/

JavaScript everywhere – the BlueGenes Tool API version 1 is released!

If you attended the 2017 InterMine developer workshop, you may recall the discussion we had about embedding tools in InterMine’s new UI, BlueGenes. One of the biggest priorities was to make sure that it was easy and fun to create visualisations for your mine.

It’s taken a lot of sweat, toil, testing, and iteration, but I’m incredibly excited to announce that Version 1.0 of the Tool API is released today. This means that you’ll be able to view all of your favourite client-side tools in BlueGenes, hopefully with just a few quick tweaks. It’ll also be relatively straightforward to install tools that other people have created.

bluegenes_protein_report_page_for_A0A0B4KEJ0_DROME
Preview of a protein report page for A0A0B4KEJ0_DROME, with the InterMine Cytoscape interaction viewer and ProtVista protein feature viewer both embedded.

 

You can try it out yourself, at bluegenes.apps.intermine.org – just search for a gene or protein report page.

How does it work?

We didn’t want to re-invent the wheel, and JavaScript definitely has package managers out there already. We use npm (node package manager) to package and install all BlueGenes tools. You can find all BlueGenes compatible tools by browsing the tag bluegenes-intermine-tool, and once you’ve configured your InterMine to work with the tool API, installing a new tool is often as simple as typing npm install @intermine/my-new-tool-name --save into a terminal. Equally, updating tools is as simple as running npm update from your tools folder. BlueGenes then looks inside a file in your tool folder called package.json, and outputs all installed tools listed there. package.json is an npm configuration file which contains a manifest of all the installed packages.

Getting started

Running tools in BlueGenes

If you have an InterMine that’s at least at version 2.1, you can start bluegenes with the ./gradlew blueGenesStart task. See our docs for details.

It’s also possible to run tools on standalone BlueGenes.

Once tools are installed, you can see the entire list of them under the developer menu in BlueGenes (top right > click the cog > developer > tool “app store”).

Developing tools and converting your existing tools

So, you’ve gotten some tools installed and you’d like to add some more of your own? We have the full tool API specs, a tutorial to walk you through creating your first tool, and a nice tool-scaffolder yeoman generator that will create most of the boilerplate files you need automatically, so you can spend time on more important things like eating cake 🎂 and feeding the cat.

Some credits and thanks

A huge thanks to Vivek Krishnakumar for the very first draft of the Tool API specs, and to Josh Heimbach for further work on the spec. I’d also like to thank Julie, who patiently tested the Tool API installation process and helped me iron out a lot of the bugs.

Future plans

What’s next for BlueGenes and the Tool API? Well, we have some updates planned specifically for the Tool API, including: 

  • extending the tool support to list result pages working with legacy tools that aren’t packaged as Javascript modulesBetter integration with BioJS.

You can see a roadmap here for the Tool API. First, though, we’ll probably be thinking about some of the final bits of polish BlueGenes needs before it can be officially launched as a non-beta UI, including: 

  • Authentication. When the InterMine web services were initially implemented, it was with an eye to enable data scientists and bioinformaticians to access data from InterMine. Some authentication-related services had been implemented, such as token-based authentication, but given the fact that the web services weren’t designed with a full application layer in mind, we need to add some more, including as user registration.
  • A MyMine section. We started with some prototypes late last year / early this year, but ended up rolling them back due to unexpected complications.
  • Speedier and more configurable report pages. Is there anything you’ve always wished for in a report page that’s not a tool? Feel free to ping us with ideas.

 

Questions, concerns, confusion, ideas?

Drop by the InterMine chat, email the group at info@intermine.org, or drop me a mail directly – yo@intermine.org.

Google Summer of Code 2019 – it’s time to get your thinking caps on!

TL;DR: Send us your awesome project ideas and/or volunteer to be a mentor!

Longer version:

We need your ideas!

GSoC 2019 has been announced, and as in 2017 and 2018, InterMine will be applying again to become a mentor organisation.  This means we’re back at the “we want your project ideas!” phase – and we do! If you work with or use an InterMine and have ideas for its improvement – might it be something big enough for a student to work on for three months? Any of these types of ideas would be great:

  • An interesting exploratory project that answers a question – “is x likely to be possible or practical with InterMine?”
  • Fixing something that’s always bothered you – in 2018, we managed this with the Solr Search update!
  • A well-scoped application like the InterMine iOS app
  • A set of Javascript Visualisations for your InterMine’s data.

Even if you don’t have time to mentor the project yourself, any ideas like this would be greatly appreciated! You can add them directly to the ideas doc, or feel free to contact us to chat about it more.

Could you mentor for an InterMine project?

In 2017 and 2018 we’ve had mentors from the community, including mentors who were previously GSoC students. Ideally, interested mentors should be known to us, perhaps because you are an InterMine user, developer/maintainer/administrator, or a previous student. If you don’t have any project ideas of your own, you may be able to pick one from the project list that suits your skills and interests.

What is mentoring like, you might ask? We have the basics set out in our mentor terms and conditions (which isn’t as dry as the title suggests). Some things to note:

  • The busiest period is the application phase, when multiple students will be interacting with you to learn about and contribute to the community.
  • After this, things calm down a lot. You’re expected to meet (virtually) with your student at least once a week for the three months of the coding phase. Proactive students often only need an hour or two here or there, but other students may need more hands-on attention.
  • Students wages are paid by Google. Mentors also get a small stipend and thank-you gift after GSoC is complete.
  • You’ll be paired with a Cambridge-based mentor for support, guidance, and cover while on vacation.

A great opportunity all around

GSoC as a program ends up being incredibly valuable two-way exchange. Students get three months of paid work experience at an open source organisation, and on the other side InterMine and InterMine mentors end up with the chance to guide projects and see some truly fantastic work implemented. Promising students might even end up applying for vacancies when they come up – it’s a great way to broaden your community!


Research software engineering as a career, #RSE2018, and programming languages as poetry…

Research software engineering as a career path

What is an RSE (Research Software Engineer), you may ask? It’s a role that has existed for decades, but has only been using this name for a few years. As RSEs, we tend to be software engineers who work in academia, or perhaps academics who write production-ready code – or maybe both.

A common theme seems to be universities establishing RSE groups who work in consultancy-style ways – academics who have code, or have a need for code, approach the groups and are helped through their tasks, whether it be refactoring some old/messy/slow code, providing suggestions, or writing code to make their research easier. The RSE group may also provide training in programming languages, version control, best practices and other relevant computational basics that ease the needs of researchers.

Whilst I think most or all of us at InterMine would consider ourselves to be RSEs, we don’t really fit this model – we all write code, we all contribute to papers, but all of our sub-projects and work focus around a single primary project – some of us are working to make InterMine more FAIR, others to make it easy to launch InterMine on the cloud, but it’s still all InterMine. I’m sure we’re not the only group like this, and it makes me wonder if there should be names for the different flavours of RSE groups out there. Central RSE groups vs. dedicated RSE groups? Consultancy / support / advocacy RSEs vs. RSE specialist groups? I’m not sure if any of these are quite right, and I’d be curious to hear what others think.

RSE 2018: a grassroots conference for research software engineers

Moving on from musing about job titles, though – a bit about the recent conference. RSE2018 is the third annual UK conference for Research Software Engineers, but it’s the first time I’ve attended, personally. It made a change to have a conference where everyone around was working in research and software development, but not all of it was open source or bioinformatics related. I relished the chance to meet and discuss career paths with others, and enjoyed perhaps too much when the late-night conference dinner descended into attempts to assign poetry genres to different programming languages. Java is obviously epic poetry, but others get trickier. Terse Clojure might be a haiku, and perhaps Python, with its structured whitespace, is a form of concrete poetry?

The conference keynotes varied – there was an introduction to a digital humanities project, Oracc, which hosts annotated and transcribed cuneiform, we were introduced to the Microsoft Hololens and some of the challenges and history of its creation, a talk about Google Deepmind, and I particularly enjoyed the keynote talking about the sustainability of research software. Given how chaotic dependencies make everything, it’s no wonder that maintaining software takes a significant amount of time and money!

There were some hands-on tutorials and workshops, but I mostly attended RSE-community related sessions. A couple that stood out to me, in no particular order:

  • Diversity in recruiting RSEs. We had speakers from Microsoft talking about their efforts to make their research staffing pool more diverse, which included gruelling-sounding half-days sessions where candidates were interviewed by four different interviewers in an attempt to remove bias. Somewhat entertainingly, the room this was conducted in – the senate chamber – had red throne-like seats and eight large portraits on the walls, every single one depicting an older white male. The irony was not lost upon the session attendees!
  • The RSE community AGM. Rather than being an informal gathering of individuals, the UK RSE group will soon be re-launching as an official society that members can join for a nominal fee. The AGM gave us a chance to hear about some of their plans (you can sign up to hear about the launch date), as well as the opportunity to share your wish list of likes, dislikes, and comments on the activities the group performs. I’m looking forward to interacting with the society and seeing where they head!

It’s a conference I’d definitely like to attend again. If you missed out, you can catch up with many of the relevant points on twitter, under the hashtag #RSE18.