InterMine 2.0 – Gradle

NB: To upgrade to InterMine 2.0 you must not have custom code in the core InterMine repository.

We have been planning out the tasks for future InterMine, and there is a lot of exciting projects on the horizon. We’re making InterMine more FAIR, putting InterMine in Docker and the cloud, our beautiful new user interface, Semantic Web and so on.

However a prerequisite for these exciting features is to update our build system. We are still using ant and it’s grown, let’s say, “organically” over the years — making updates and maintenance expensive and tedious.

After careful consideration and looking very seriously at other build and dependency management systems we’ve decided on Gradle. Gradle is hugely popular with a great community, and it’s used by such projects as Android, Spring and Hibernate. We were really impressed with Gradle’s power and flexibility, being able to run scripts in Gradle will give us the power we need to accomplish all our lofty goals.

Our goals for moving to Gradle

Managed dependencies

Our dependencies currently are manually managed — meaning if we need a third party library, we copy the JAR manually into our /lib directory. This is unsupportable for modern software and has resulted in lots of duplication and general heartache. With Gradle we can instead fetch dependencies automatically from online repositories.

A smaller repository

Implementing Gradle will allow us to replace many of our custom Ant-based facilities with Gradle infrastructure and widely-supported plugins. Our codebase will become smaller and more maintainable as a result.

A faster build

Currently, due to the way that InterMine implemented a custom project dependency system in Ant,  every InterMine JAR is compiled on every build and every time a webapp is deployed. This is unnecessary and wastes developer time. We will use Gradle’s sophisticated dependency management system to make the InterMine build more robust and efficient.

Maintainable, extensible, documented

The current Ant-based InterMine build system has been extended over the years as needed in an ad-hoc manner, and unfortunately no documentation exists. Adding a new ant task is a challenge, and debugging the current build process is time consuming and difficult. Moving to Gradle will base InterMine on a well maintained, extensible, documented and widely-used build system.

Simpler to run test suite

Currently, developers have to create property files and databases to run the full system tests, steps that are not straightforward to perform or execute. With Gradle’s help we hope to make this much easier, so that the wider InterMine community can benefit from running the InterMine test suite on their installations and code patches.

Simplicity

Finally, Gradle’s tests are in the same project as the main directory, thus cutting the number of separate projects will be cut in half. In addition, when building, the tests will be run automatically.

As an example, here is a new standard Gradle directory layout:

src/main/java
src/main/resources
src/test/java
src/test/resources

Currently our main and test projects are in different packages but in InterMine 2.0 these will be unified under single projects, as per standard practice.

What does this mean for you and your InterMine?

If you are currently maintaining an InterMine, moving to InterMine 2.0 is going to require a bit of effort on your part.

Operationally, commands such as database building and web application publishing are very likely to use Gradle commands rather than Ant targets or custom scripts. Users who have scripts to manage InterMine installations will need to adjust them appropriately. This shouldn’t require too much work.

InterMine users who have custom projects in the bio/sources directory to load data sources will need to make more adjustments. Project structures in InterMine 2.0 will not be the same as in earlier versions, since they will follow Gradle conventions rather than custom InterMine ones. However, the changes will not be major and we will provide a script to do as much automatic updating of custom sources as possible.

The greatest migration work will come for the most sophisticated operators who have directly patched core InterMine code. In this case, there are two options. Firstly, they can continue to patch and build core InterMine JARs themselves, though they will need to make adjustments for the Gradle build process. Secondly, we can work with them to add new configuration parameters to core InterMine to make such patching unnecessary, wherever possible. In both cases work will be required but the effort should not be large, since it is largely the structure of code that is changing rather than core logic or functionality.

 

This is a significant transition but one that should put InterMine on a solid base that lowers long-term maintenance costs and makes lots of exciting stuff possible in the future. As ever, please contact us if you have any concerns and we look forward to discussing this and any other subjects on community calls, blog comments, in our Discord chat and on the mailing list!