Best approach for a major Spring upgrade

I've been thinking - Java's been around for a while, and there's a lot of real-world situations out there where people maintain old, barely-maintainable legacy applications filled with years of technical debt. So what's the way out of that?

At a high level, I'm thinking the best approach updating an old Spring app would be a brand new project (starting with the latest Spring Boot, Spring Security, etc.) that runs alongside the existing legacy backend, and then you re-implement/move functionality over to the new backend incrementally. (Probably in conjuction with a move to some cloudy/microservicey/containerized architecture.)

Are any other approaches feasible? I believe (and I'm asking because maybe I'm wrong) that your experience would range from nightmare to impossible if you tried to incrementally update parts of a large, old enterprise application by just modifying the existing project/code in place. Especially if we're talking about one of those "agile" teams which deploys to prod every few weeks. Thoughts?

Yes, so far I've always started with a brand new project and moved features incrementally. Start with the unit tests.

For a large project, anything else is unfeasible, unless the migration is relatively simple (such as changing package names from javax.servlet to jakarta.servlet.

If you try to modify the existing project, you'll be keeping years worth of unused code, configuration and general bloat. It'll also make it more difficult to get support from the community.

Very frequently a large old project has been implemented using technology that is itself outdated. For example, don't even think of doing a simple code upgrade to a CORBA-based system. Even something SOAP-based would be questionable. Often code used originally could be replaced with something simple/cleaner/more powerful.

The beauty of Java is that you can keep old stuff running almost forever, unlike Microsoft-based systems, of which I've known a few whose functionality was literally measured in weeks. But regardless, the longer you let it run, the more expensive it will be to sort out all of the accumulated cruft and make it clean again.

As for recent experience along those lines, I can present the CodeRanch JForum, which I undertook to upgrade several years back, as the very operating system it runs under is long past end-of-life and thus subject to potential shutdown without warning. Here are some things I encountered:

1. The build was done in Ant. No problem as such. So is Tomcat. But vanilla Ant isn't good at tracking library versions, and like most major JEE apps, JForum uses a ton of external libraries. I could have added Ivy to the Ant build to better deal with that, but we're not doing something so elaborate that Maven can't do it just as well, so I Mavenized it. So I had to track down the latest versions of all of the external libraries and plug those version numbers into my POM. A few libraries aren't in the standard Maven archives and had to be manually defined to Maven. Also, in some cases, the package ownership had changed, meaning that the library's base name often also changed.

2. I ran the build and noted the things that didn't compile clean (if at all). These were generally due to package path changes, but sometimes also due to changes in the APIs that the libraries presented. Those had to be fixed. Also, because the current source builds under Java 8 and Java 9 and up have significant differences regarding introspection, I wanted to at least minimise the whining. Note that I never even considered moving from javax EE to jakarta EE. There's a limit to how much mayhem even I am willing to introduce at one time. The project's target would be Tomcat 9.

3. Once I had something resembling a clean build, I started running the unit tests. A lot of testing was done using Mockito and that uncovered 2 problems. First, since Mockito uses a LOT of introspection the bump from Java 8 to Java 17 wrought havoc with the compile-and-test process, so I had to clean that up. Secondly, the older version of Mockito was less stern about how you coded certain cases, so that in addition to having to upgrade test code to appease Mockito, there had been tests that weren't actually doing what the original authors thought they were doing that had to be fixed.

That brings me to the point where it's ready to be beta-tested. But there's more.

4. To be fully up-to-date, the javax stuff needs to be changed to Jakarta. For a smaller project I was able to do most of that using an app someone had written to scan and change source code while being aware that not every javax actually became jarkarta. It did most of the job, but some manual tweaking was still needed.

5. There are, alas, several characteristics of the current jforum that violate what we tell others is good practice. I think I could probably repair that without a whole lot of trouble, but, again, that's too much for the first stages of upgrade.

6. And, of course, in addition to the app itself, changes have to be made regarding the automated build and test processes, deployment and OS integration.

Sounds good! And this was a nice example Tim, I didn't know anyone had attempted that for the software here.

So it sounds like you've had some luck doing an actual in-place upgrade without starting from a new foundation. But I think for an "agile" team which is routinely deploying new code to production, it probably requires a different approach because it's too much change in one shot.

The Java 17 upgrade I should have included in the opening post as an important component, since it's a requirement for this kind of thing with Spring 6. That's one of the things that worries me with old Java 8 stuff. (Maybe from seeing too much carnage in JavaFX projects getting this treatment. A number of 3rd party JFX libs were seriously impacted by the changes made in J9 to the point that they were abandoned.)

I don't think Agile really is that different. Recall that my agenda for JForum is itself built from incremental stages. The bigger issue would be in moving from a fixed project (waterfall) to Agile for future development. After all, since Agile is itself a continuous upgrade process, a fully Agile project shouldn't be stuck on old tech almost by definition.

Incidentally, while the JForum codebase is fairly large, it didn't take very long on any individual stage and I credit a lot of this to Java's robustness. Meaning that agility is fairly easy. You wouldn't get off this lightly if the code was in Visual Basic or even Python 2. The work needed for a Python 2 + infrastructure upgrade on Thomas Hinkle's open-source Gourmet Recipe Manager was so much that I gave up and re-coded it as a Spring Boot app. Numerous times, changes to things like the internal database support library and other third-party assists had broken that app and I wanted something less fragile.

What I meant is in a CI/CD setting, it doesn't sound like you'd have been able to deploy that WIP upgrade. I'm assuming it took more than 2 weeks.

Python comments check out as expected. I guess it's just not a mature ecosystem.

Maybe a lot of inexperienced people go into Python, for whatever reasons. Or those that use it are only using it as a secondary tool when their real focus is something different (scentific fields). So maybe it's fast to throw stuff together, but it's very much "thrown together" without the benefit of 20+ years of learning from mistakes, which Java has earned. I guess Py will get better with enough time...

In a CI/CD system - and no, the first phase did not take more than 2 weeks, more like 4 days - one would do parallel maintenance. Or in the case of JForum, just merge in changes from the production maintenance, since the new version is branched off the production source. At some point, you pull the switch and deploy the new version, run it until it becomes troublesome, roll back, fix the changes, and repeat until the new production is suitable for full-time use. It gets a little trickier if there are significant differences to persistent storage, but that can be planned for.

Python is actually younger than Java, I believe. The difference is that Java was planned from the beginning to support industrial needs, whereas many quick-and-dirty languages like Python, JavaScript and PHP were created to address immediate needs without much regard to the larger interests. It's almost always going to be easier to work with a good base design and enhance that than to attempt to retrofit features to a popular, but incompletely designed system.

In this case I'm thinking of a system that must be larger or more complex than JForum. Something the business owner is spending millions on each year, a "project" which consists of a dozen sub-projects. And a use case which is mission critical, very risk averse to the possibility of a "troublesome" deployment or significant downtime without some way for users to work around an issue immediately. (Ex. access a legacy version of an entire parallel environment.)

So I'm picturing enough problems doing this upgrade that it's not getting done in a 2 week sprint (or even if it is, it's not getting tested well enough that fast). Since, I believe, you'd also be doing a Java 8 to 17 (or later) upgrade at the same time due to Spring's minimum version - there's bound to be some dependencies that also need upgrades. Not to mention build tool updates or even IDE updates required (probably you'd just do those first).

Java is usually pretty great about backwards compatibility in this sense, running old bytecode in a newer JRE. But after Jigsaw/JPMS that norm has weakened. And they've been making breaking changes beyond just Java 9 too, so the larger you postpone the jump, the larger the challenge. That important dependency which does "one weird trick" with reflective access fine in Java 8? (Ack!)

This goes a little beyond "Spring upgrade" obviously, but I feel there's enough coupling between all this stuff that it's not going to be quick and easy making a large jump.


Yes, I agree about the snake language.

JForum is a lot more complex than it looks on the surface.

But complex systems are almost never so tightly interlocked that they cannot be broken into smaller work units. And there are assists. For example, a Mother of All Programs (e. g/, the Master Control Program in Tron) might be teased into more modern micro-services and in extreme cases, some micro-services might slowly replace subsystems within the MOP as the migration proceeds. Note also that MOPs aren't generally Agile-friendly anyway, so we're essentially considering cases that differ across different generations of development styles.

Another common migration tactic is to mirror data to the new database using ETL utilities. That can eliminate a lot of work, since you don't have to write custom data migration programs.

I've had a long and evil career and migrated/upgraded/optimized many systems on many platforms and frankly, the hardest part hasn't been the code, it has been getting the corporate will to get the process going in the first place. Followed by fighting to give in to the "Second System Effect" that Fred Brooks described and that I've personally had the honor of observing at a safe distance. In particular, one or two that attempted to use Flashy New Technology at the same time.

Yeah breaking down a large problem into smaller ones appears to be the go-to strategy for everything, and I mean everything including outside of dev from math/science to whatever the opposite of that would be.

Normally I would agree that organizational process and things like requirements elaboration are always the hardest. But if you're neglecting a complex system by only adding functionality and little else (as these big legacy enterprisey things tend to do), I feel like over time the technical difficulty is going to increase and flip that equation as technical debt piles up. When I encounter these things in practice, they are intimidating beasts.

So are you taking the position that even in the case presented (big legacy Spring application loaded with technical debt, consisting of a number of dependent sub-projects and 100+ dependencies which are just as outdated as the Spring lib), you should be able to do a Spring upgrade in-place, without a need to recreate the foundation of the project from scratch? (By foundation I mean you're re-writing old Spring configuration, upgrading or even switching your build tool, and you're upgrading to a later version of Java--from Java 8 since that's the hardest one. Probably starting with an empty Spring Boot application and moving parts to it in small increments.)

Because I'm skeptical that's generally always possible. In my opinion, if you're unable to do the full upgrade in a short period that fits in a sprint or two with a short-lived Git branch, it's pretty much a requirement that you're creating a parallel project and then porting functionality over to it gradually. But maybe that's not necessarily the case and I should reconsider? This is all assuming of course that you've taken steps beforehand to modularize things enough that they can be moved in parts, in the first place.

I just can't imagine an upgrade in place going well in this scenario. I feel like you'd be fighting to fix numerous problems for a while, and even if you manage to get those fixed, you'd be too nervous about it by the time it was done. (Because let's be really real here - a lot of these things simply don't have useful automated tests at all.)

There comes a point where you simply have to cut your losses and replace, rather than upgrade because the original project likely had serious flaws that are unfixable and no longer tolerable and/or you need features that the original project simply cannot support.

I should note that both the IBM DB/2 and Oracle products show what happens when the icing  becomes more important than the cake. DB/2 really irritated me because the data file formats for the eSeries, iSeries and eSeries implementations are completely different and the ability to do basic things like dump and restore between the different platforms was lacking when I needed it, if not even today. Nothing as simple as a "dump to SQL" option. I had to resort to a third-party ETL tool. That wouldn't have been necessary when working with MySQL or PostgreSQL, because even though they don't have major corporations owning them (at least before Oracle bought MySQL), they had a more solid foundation that wasn't originally designed to fit within 32KB RAM. Oracle has driven me to rage when a SELECT statement pulling close to 100 columns failed with a message saying something like "invalid column (and guess which one!)".

Sometimes these omissions can be retro-fitted, but other times you may find that, for example, the core database engine has no internal way of back-tracing for the information the user so desperately needs.

Spring projects are relatively new in my career and so the level of technical  debt that they can accrue is likely to be less severe, especially in view of Java's anti-aging features. You've also now got easy ways to break off parts of a project into micro-services to reduce exposure in the future.

As for testing, a Maven-built Spring project has no excuse for being short on testing, and if the current system doesn't test enough, adding tests is something I'd do even before modifying application code.

Ahh yes, those vague Oracle errors, my favorite! (I don't know if PostgreSQL is any better about that, but I hope it is...)