Project

Apache Tomcat. A real world example in a large international Banking Organization.

40 Comments

Introduction

I have 5 years experience with Apache Tomcat in a real world (Enterprise) environment which does make me know what I’m talking about. Yes, I know my Java, Apache Lucene, Tomcat and some others.

My adventure started at ABN AMRO in Amsterdam where I came in to troubleshoot a memory-leak which caused the Tomcat server to be unstable. The problem was solved in 2 weeks but I left only 5 ‘years later.

Apache Tomcat
Apache Tomcat
The problem

The problems where I was hired for was believed by the ABN AMRO IT Management  to be caused by a Web Application (Global Intranet), which was the homepage of all WorkStations inside the ABN AMRO World Wide. These were little more than 100 thousand workstations, spread across the globe.

The reason why I was called was because the IT staff knew the Global Intranet used a certain library called Lucene. And when you searched Google at that time for Lucene and my location (Netherlands) my name came up at the top of the list. And yes. I knew (and still know) something  about Apache Lucene.

Lucene was at that time little known. Which is what it made a suspect for the IT staff to blame. “Onbekend maakt onbemind” is a Dutch expression. Translated into English this is “unknown makes unloved”. So when I arrived there, I found a  hectic environment of “experts” from a wide range of countries. The US, GB, India, Ireland, Scotland, Norway, Canada and  more. At first sight nobody had a method or plan. They all seemed to work unstructured without any plans or goals.

I soon learned that the Irish guy was hired because of his knowledge of load testing and indeed he had brought an impressive collection of tools which he used to bring the Server under a heavy load with the only goal (as far as I could see) to bring it down, in which he didn’t succeed.

When I joined the team my first thoughts were to clearly define the goals and write down a strategy to focus on and follow until the goals were met: The list of goals was as follows:

  1. Find out if the problem was really a memory leak.
  2. If it was really a memory leak, provide proof.
  3. Track down where the problem came from.
  4. Find a procedure to repeat the problem.
  5. Find a solution to fix the problem.

Off course nobody took the effort to read my plans. Everyone was too busy with their own things, which was proving that Lucene was indeed the root- cause of the memory leak.

I call this  ‘Wishful thinking’ because by showing the  Manager  that he was  right would increase the chance of being hired again, which everyone involved would like to.

Most of these people (except for the Indians and Egyptians) were bad, mean and willing to stick their  arm up to the elbow into the manager’s ass just to get an extension.  The whole culture inside the Bank was sick. The truth was less important than making the Manager happy and one thing to do this was to support his (discussable) ideas.

I wasn’t like them and was the only one focused on finding the real cause. The Indians and Egyptians included, since they were assigned to me.

Logically I started with point 1. I wanted to see the actual free memory. Every JDK comes with a set of tools to measure the Java Memory as well as many other properties. This is called JMX.

A program of the JMX suite to measure (and graph) Java Memory was the jConsole program. Using this in the life system showed me that the memory was gradually increasing but it didn’t provide any proof yet.

I needed a way to repeat the problem (without disturbing the life environment) so I asked IT for a test environment. Which they gave me the next day.

I skipped to point 4. Repeating the problem in my test environment now had the highest priority.

I knew about an Open Source tool for load testing called Apache jMeter. I read the documentation and decided that this was an excellent tool to use so I downloaded it.

In the jMeter documentation I saw it had a nice feature: speeded up Simulation of the Access-Logs. This was exactly what I needed. I asked the IT for the log files in which the problem had occurred and 2 hours later I hit the jackpot! I could repeat the problem within roughly one hour. Just by playing the logfiles in a faster tempo.

After I repeated the problem 3 times it became obvious to me what was going on. There was no Lucene Memory leak. It was the Global Intranet which caused the problem.

OOME in Java

Java itself has a great Memory Management build in. This is the Garbage Collector (GC). Normally this works fine and programmers don’t need to manage the memory since this is automatically done by the GC. In seldom cases Java Programs can give an Out-Of-Memory Error (OOME) . When such an error occurs it means the end of the Java Processes running and the entire process stops. It’s the death of the Server and the only way to get it running again is to restart the Server, including the Web Application.

What happened?

In fact, everyone had overseen the issues which can arise when an Apache Tomcat Server is working 24/7 and also is in use 24/7. In case of a global Intranet we can expect the following to happen:

  1. Asia wakes up, all the employees go to their jobs and switch on their Computers, which causes the Server to instantiate Objects which cause memory consumption on the Apache Tomcat Server.
  2. In the meantime the Middle East wakes up and does the same as in 1.
  3. Asia is done with their jobs and return back home. The Objects on the server don’t suddenly disappear but wait until their Time-To-Life has expired before they destroy themselves. (Invalidation).
  4. Europe wakes up and starts with  1.
  5. The Middle East goes home.
  6. The US wakes up and starts with 1.
  7. Europe goes home.

The world is turning round and round. The process never stops. When the US is going home Asia is already at work. etc, etc, etc.

Java is Object Oriented and with each user Session, Objects are instantiated to live Objects which consume memory on our Apache Tomcat Server. You can imagine that the Global Intranet on our Apache Tomcat server should need a system in place which carefully could handle the number of live Objects.

  1. A quick look inside the Web Application told me that the original Deveopers (oyster.com) had taken this into account and had built in a Caching system for this and even made it Configurable.

After I altered the Cache configuration, the problem was solved. Just to make sure I ran the jMeter test a few times without crashing the server the changes were documented and the problem was gone forever.

So I finished my job and wanted to go home but then my Project Managers decided that I was useful in their organization and they had some requirements for some new projects so I was asked to stay for some time and work on more Apache Tomcat projects.

Lessons learned
  1. Attempts to make a career just by putting your arms in the Boss’s ass doesn’t work. Commitment to your job and trying to find the truth (even if it’s not what the Manager wants to hear) works better in the end.
  2. Apache Tomcat is a good and stable platform. Even in multinationals with 100.000 and more concurrent clients.
  3. Apache Lucene is  a stable and good IR Library which can easily handle thousands of concurrent searches per second, as long as the Index isn’t extremely large.
  4. Trust Open Source Software! It’s usually better and more stable than Microsoft or other Commercial Software suppliers. Modern Open Source software (especially Apache) is more stable than commercial variants because it has been tested thoroughly by a large community of users, developers, administrators and automated systems.
  5. If you start working with a new system, take the time to read the documentation. It’s always better to spend extra time on making sure you are working with a well configured system than finding out years later that you have invested millions in troubleshooting by expensive hired consultants who actually weren’t the “experts” they claimed to be.
  6. In case you doubt this story, contact some of my LinkedIn connections, who were there. Here are some names: Adrena Keating, Wim Fokkert, Melle de Wit, Satnam Agrawal, Ron Warlich. None of them know I’m mentioning them here so if you want to contact them, also refer to this article.

To experience the performance of my Tomcat Hosting environment yourself you can try a few of my Tomcat Applications yourself:

  1. My Carrot2 clustering search server.
  2. LingoWeb.
  3. The Tomcat examples.

1 and 2 are Lucene based. You can try to put them under heavy load. Especially 2. It’s already under a load of 5 searches per second. I tested it with 2000 queries per second which didn’t affect performance yet. Currently even complex queries take about 1 to 2 ms.

 

 

40 thoughts on “Apache Tomcat. A real world example in a large international Banking Organization.”

  1. Thanks for another great article. Where else may just anyone get that
    type of info in such a perfect manner of writing? I have
    a presentation subsequent week, and I’m at the search
    for such information.

    Reply

  3. Hello. I see that you don’t update your page too often. I know that writing posts
    is time consuming and boring. But did you know that there is a tool that allows you to create new articles using existing content (from article directories or
    other pages from your niche)? And it does it very well.
    The new posts are unique and pass the copyscape test. Search in google and try:
    miftolo’s tools

    Reply

  5. What i don’t understood is in truth how you’re no longer really much more
    neatly-preferred than you may be right now.
    You’re so intelligent. You already know therefore significantly when it comes to this topic, produced me personally consider
    it from numerous various angles. Its like men and women don’t seem to be fascinated unless it is something to do with Girl gaga!

    Your individual stuffs great. At all times deal with it up!

    Reply

  6. Awesome site you have here but I was curious if
    you knew of any discussion boards that cover the same topics talked about
    in this article? I’d really like to be a part of online community where
    I can get responses from other knowledgeable people that share the same interest.
    If you have any suggestions, please let me know. Many thanks!

    Reply

  13. May I just say what a relief to uncover someone that truly
    knows what they are talking about on the internet.
    You certainly understand how to bring an issue to light and
    make it important. More people should read this and understand this side of your story.
    I can’t believe you aren’t more popular given that you surely have
    the gift. http://nafalaw.com/blog/?p=624

    Reply

  14. May I just say what a relief to uncover someone that truly knows
    what they are talking about on the internet. You certainly understand how
    to bring an issue to light and make it important. More people should read this and understand
    this side of your story. I can’t believe you aren’t more popular given that you surely
    have the gift. http://nafalaw.com/blog/?p=624

    Reply

  20. Hmm it looks like your blog ate my first comment (it was extremely long) so I
    guess I’ll just sum it up what I had written and say, I’m thoroughly enjoying your blog.
    I as well am an aspiring blog writer but I’m still new to the whole
    thing. Do you have any points for newbie blog writers?
    I’d really appreciate it.

    Reply

  22. The moment making a new kitchen or renovating an existing kitchen, it is highly recommended and highly recommended to keep an open mind and find out
    everything possible – at the conclusion of the process, it will definitely pay off.
    Here are some common myths about the design process and its particular
    results:

    “You may easily add later” -the best planning is to take into account and add all the characteristics of the
    kitchen at the planning and as well as or renewal stages, and not at a later stage.
    For example, although you may are not thinking
    about setting up a flat screen TV SET in the kitchen space or if you do not currently have a water dispenser, the best thing is to put together for future
    installation of these and plan electric powered, communication and water details and keep the choices open up.
    Also, since the kitchen is supposed to provide us for
    quite some time, it is highly recommended to measure all existing
    ergonomic solutions such as drawer drawers, cabinet style dishwashers, and so forth Even if you are young and fit at the moment,
    these additions will prove themselves Mobility later on.
    Either way, if you prefer to deal with kitchen planning
    in stages – one aspect or specific area at a time –
    be certain to plan carefully so that everything will fit in stylistic
    and functional harmony when the kitchen will finally be complete.

    ” I do not have so many things” – whether we like it or not, your kitchen has tremendous potential for ruining due
    to multitude of items kept in it, which is why space is sufficient to store, This means
    that there is a relative absence of overhead storage cabinetry, so it is important to balance function and style.

    “Bigger Better” – Your kitchen space is large and you need a huge kitchen, but then you will find yourself
    going back and out from one machine to another, and this situation,
    preferably in the look stages, consider adding a job sink beside
    the refrigerator or Reduce the need to cross these distances, and there is without doubt that these enhancements mean additional costs, and if your budget
    is limited, it is unquestionably worthwhile considering a compact kitchen.

    “I do not desire a designer ” -not every kitchen design project requires
    a designer, but many individuals need someone to help us
    see the overall picture and administer the work.
    Renting consult with an architect or designer does involve quite
    a lot of cost, but in the long run it is around keeping money
    and time. Even though you have a very clear picture of what you want, do not save the expense of a designer as it can best fulfill ideal.
    Proper and efficient planning of your kitchen and coordination between all the many professionals included in building your kitchen by a designer can keep your project within it is as well as budget, save you
    unnecessary money and unneeded problems and particularly keep your
    sanity.

    Reply

  24. I do accept as true with all the concepts you’ve introduced to your post.

    They’re really convincing and can certainly work. Still, the posts are too quick for
    beginners. May you please extend them a bit from subsequent
    time? Thank you for the post.

    Reply

  30. hello!,I like your writing so a lot! share we keep in touch extra approximately your post
    on AOL? I need a specialist in this space to resolve
    my problem. Maybe that’s you! Having a look forward to look you.

    Reply

  31. Hey I know this is off topic but I was wondering if
    you knew of any widgets I could add to my blog that automatically tweet my newest twitter updates.
    I’ve been looking for a plug-in like this
    for quite some time and was hoping maybe you would have some experience with
    something like this. Please let me know if you run into
    anything. I truly enjoy reading your blog and I look forward to your new updates.

    Reply

  38. Excellent weblog right here! Additionally your web site lots up fast!
    What web host are you the use of? Can I am getting your
    affiliate hyperlink in your host? I want my site loaded up as fast as
    yours lol

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *