I have 5 years experience with Apache Tomcat in a real world (Enterprise) environment which does make me know what I’m talking about. Yes, I know my Java, Apache Lucene, Tomcat and some others.
My adventure started at ABN AMRO in Amsterdam where I came in to troubleshoot a memory-leak which caused the Tomcat server to be unstable. The problem was solved in 2 weeks but I left only 5 ‘years later.
Apache Tomcat
The problem
The problems where I was hired for was believed by the ABN AMRO IT Management to be caused by a Web Application (Global Intranet), which was the homepage of all WorkStations inside the ABN AMRO World Wide. These were little more than 100 thousand workstations, spread across the globe.
The reason why I was called was because the IT staff knew the Global Intranet used a certain library called Lucene. And when you searched Google at that time for Lucene and my location (Netherlands) my name came up at the top of the list. And yes. I knew (and still know) something about Apache Lucene.
Lucene was at that time little known. Which is what it made a suspect for the IT staff to blame. “Onbekend maakt onbemind” is a Dutch expression. Translated into English this is “unknown makes unloved”. So when I arrived there, I found a hectic environment of “experts” from a wide range of countries. The US, GB, India, Ireland, Scotland, Norway, Canada and more. At first sight nobody had a method or plan. They all seemed to work unstructured without any plans or goals.
I soon learned that the Irish guy was hired because of his knowledge of load testing and indeed he had brought an impressive collection of tools which he used to bring the Server under a heavy load with the only goal (as far as I could see) to bring it down, in which he didn’t succeed.
When I joined the team my first thoughts were to clearly define the goals and write down a strategy to focus on and follow until the goals were met: The list of goals was as follows:
Find out if the problem was really a memory leak.
If it was really a memory leak, provide proof.
Track down where the problem came from.
Find a procedure to repeat the problem.
Find a solution to fix the problem.
Off course nobody took the effort to read my plans. Everyone was too busy with their own things, which was proving that Lucene was indeed the root- cause of the memory leak.
I call this ‘Wishful thinking’ because by showing the Manager that he was right would increase the chance of being hired again, which everyone involved would like to.
Most of these people (except for the Indians and Egyptians) were bad, mean and willing to stick their arm up to the elbow into the manager’s ass just to get an extension. The whole culture inside the Bank was sick. The truth was less important than making the Manager happy and one thing to do this was to support his (discussable) ideas.
I wasn’t like them and was the only one focused on finding the real cause. The Indians and Egyptians included, since they were assigned to me.
Logically I started with point 1. I wanted to see the actual free memory. Every JDK comes with a set of tools to measure the Java Memory as well as many other properties. This is called JMX.
A program of the JMX suite to measure (and graph) Java Memory was the jConsole program. Using this in the life system showed me that the memory was gradually increasing but it didn’t provide any proof yet.
I needed a way to repeat the problem (without disturbing the life environment) so I asked IT for a test environment. Which they gave me the next day.
I skipped to point 4. Repeating the problem in my test environment now had the highest priority.
I knew about an Open Source tool for load testing called Apache jMeter. I read the documentation and decided that this was an excellent tool to use so I downloaded it.
In the jMeter documentation I saw it had a nice feature: speeded up Simulation of the Access-Logs. This was exactly what I needed. I asked the IT for the log files in which the problem had occurred and 2 hours later I hit the jackpot! I could repeat the problem within roughly one hour. Just by playing the logfiles in a faster tempo.
After I repeated the problem 3 times it became obvious to me what was going on. There was no Lucene Memory leak. It was the Global Intranet which caused the problem.
OOME in Java
Java itself has a great Memory Management build in. This is the Garbage Collector (GC). Normally this works fine and programmers don’t need to manage the memory since this is automatically done by the GC. In seldom cases Java Programs can give an Out-Of-Memory Error (OOME) . When such an error occurs it means the end of the Java Processes running and the entire process stops. It’s the death of the Server and the only way to get it running again is to restart the Server, including the Web Application.
What happened?
In fact, everyone had overseen the issues which can arise when an Apache Tomcat Server is working 24/7 and also is in use 24/7. In case of a global Intranet we can expect the following to happen:
Asia wakes up, all the employees go to their jobs and switch on their Computers, which causes the Server to instantiate Objects which cause memory consumption on the Apache Tomcat Server.
In the meantime the Middle East wakes up and does the same as in 1.
Asia is done with their jobs and return back home. The Objects on the server don’t suddenly disappear but wait until their Time-To-Life has expired before they destroy themselves. (Invalidation).
Europe wakes up and starts with 1.
The Middle East goes home.
The US wakes up and starts with 1.
Europe goes home.
The world is turning round and round. The process never stops. When the US is going home Asia is already at work. etc, etc, etc.
Java is Object Oriented and with each user Session, Objects are instantiated to live Objects which consume memory on our Apache Tomcat Server. You can imagine that the Global Intranet on our Apache Tomcat server should need a system in place which carefully could handle the number of live Objects.
A quick look inside the Web Application told me that the original Deveopers (oyster.com) had taken this into account and had built in a Caching system for this and even made it Configurable.
After I altered the Cache configuration, the problem was solved. Just to make sure I ran the jMeter test a few times without crashing the server the changes were documented and the problem was gone forever.
So I finished my job and wanted to go home but then my Project Managers decided that I was useful in their organization and they had some requirements for some new projects so I was asked to stay for some time and work on more Apache Tomcat projects.
Lessons learned
Attempts to make a career just by putting your arms in the Boss’s ass doesn’t work. Commitment to your job and trying to find the truth (even if it’s not what the Manager wants to hear) works better in the end.
Apache Tomcat is a good and stable platform. Even in multinationals with 100.000 and more concurrent clients.
Apache Lucene is a stable and good IR Library which can easily handle thousands of concurrent searches per second, as long as the Index isn’t extremely large.
Trust Open Source Software! It’s usually better and more stable than Microsoft or other Commercial Software suppliers. Modern Open Source software (especially Apache) is more stable than commercial variants because it has been tested thoroughly by a large community of users, developers, administrators and automated systems.
If you start working with a new system, take the time to read the documentation. It’s always better to spend extra time on making sure you are working with a well configured system than finding out years later that you have invested millions in troubleshooting by expensive hired consultants who actually weren’t the “experts” they claimed to be.
In case you doubt this story, contact some of my LinkedIn connections, who were there. Here are some names: Adrena Keating, Wim Fokkert, Melle de Wit, Satnam Agrawal, Ron Warlich. None of them know I’m mentioning them here so if you want to contact them, also refer to this article.
To experience the performance of my Tomcat Hosting environment yourself you can try a few of my Tomcat Applications yourself:
1 and 2 are Lucene based. You can try to put them under heavy load. Especially 2. It’s already under a load of 5 searches per second. I tested it with 2000 queries per second which didn’t affect performance yet. Currently even complex queries take about 1 to 2 ms.
41 thoughts on “Apache Tomcat. A real world example in a large international Banking Organization.”
Thanks for another great article. Where else may just anyone get that
type of info in such a perfect manner of writing? I have
a presentation subsequent week, and I’m at the search
for such information.
Hello. I see that you don’t update your page too often. I know that writing posts
is time consuming and boring. But did you know that there is a tool that allows you to create new articles using existing content (from article directories or
other pages from your niche)? And it does it very well.
The new posts are unique and pass the copyscape test. Search in google and try:
miftolo’s tools
What i don’t understood is in truth how you’re no longer really much more
neatly-preferred than you may be right now.
You’re so intelligent. You already know therefore significantly when it comes to this topic, produced me personally consider
it from numerous various angles. Its like men and women don’t seem to be fascinated unless it is something to do with Girl gaga!
Your individual stuffs great. At all times deal with it up!
Awesome site you have here but I was curious if
you knew of any discussion boards that cover the same topics talked about
in this article? I’d really like to be a part of online community where
I can get responses from other knowledgeable people that share the same interest.
If you have any suggestions, please let me know. Many thanks!
Therefore prior to buying them a detailed scrutiny
ought to comply sufficient reason for. Even if you might have multiple children, you only will feel contains almost
once. They will tend enhance in magnitude as days pass.
Treat volunteers as important members of one’s team and thank them as these
kind of. A fat burning furnace diet should have likewise amino fatty
acids. We’re seeing more discretionary car purchases now. https://win88.today/live22/
May I just say what a relief to uncover someone that truly
knows what they are talking about on the internet.
You certainly understand how to bring an issue to light and
make it important. More people should read this and understand this side of your story.
I can’t believe you aren’t more popular given that you surely have
the gift. http://nafalaw.com/blog/?p=624
May I just say what a relief to uncover someone that truly knows
what they are talking about on the internet. You certainly understand how
to bring an issue to light and make it important. More people should read this and understand
this side of your story. I can’t believe you aren’t more popular given that you surely
have the gift. http://nafalaw.com/blog/?p=624
Hmm it looks like your blog ate my first comment (it was extremely long) so I
guess I’ll just sum it up what I had written and say, I’m thoroughly enjoying your blog.
I as well am an aspiring blog writer but I’m still new to the whole
thing. Do you have any points for newbie blog writers?
I’d really appreciate it.
I have fun with, cause I found just what I used to be having a look for.
You have ended my four day lengthy hunt! God Bless you man. Have a
nice day. Bye
The moment making a new kitchen or renovating an existing kitchen, it is highly recommended and highly recommended to keep an open mind and find out
everything possible – at the conclusion of the process, it will definitely pay off.
Here are some common myths about the design process and its particular
results:
“You may easily add later” -the best planning is to take into account and add all the characteristics of the
kitchen at the planning and as well as or renewal stages, and not at a later stage.
For example, although you may are not thinking
about setting up a flat screen TV SET in the kitchen space or if you do not currently have a water dispenser, the best thing is to put together for future
installation of these and plan electric powered, communication and water details and keep the choices open up.
Also, since the kitchen is supposed to provide us for
quite some time, it is highly recommended to measure all existing
ergonomic solutions such as drawer drawers, cabinet style dishwashers, and so forth Even if you are young and fit at the moment,
these additions will prove themselves Mobility later on.
Either way, if you prefer to deal with kitchen planning
in stages – one aspect or specific area at a time –
be certain to plan carefully so that everything will fit in stylistic
and functional harmony when the kitchen will finally be complete.
” I do not have so many things” – whether we like it or not, your kitchen has tremendous potential for ruining due
to multitude of items kept in it, which is why space is sufficient to store, This means
that there is a relative absence of overhead storage cabinetry, so it is important to balance function and style.
“Bigger Better” – Your kitchen space is large and you need a huge kitchen, but then you will find yourself
going back and out from one machine to another, and this situation,
preferably in the look stages, consider adding a job sink beside
the refrigerator or Reduce the need to cross these distances, and there is without doubt that these enhancements mean additional costs, and if your budget
is limited, it is unquestionably worthwhile considering a compact kitchen.
“I do not desire a designer ” -not every kitchen design project requires
a designer, but many individuals need someone to help us
see the overall picture and administer the work.
Renting consult with an architect or designer does involve quite
a lot of cost, but in the long run it is around keeping money
and time. Even though you have a very clear picture of what you want, do not save the expense of a designer as it can best fulfill ideal.
Proper and efficient planning of your kitchen and coordination between all the many professionals included in building your kitchen by a designer can keep your project within it is as well as budget, save you
unnecessary money and unneeded problems and particularly keep your
sanity.
I do accept as true with all the concepts you’ve introduced to your post.
They’re really convincing and can certainly work. Still, the posts are too quick for
beginners. May you please extend them a bit from subsequent
time? Thank you for the post.
Submitting articles to article databases with
strong ranking. Stay active, and YouTube will thank
you with prospects. Blogging demonstrates true commitment and passion to your
industry that you really can’t fake long-term. https://918kiss.host/76-lpe88
Submitting articles to article databases with strong ranking.
Stay active, and YouTube will thank you with prospects.
Blogging demonstrates true commitment and passion to your industry that you really can’t fake long-term. https://918kiss.host/76-lpe88
hello!,I like your writing so a lot! share we keep in touch extra approximately your post
on AOL? I need a specialist in this space to resolve
my problem. Maybe that’s you! Having a look forward to look you.
Hey I know this is off topic but I was wondering if
you knew of any widgets I could add to my blog that automatically tweet my newest twitter updates.
I’ve been looking for a plug-in like this
for quite some time and was hoping maybe you would have some experience with
something like this. Please let me know if you run into
anything. I truly enjoy reading your blog and I look forward to your new updates.
The games revolve around math problems, visual
puzzles, and thinking puzzles. Make sure you get about 10 articles to your
website and confident that they have links connected to them. https://918kiss.bid/games/rollex-11
Excellent weblog right here! Additionally your web site lots up fast!
What web host are you the use of? Can I am getting your
affiliate hyperlink in your host? I want my site loaded up as fast as
yours lol
Thanks for another great article. Where else may just anyone get that
type of info in such a perfect manner of writing? I have
a presentation subsequent week, and I’m at the search
for such information.
Very energetic article, I liked that bit. Will there be a part 2?
Hello. I see that you don’t update your page too often. I know that writing posts
is time consuming and boring. But did you know that there is a tool that allows you to create new articles using existing content (from article directories or
other pages from your niche)? And it does it very well.
The new posts are unique and pass the copyscape test. Search in google and try:
miftolo’s tools
The chestnut “you get what you spend for” doesn’t necessarily apply to regrowing flowing hair.
Ensure way that often work is exercise and diet.
Mingle2 indicates send emails to other members. http://Saudiairlines.tv/__media__/js/netsoltrademark.php?d=win88.today%2Fjoker123%2F
What i don’t understood is in truth how you’re no longer really much more
neatly-preferred than you may be right now.
You’re so intelligent. You already know therefore significantly when it comes to this topic, produced me personally consider
it from numerous various angles. Its like men and women don’t seem to be fascinated unless it is something to do with Girl gaga!
Your individual stuffs great. At all times deal with it up!
Awesome site you have here but I was curious if
you knew of any discussion boards that cover the same topics talked about
in this article? I’d really like to be a part of online community where
I can get responses from other knowledgeable people that share the same interest.
If you have any suggestions, please let me know. Many thanks!
Pretty! This was an incredibly wonderful article. Thank you for supplying these details.
Therefore prior to buying them a detailed scrutiny
ought to comply sufficient reason for. Even if you might have multiple children, you only will feel contains almost
once. They will tend enhance in magnitude as days pass.
What’s up every one, here every person is sharing such knowledge, so it’s nice
to read this weblog, and I used to pay a visit this blog
daily.
Treat volunteers as important members of one’s team and thank them as these
kind of. A fat burning furnace diet should have likewise amino fatty
acids. We’re seeing more discretionary car purchases now. https://win88.today/live22/
I pay a visit day-to-day some sites and information sites to read articles, however this webpage provides quality based writing. http://donayresphotography.com/__media__/js/netsoltrademark.php?d=918.network%2Fdownloads%2F85-download-live22
I pay a visit day-to-day some sites and information sites to read articles, however this webpage provides quality
based writing. http://donayresphotography.com/__media__/js/netsoltrademark.php?d=918.network%2Fdownloads%2F85-download-live22
May I just say what a relief to uncover someone that truly
knows what they are talking about on the internet.
You certainly understand how to bring an issue to light and
make it important. More people should read this and understand this side of your story.
I can’t believe you aren’t more popular given that you surely have
the gift. http://nafalaw.com/blog/?p=624
May I just say what a relief to uncover someone that truly knows
what they are talking about on the internet. You certainly understand how
to bring an issue to light and make it important. More people should read this and understand
this side of your story. I can’t believe you aren’t more popular given that you surely
have the gift. http://nafalaw.com/blog/?p=624
But if it’s discouraging to obtain that as the title it needs to be short, sweet and clear.
They’ll promise the world and never come by going to. But nonetheless can’t
obtain that top get. http://hurriseal.com/__media__/js/netsoltrademark.php?d=918kiss.bid%2Fgames%2Frollex-11
But if it’s discouraging to obtain that as the title it needs
to be short, sweet and clear. They’ll promise the world and never come by
going to. But nonetheless can’t obtain that top get. http://hurriseal.com/__media__/js/netsoltrademark.php?d=918kiss.bid%2Fgames%2Frollex-11
Very interesting subject, thank you for putting up.
Hello everyone, it’s my first visit at this web site, and article
is truly fruitful in support of me, keep up posting these types of articles.
Informative article, just what I was looking for.
Hmm it looks like your blog ate my first comment (it was extremely long) so I
guess I’ll just sum it up what I had written and say, I’m thoroughly enjoying your blog.
I as well am an aspiring blog writer but I’m still new to the whole
thing. Do you have any points for newbie blog writers?
I’d really appreciate it.
I have fun with, cause I found just what I used to be having a look for.
You have ended my four day lengthy hunt! God Bless you man. Have a
nice day. Bye
Thanks 4 your nice comment Minecraft!
The moment making a new kitchen or renovating an existing kitchen, it is highly recommended and highly recommended to keep an open mind and find out
everything possible – at the conclusion of the process, it will definitely pay off.
Here are some common myths about the design process and its particular
results:
“You may easily add later” -the best planning is to take into account and add all the characteristics of the
kitchen at the planning and as well as or renewal stages, and not at a later stage.
For example, although you may are not thinking
about setting up a flat screen TV SET in the kitchen space or if you do not currently have a water dispenser, the best thing is to put together for future
installation of these and plan electric powered, communication and water details and keep the choices open up.
Also, since the kitchen is supposed to provide us for
quite some time, it is highly recommended to measure all existing
ergonomic solutions such as drawer drawers, cabinet style dishwashers, and so forth Even if you are young and fit at the moment,
these additions will prove themselves Mobility later on.
Either way, if you prefer to deal with kitchen planning
in stages – one aspect or specific area at a time –
be certain to plan carefully so that everything will fit in stylistic
and functional harmony when the kitchen will finally be complete.
” I do not have so many things” – whether we like it or not, your kitchen has tremendous potential for ruining due
to multitude of items kept in it, which is why space is sufficient to store, This means
that there is a relative absence of overhead storage cabinetry, so it is important to balance function and style.
“Bigger Better” – Your kitchen space is large and you need a huge kitchen, but then you will find yourself
going back and out from one machine to another, and this situation,
preferably in the look stages, consider adding a job sink beside
the refrigerator or Reduce the need to cross these distances, and there is without doubt that these enhancements mean additional costs, and if your budget
is limited, it is unquestionably worthwhile considering a compact kitchen.
“I do not desire a designer ” -not every kitchen design project requires
a designer, but many individuals need someone to help us
see the overall picture and administer the work.
Renting consult with an architect or designer does involve quite
a lot of cost, but in the long run it is around keeping money
and time. Even though you have a very clear picture of what you want, do not save the expense of a designer as it can best fulfill ideal.
Proper and efficient planning of your kitchen and coordination between all the many professionals included in building your kitchen by a designer can keep your project within it is as well as budget, save you
unnecessary money and unneeded problems and particularly keep your
sanity.
So do your research certainly in this area, as
well as get around. These commandments my partner and i give you today
in order to be on top of your hearts. A time management skills system may
give families adaptability. http://www.deathjr.com/__media__/js/netsoltrademark.php?d=win88.today%2Fdownload-live22-android-ios%2F
I do accept as true with all the concepts you’ve introduced to your post.
They’re really convincing and can certainly work. Still, the posts are too quick for
beginners. May you please extend them a bit from subsequent
time? Thank you for the post.
Anyone have a LIST of all STATES that have free RCC now?
What domain name can reflect the website’s market?
Lots of commercial foods contain additives, chemicals and quickly that saturate need
or want. Really like individual with untidy and uncared for
hair. http://keyconstruction.net/__media__/js/netsoltrademark.php?d=win88.today%2Fdownload-lpe88-android-windows%2F
Submitting articles to article databases with
strong ranking. Stay active, and YouTube will thank
you with prospects. Blogging demonstrates true commitment and passion to your
industry that you really can’t fake long-term. https://918kiss.host/76-lpe88
Submitting articles to article databases with strong ranking.
Stay active, and YouTube will thank you with prospects.
Blogging demonstrates true commitment and passion to your industry that you really can’t fake long-term. https://918kiss.host/76-lpe88
I’m still learning from you, as I’m improving myself.
I absolutely liked reading all that is posted on your site.Keep the tips
coming. I loved it!
hello!,I like your writing so a lot! share we keep in touch extra approximately your post
on AOL? I need a specialist in this space to resolve
my problem. Maybe that’s you! Having a look forward to look you.
Hey I know this is off topic but I was wondering if
you knew of any widgets I could add to my blog that automatically tweet my newest twitter updates.
I’ve been looking for a plug-in like this
for quite some time and was hoping maybe you would have some experience with
something like this. Please let me know if you run into
anything. I truly enjoy reading your blog and I look forward to your new updates.
The games revolve around math problems, visual
puzzles, and thinking puzzles. Make sure you get about 10 articles to your
website and confident that they have links connected to them. https://918kiss.bid/games/rollex-11
Keep on working, great job!
What’s up, I log on to your blog regularly. Your story-telling style is witty, keep doing what you’re
doing!
My family every time say that I am killing my time here at web, but I
know I am getting knowledge daily by reading such good content.
Everyone loves it when individuals get together and share ideas.
Great blog, keep it up!
Pretty! This was an incredibly wonderful article. Thank you for providing this information.
Excellent weblog right here! Additionally your web site lots up fast!
What web host are you the use of? Can I am getting your
affiliate hyperlink in your host? I want my site loaded up as fast as
yours lol
Yes, go ahead please!
Like!! I blog frequently and I really thank you for your content. The article has truly peaked my interest.