Why Do I need a centralized Logging application ?
In which case it is must to have a centralized logging application?
My application is heavily IO bound how can I improve its performance ?
If answers to any of the above questions is “Yes”, then this blog will give you some ideas on to tackling the above scenarios.
Any application design is best performing if it has a optimal
utilization of all the three heads namely IO , Memory & CPU. If any of them is heavily utilized , it will have ripple effects on
the performance of your application. In current world where memory is
cheap, RAM (memory) is no longer the blocking factor for the
application.
How to Find whether application is CPU, IO or Memory bound?
Run top and look at the cpu usage line. A high user % indicates that it
is CPU bound meaning the rate at which process progresses is limited by
the speed of the CPU.
A high wait %( wa ) indicates that it is IO bound as the tasks are disk bound.
|
| Top Command |
To check if the application is memory bound check %MEM, this will give you the share of total available memory used by that
process. VIRT gives the virtual memory footprint of the process, of which
only RESamount is currently in physical memory (the rest is swapped out, and
not currently shown on the table) i.e the rate at which a process
progresses is limited by the amount of available memory & speed of
that memory access.
So what if my application is I/O bound, How can we improve performance ?
If we analyse the IO patterns of our application we find varied type of logs namely application logs, monitoring/status check logs at regular intervals, activity/request logs etc. Applications utilize below approaches for log file control :
1) External cron job like logrotate which rotates at mid night.
2) Use log4j to write logs into your application specific logs rather
than writing everything in catalina with buffer flush at regular
intervals.
Most of the dev’s start to reduce logging in application, but many times in heavy usage application, where TB’s of logs are generated this reduced logging doesn’t make any dent in reducing IO.
Solution ?
Build a centralized Logging application.
Below is the architecture diagram of a centralized Logging Application which uses minimum app memory and cpu overhead. This logging process is made as asynchronous as it can. Maximum benefit can be obtained by transferring the information in a serialized format (like protobuf /thrift objects. Read more on serialization here) over the network.
|
Networking requirements: Gigabit Ethernet or 10 Gigabit EthernetCPU requirements : 1 core at mostMemory Requirement: Very lowIO requirement on Application Instances: Zero (or high only when network is down which usually never happens)
Below wrapper class can be used for logging asynchronously to remote system, local queue keeps on transmitting to central logging application and in case of network unavailability the facade dumps locally to disk and starts transmitting again once the network gets back online.
public final class LoggingClientFacade
{
public static void log(int logLevel, String hostname, int instanceId, String className, String message);
public static void logWithContext(int logLevel, String hostname, int instanceId, String className, String message, Object[] contextInfo);
public static void logWithException(int logLevel, String hostname, int instanceId, String className, String message, int errorCode, Throwable exception);
...
}
Two important things to note is :
It would be advisable not to use message brokers or bridges for the transport or a centralized queue (No JMS or AMQP) and doing remoting calls to the same. A KAFKA based system or any JMS based queue adds extra infrastructure costs hence I would not recommend using them.
Use programs which offer features of log collector and processor but not aggregator as it reduces cost of additional infrastructure. Preferable options would be FluentBit (requires only 450 KB of memory and is ultra fast) , Facebook Scribe (fork maintained by Inmobi) or NSQ.
What am I missing here ? Let me know in comments section and I'll add in!
