FluentD - Log

FluentD v2.0



Installation Procedure

Code

FluentD.zip

docker compose up -d

Reference

https://www.digitalocean.com/community/tutorials/how-to-set-up-an-elasticsearch-fluentd-and-kibana-efk-logging-stack-on-kubernetes


pip install fluent-logger
from fluent import sender
from fluent import event


Function to call and Insert Record in to FluentD

#Error Log for fluent
@app.route('/ApplicationErrorLog', methods=['GET', 'POST'])
def ApplicationErrorLog():

	
	try:
	
		req_data = request.get_json() 
		serrortype = req_data['serrortype']
		sprogram = req_data['sprogram']
		sfnname = req_data['sfnname']
		smsg = req_data['smsg']
		sAppicationName = req_data['sAppicationName']
		
		if(DBKEY == "PROD"):
			sender.setup(sAppicationName, host='192.168.50.36', port=24224)
			event.Event('PROD', {
			'error_Type': serrortype,
			'program': sprogram,
			'function name':  sfnname,
			'error message': smsg
			})
			return "PROD"
		else:
			sender.setup(sAppicationName, host='192.168.50.36', port=24224)
			event.Event('DEVL', {
			'error_Type': serrortype,
			'program': sprogram,
			'function name': sfnname,
			'error message': smsg
			})
			return "DEVL"
	except :
		#raise werkzeug.exceptions.HTTPException("Error: %s" % str("ERROR") ) 
		errmsg = str(sys.exc_info()[1]) + ' Line: ' + str(sys.exc_info()[2].tb_lineno)
		print(errmsg)
		raise    
		return errmsg


docker build -t fluentdapp:latest .
docker run -it -d -p5015:5015 --name name_flientd  fluentdapp –restart always

       

SNo. Fields Remarks
1 serrortype INFO , WARNNING , ERROR
2 sprogram (*) Program Name of application for example FileName of the code that we have written
3 sfnname Function Name that is causing the event  ( Optional )
4 smsg Error Message 
5 sAppicationName (*) Application name - Example LMS., JOURNEY etc


Scope of the project.
  1. Run the EFK using docker compose.
  2. Run the middle ware and make sure it connects with EFK
  3. Create Sample Index in elastic search.
  4. Integrate Copy LE and Delete LE functionality with EFK.

FluentD v1.0

Introduction

FluentD is the centralized Log collector and format that in to standard format and send it to elastic search & kibana for visualization.

Video URL for introduction

https://www.youtube.com/watch?v=5ofsNyHZwWE

Installation ( EFK stack using Docker and kubernet )

https://www.youtube.com/watch?v=MNId4HG0wV8

https://github.com/justmeandopensource/elk/tree/master/docker-efk

EFK ( Elastic Serach / Kibana / FluentD / http )

https://www.youtube.com/watch?v=Gp0-7oVOtPw