worker

package
v0.0.0-...-112f806 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 5, 2016 License: GPL-2.0 Imports: 17 Imported by: 0

Documentation

Index

Constants

View Source 
const DefaultParseLogPattern = `(?P<line>.*)` // `(?P<host>\S+) (?P<client>\S+) (?P<user>\S+) \[(?P<created>[^\]]+)\] "((?P<method>[A-Z]+) )?(?P<uri>\S+).*"`

DefaultParseLogPattern is the default pattern for understanding log patterns

Variables

This section is empty.

Functions

func ConfiguredElasticSearchDocumentType 

func ConfiguredElasticSearchDocumentType() string

func ConfiguredElasticSearchHosts 

func ConfiguredElasticSearchHosts() []string

func ConfiguredElasticSearchIndex 

func ConfiguredElasticSearchIndex() string

func ConfiguredElasticSearchMax 

func ConfiguredElasticSearchMax() int

func ConfiguredElasticSearchMocking 

func ConfiguredElasticSearchMocking() bool

func ConfiguredElasticSearchPort 

func ConfiguredElasticSearchPort() int

func ConfiguredElasticSearchReportEvery 

func ConfiguredElasticSearchReportEvery() int64

func ConfiguredElasticSearchScheme 

func ConfiguredElasticSearchScheme() string

func ConfiguredElasticSearchUseDateSuffix 

func ConfiguredElasticSearchUseDateSuffix() bool

func ConfiguredFileOutputName 

func ConfiguredFileOutputName() string

func EsSetDefaults 

func EsSetDefaults()

Types

type ElasticSearchWorker 

type ElasticSearchWorker struct {
	WorkChannel  chan map[string]interface{}
	QuitChannel  chan bool
	WorkerNumber int
	// contains filtered or unexported fields
}

ElasticSearchWorker bulk uploads to ElasticSearch

func (*ElasticSearchWorker) CurrentCount 

func (w *ElasticSearchWorker) CurrentCount() int

func (*ElasticSearchWorker) CurrentItems 

func (w *ElasticSearchWorker) CurrentItems() []string

func (*ElasticSearchWorker) DocumentType 

func (w *ElasticSearchWorker) DocumentType() string

func (*ElasticSearchWorker) Endpoint 

func (w *ElasticSearchWorker) Endpoint() string

func (*ElasticSearchWorker) Index 

func (w *ElasticSearchWorker) Index() string

func (*ElasticSearchWorker) Init 

func (w *ElasticSearchWorker) Init() (err error)

func (*ElasticSearchWorker) Mocking 

func (w *ElasticSearchWorker) Mocking() bool

func (*ElasticSearchWorker) NextHost 

func (w *ElasticSearchWorker) NextHost() string

func (*ElasticSearchWorker) ReportEvery 

func (w *ElasticSearchWorker) ReportEvery() int64

func (*ElasticSearchWorker) SetWorkChannel 

func (w *ElasticSearchWorker) SetWorkChannel(channel chan map[string]interface{})

func (*ElasticSearchWorker) Start 

func (w *ElasticSearchWorker) Start()

Start the work

func (*ElasticSearchWorker) Stop 

func (w *ElasticSearchWorker) Stop()

Stop stops the w by send a message on its quit channel

func (*ElasticSearchWorker) UseDateSuffix 

func (w *ElasticSearchWorker) UseDateSuffix() bool

func (*ElasticSearchWorker) Work 

func (w *ElasticSearchWorker) Work()

Work the queue

type FileWorker 

type FileWorker struct {
	WorkChannel chan map[string]interface{}
	QuitChannel chan bool
	// contains filtered or unexported fields
}

func (*FileWorker) CachedFileHandle 

func (w *FileWorker) CachedFileHandle() *os.File

func (*FileWorker) Init 

func (w *FileWorker) Init() (err error)

func (*FileWorker) SetWorkChannel 

func (w *FileWorker) SetWorkChannel(channel chan map[string]interface{})

func (*FileWorker) Start 

func (w *FileWorker) Start()

Start the work

func (*FileWorker) Stop 

func (w *FileWorker) Stop()

Stop stops the worker by send a message on its quit channel

func (*FileWorker) Work 

func (w *FileWorker) Work()

Work the queue

type LogParser 

type LogParser struct {
	Channel      chan map[string]interface{}
	TimePatterns []string

	Regex *regexp.Regexp
	// contains filtered or unexported fields
}

LogParser parses the imput and puts events on a channel

func (*LogParser) ConfigureRegex 

func (w *LogParser) ConfigureRegex()

func (*LogParser) Init 

func (w *LogParser) Init()

Init initializes worker's Regex

func (*LogParser) ParseEvents 

func (w *LogParser) ParseEvents(line string) (map[string]interface{}, error)

ParseEvents parses the line (including a call to ParseURI) to add events to the map of strings -> anything. It returns that map

func (*LogParser) ParseStringForValue 

func (w *LogParser) ParseStringForValue(ts string) interface{}

func (*LogParser) ParseTime 

func (w *LogParser) ParseTime(ts string) (t time.Time, err error)

func (*LogParser) ParseURI 

func (w *LogParser) ParseURI(uri string, v map[string]interface{})

ParseURI parses the URI string and adds the relevant query parameters into the main map. it also attempts to determine the data type of the items by parsing as date, int, bool, float, and if all of these fail, then keeping as string

func (*LogParser) SetWorkChannel 

func (w *LogParser) SetWorkChannel(channel chan map[string]interface{})

func (*LogParser) Start 

func (w *LogParser) Start()

Start starts the LogWorker. it starts tailing the log file, and parsing lines from it putting parsed lines on the shared channel.

func (*LogParser) Stop 

func (w *LogParser) Stop()

Stop stops the worker and cleans up. Does *not* stop ElasticSearchWorker

type StdOutWorker 

type StdOutWorker struct {
	WorkChannel chan map[string]interface{}
	QuitChannel chan bool
	// contains filtered or unexported fields
}

func (*StdOutWorker) Init 

func (w *StdOutWorker) Init() (err error)

func (*StdOutWorker) SetWorkChannel 

func (w *StdOutWorker) SetWorkChannel(channel chan map[string]interface{})

func (*StdOutWorker) Start 

func (w *StdOutWorker) Start()

Start the work

func (*StdOutWorker) Stop 

func (w *StdOutWorker) Stop()

Stop stops the worker by send a message on its quit channel

func (*StdOutWorker) Work 

func (w *StdOutWorker) Work()

Work the queue

type Worker 

type Worker interface {
	Init() (err error)
	Start()
	Stop()
	SetWorkChannel(chan map[string]interface{})
}

A Worker defines a standard interface for Initializing, Starting, and Stopping work

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.