Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

the ignore_older of tail config is of no effect #716

Closed
yubobo opened this issue Aug 18, 2018 · 6 comments
Closed

the ignore_older of tail config is of no effect #716

yubobo opened this issue Aug 18, 2018 · 6 comments
Assignees
@edsiper
Labels
fixed question

Comments

@yubobo
Copy link

yubobo commented Aug 18, 2018

image
but，in fact，add all files to scan the queue whenever the file is generated.
@yubobo
Copy link
Author

yubobo commented Aug 18, 2018

image

@yubobo
Copy link
Author

yubobo commented Aug 18, 2018

image

@yubobo
Copy link
Author

yubobo commented Aug 18, 2018

I change the value of ignore_older to 1m ,the problem is the same.

@edsiper
Copy link
Member

edsiper commented Aug 20, 2018 •
edited
Loading

The ignore_older option is only available when a Parser is specificied and that parser matched a line, so when the parser lookup the time of that specific line/record it checks if is older than ignore_older or not. It does not apply to the file age, it's record based.

@edsiper edsiper self-assigned this Aug 20, 2018
@edsiper edsiper closed this as completed Aug 21, 2018
kilya11 added a commit to kilya11/fluent-bit-docs that referenced this issue Dec 10, 2019
@kilya11
Ignore_Older is record based
28a8097 
Based on the comment: 
fluent/fluent-bit#716 (comment)
@kilya11 kilya11 mentioned this issue Dec 10, 2019
Merged
kilya11 added a commit to kilya11/fluent-bit-docs that referenced this issue Dec 10, 2019
@kilya11
Ignore_Older is record based
5f38808 
Based on the comment:
fluent/fluent-bit#716 (comment)

Signed-off-by: Ilya Kil <ilya.kil@mll.com>
edsiper pushed a commit to fluent/fluent-bit-docs that referenced this issue Dec 12, 2019
@kilya11 @edsiper
input: tail: Ignore_Older is record based (#244)
70339be 
Based on the comment:
fluent/fluent-bit#716 (comment)

Signed-off-by: Ilya Kil <ilya.kil@mll.com>
edsiper pushed a commit to fluent/fluent-bit-docs that referenced this issue Dec 12, 2019
@kilya11 @edsiper
input: tail: Ignore_Older is record based (#244)
19138e5 
Based on the comment:
fluent/fluent-bit#716 (comment)

Signed-off-by: Ilya Kil <ilya.kil@mll.com>
@acarsercan
Copy link

Someone be kind enough to paste an example config with ignore_older & parser

Thanks

@xPB12 xPB12 mentioned this issue Mar 9, 2020
Closed
@avdhoot
Copy link

avdhoot commented Jul 19, 2020

@acarsercan Might late. Set parser=myparser in input plug-in tail. tail plugin will use Time_Key & Time_Format find out correct time to compare.

[PARSER]                                                                                                                                                                
    Name         myparser                                                                                                                                                 
    Format       json                                                                                                                                                   
    Time_Key     time                                                                                                                                                   
    Time_Format  %Y-%m-%dT%H:%M:%S.%L                                                                                                                                   
    Time_Keep    On
[INPUT]
    Name             tail
    Path             /var/log/containers/*.log
    Tag              containers.*
    Refresh_Interval 1
    Rotate_Wait      60
    Mem_Buf_Limit    5MB
    Skip_Long_Lines  On
    DB               /tail-db/tail-containers-state-sumo.db
    DB.Sync          Normal
    Ignore_Older 1m
    Parser  myparser

@avdhoot avdhoot mentioned this issue Jul 19, 2020
Merged
3 tasks
This was referenced Aug 11, 2020
Closed
Closed
rawahars pushed a commit to rawahars/fluent-bit that referenced this issue Oct 24, 2022
@lecaros
output: docs for default workers setting (fluent#716)
a22cdf5 
* output: elasticsearch: add default for Workers

Signed-off-by: lecaros <lecaros@calyptia.com>

* output: file: add default for Workers

Signed-off-by: lecaros <lecaros@calyptia.com>

* output: forward: add default for Workers

Signed-off-by: lecaros <lecaros@calyptia.com>

* output: http: add default for Workers

Signed-off-by: lecaros <lecaros@calyptia.com>

* output: prometheus_remote_write: add default for Workers

Signed-off-by: lecaros <lecaros@calyptia.com>

* output: splunk: add default for Workers

Signed-off-by: lecaros <lecaros@calyptia.com>

* output: stackdriver: add default for Workers

Signed-off-by: lecaros <lecaros@calyptia.com>

* output: stdout: add default for Workers

Signed-off-by: lecaros <lecaros@calyptia.com>

* output: tcp: add default for Workers

Signed-off-by: lecaros <lecaros@calyptia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@edsiper edsiper

Labels
fixed question
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@edsiper @avdhoot @yubobo @acarsercan