Failed to save data for plugin storage under antivirus solution

[nmaludy]

Hello, i'm running fluentd (1.10.2) using td-agent (3.7.1) on Windows 2016

My FluentD config looks like:

<match>
  flush_interval 1s
  host xxx
  port 12345
  protocol udp
  @type gelf
</match>

<filter>
  @type record_transformer
  <record>
    message ${record[description]}
  </record>
</filter>

<source>
  channels application,system,security
  @id windows_eventlog
  <storage>
    persistent true
    @type local
  </storage>
  tag windows_eventlog
  @type windows_eventlog
</source>

<system>
  root_dir C:/opt/td-agent
</system>

And i'm seeing the following errors coming through on the logging stream:

[windows_eventlog] failed to save data for plugin storage to file path="C:/opt/td-agent/worker0/windows_eventlog/storage.json" tmp="C:/opt/td-agent/worker0/windows_eventlog/storage.json.tmp" error_class=Errno::EACCES error="Permission denied @ rb_file_s_rename - (C:/opt/td-agent/worker0/windows_eventlog/storage.json.tmp, C:/opt/td-agent/worker0/windows_eventlog/storage.json)"

I checked and can see that this directory does exist and the file C:/opt/td-agent/worker0/windows_eventlog/storage.json is created and has the following contents:

{"application":[541177,41519],"system":[648,54300],"security":[8973443,1]}

Any ideas why i might be getting a permission denied error? It appears that the service is running as System:

Not sure what else to check, any thoughts?

[philipsabri]

Are you running the td-agent as admin?

It's required if your are getting logs from channels like "Security"

[nmaludy]

@Flurren I haven't changed the Log On settings of the service. Does the default SYSTEM user not have access to the Security channel? Should i change the Log On setting to a local user with Administator rights?

[nmaludy]

@Flurren I have changed the service to login as local admin and no change. Still seeing the errors above and now that you mention the Security channel i'm not seeing logs from that either. Any thoughts on next-steps for debugging this?

[philipsabri]

I manage to replicate your issue by denying write permission for SYSTEM on the opt folder. Which makes sense. The error is actually saying it dont have permission to write to C:\opt\td-agent\worker0\windows_eventlog\storage.json

Can you double check that you are not denying any SYSTEM permission on the storage.json file?

[nmaludy]

@Flurren I confirmed that all folders and files in the path including C:\opt\td-agent\worker0\windows_eventlog\storage.json are all writable by SYSTEM.

I also confirmed that the ruby processes are indeed running as SYSTEM in Task Manager:

[nmaludy]

What is interesting in this is that i see the timestamp of the storage.json file being updated and the storage.json.tmp files coming and going.

[philipsabri]

I also managed to replicate the issue by having it run as a service and from the td-agent command promt at the same time. I guess they are both trying to access it by the same time, do you think you running anything in the background?

[nmaludy]

@Flurren I don't think i'm running anything else in the background. Here are the current Ruby / FluentD processes running:

Do any of those look like duplicates or different from your test system?

[philipsabri]

Its also running 3x Ruby interpeter for me so that should be fine

Try storing with a path

<storage>
    @type local
    persistent true
    path "c:/opt/td-agent/storage.json"
</storage>

[nmaludy]

@Flurren tried that config and am still seeing the error:

[windows_eventlog] failed to save data for plugin storage to file path="c:/opt/td-agent/storage.json" tmp="c:/opt/td-agent/storage.json.tmp" error_class=Errno::EACCES error="Permission denied @ rb_file_s_rename - (c:/opt/td-agent/storage.json.tmp, c:/opt/td-agent/storage.json)"

[philipsabri]

@nmaludy
Hmm not sure, I think the devs would know more about this.

Also you should use in_windows_eventlog2 it has more option and will replace @in_windows_eventlog, it will probably be what you are looking for when it comes to eventlogs in the end anyway. I'm not sure if it will fix your issue but who knows, maybe.

[nmaludy]

@Flurren i tried your suggestion of in_windows_eventlog2 but am seeing the worker fail to start with the following error in the td-agent.log:

2020-06-08 08:31:59 -0400 [error]: #0 unexpected error error_class=TypeError error="no implicit conversion of Array into String"
  2020-06-08 08:31:59 -0400 [error]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/nokogiri-1.10.9-x64-mingw32/lib/nokogiri/xml/sax/parser.rb:108:in `memory'
  2020-06-08 08:31:59 -0400 [error]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/nokogiri-1.10.9-x64-mingw32/lib/nokogiri/xml/sax/parser.rb:108:in `parse_memory'
  2020-06-08 08:31:59 -0400 [error]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/nokogiri-1.10.9-x64-mingw32/lib/nokogiri/xml/sax/parser.rb:83:in `parse'
  2020-06-08 08:31:59 -0400 [error]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-windows-eventlog-0.5.4/lib/fluent/plugin/in_windows_eventlog2.rb:163:in `bookmark_validator'
  2020-06-08 08:31:59 -0400 [error]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-windows-eventlog-0.5.4/lib/fluent/plugin/in_windows_eventlog2.rb:138:in `subscribe_channel'
  2020-06-08 08:31:59 -0400 [error]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-windows-eventlog-0.5.4/lib/fluent/plugin/in_windows_eventlog2.rb:131:in `block in start'
  2020-06-08 08:31:59 -0400 [error]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-windows-eventlog-0.5.4/lib/fluent/plugin/in_windows_eventlog2.rb:130:in `each'
  2020-06-08 08:31:59 -0400 [error]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-windows-eventlog-0.5.4/lib/fluent/plugin/in_windows_eventlog2.rb:130:in `start'
  2020-06-08 08:31:59 -0400 [error]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.10.2/lib/fluent/root_agent.rb:200:in `block in start'
  2020-06-08 08:31:59 -0400 [error]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.10.2/lib/fluent/root_agent.rb:189:in `block (2 levels) in lifecycle'
  2020-06-08 08:31:59 -0400 [error]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.10.2/lib/fluent/root_agent.rb:188:in `each'
  2020-06-08 08:31:59 -0400 [error]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.10.2/lib/fluent/root_agent.rb:188:in `block in lifecycle'
  2020-06-08 08:31:59 -0400 [error]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.10.2/lib/fluent/root_agent.rb:175:in `each'
  2020-06-08 08:31:59 -0400 [error]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.10.2/lib/fluent/root_agent.rb:175:in `lifecycle'
  2020-06-08 08:31:59 -0400 [error]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.10.2/lib/fluent/root_agent.rb:199:in `start'
  2020-06-08 08:31:59 -0400 [error]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.10.2/lib/fluent/engine.rb:248:in `start'
  2020-06-08 08:31:59 -0400 [error]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.10.2/lib/fluent/engine.rb:147:in `run'
  2020-06-08 08:31:59 -0400 [error]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.10.2/lib/fluent/supervisor.rb:600:in `block in run_worker'
  2020-06-08 08:31:59 -0400 [error]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.10.2/lib/fluent/supervisor.rb:837:in `main_process'
  2020-06-08 08:31:59 -0400 [error]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.10.2/lib/fluent/supervisor.rb:594:in `run_worker'
  2020-06-08 08:31:59 -0400 [error]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.10.2/lib/fluent/command/fluentd.rb:350:in `<top (required)>'
  2020-06-08 08:31:59 -0400 [error]: #0 C:/opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  2020-06-08 08:31:59 -0400 [error]: #0 C:/opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  2020-06-08 08:31:59 -0400 [error]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.10.2/bin/fluentd:8:in `<top (required)>'
  2020-06-08 08:31:59 -0400 [error]: #0 C:/opt/td-agent/embedded/bin/fluentd:23:in `load'
  2020-06-08 08:31:59 -0400 [error]: #0 C:/opt/td-agent/embedded/bin/fluentd:23:in `<main>'
2020-06-08 08:31:59 -0400 [error]: #0 unexpected error error_class=TypeError error="no implicit conversion of Array into String"
  2020-06-08 08:31:59 -0400 [error]: #0 suppressed same stacktrace
2020-06-08 08:31:59 -0400 [info]: Worker 0 finished unexpectedly with status 1

I copy/pasted the config snippet from the link you sent me.

[nmaludy]

Fixed that error by deleting the old storage.json file i had in the same place. Looks like the new plugin has a different storage format. The plugin is running now, but i'm still seeing errors related to renaming the storage file.

[philipsabri]

Did you try to run the config with the td-agent command promt as admin (not as a service)?
Not sure if it would give another result but might be worth a try. If nothing works I would also try to reinstall the whole td-agent.
@nmaludy

[nmaludy]

@Flurren No change running it directly from the td-agent command prompt as Administrator. Also with this setup it still isn't able to read the Security channel.

Also with the windows_eventlog2 setup, i was never able to get it to read from any channel. No logs seemed to have ever been sent to the central logging server. Debugging this further now.